A multi-port remote RSYSLOG log server: multiple separate ports with different store forward rules for remote syslog clients

DECnet/Python nodes logging to Papertrail cloud-hosted log management service

The Requirement

I have six instances of Paul Koning's DECnet/Python (e.g. PYRTR) doing DECnet routing and wanted to send log messages from all six to one central location for analysis and further distribution.

While the above goal is very specific, the general requirement is to be able to forward logs from chosen applications to a dedicated TCP port different from the standard port 514 on a multi-port remote log server so that these logs are independently processed and and can be saved into separate log files and forwarded to other log collector and analyzer tools. The second objective is to do this with minimal impact to existing system-logging infrastructure on the existing remote log server.

The Design

I have the usual RSYSLOG (the rocket-fast system for log processing) version 7.4.4 running on Ubuntu 14.04 Linux on an old VPS. I wanted to open an additional port for RSYSLOG to listen to, without disturbing existing logging and rules. I would then send over TCP/IP all DECnet/Python logs to this special port and define rules for RSYSLOG to apply to only this dedicated port. These rules would write the DECnet/Pyhton logs received over TCP/IP to a separate log file and forward only DECnet/Python logs to other log collectors including my favorite Papertrail cloud-hosted log management service.

The Implementation

Ubuntu 14.04's configuration of RSYSLOGD is simple. /etc/rsyslog.conf pretty much loads required modules and hands off specific configuration to separate files in /etc/rsyslog.d. This makes it very easy to add to the configuration by simply introducing a new configuration file in /etc/rsyslog.d.

For this example, we will use port 8514 as our dedicated log collection port for DECnet/Python. In addition to TCP/IP, we will also be ready to accept logs from remote syslog clients over UDP/IP.

First, enable the imudp and imtcp modules by uncommenting them near the top of /etc/rsyslog.conf. For internet-facing servers, you can use a firewall (UFW and friends will do fine) to block access to TCP and UDP port 514 from the internet.

# provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Now create a configuration file 97-pydecnet-collector.conf in /etc/rsyslog.d. This file should have contents like the following. Essentially, this configuration results in RSYSLOG listening to the ports mentioned in the last two lines, and then when it receives log entries on those ports, it performs the "actions" in the ruleset till it hits "stop". "stop" means discard the received log message at that point with no further processing.

See the inline comments for details and adjust according to your requirements.

# /etc/rsyslog.d/97-pydecnet-collector.conf

ruleset(name="pydecnet-collect"){

    # write the incoming log message to the indicated file

    action(type="omfile" file="/var/log/pydecnet-collect.log")

    # forward to another log server over network, queuing up at most

    # the indicated number of log entries if network link goes down

    # adjust target, port-number as per your other remote log server

    action(type="omfwd" target="<remote-hostname>" protocol="tcp" port="<remote-port>"

           queue.filename="pydecnet-queue-to-vps" queue.size="1000" queue.type="LinkedList")

    # forward to papertrail log server over network, queuing up 

    # at most the indicated number of log entries if 

    # network link goes down.
    # actual target host and port number will be provided by 

    # papertrail when you sign up

    action(type="omfwd" target="<host>.papertrailapp.com" protocol="tcp" port="<remote-papertrail-port>"

           queue.filename="pydecnet-queue-to-papertrail" queue.size="1000" queue.type="LinkedList")

    # forward to yet another log server over network!

    action(type="omfwd" target="<remote-hostname>" protocol="tcp" port="<remote-port>"

           queue.filename="pydecnet-queue-to-svr2" queue.size="1000" queue.type="LinkedList")

    # More actions can of course be added ...

    # Stop processing here and discard the log message

    stop

}

# End of ruleset

# Listen for and process (apply ruleset) on incoming log 

# messages from remote syslog clients

input(type="imudp" port="8514" ruleset="pydecnet-collect")

input(type="imtcp" port="8514" ruleset="pydecnet-collect")

# EOF

Then restart RSYSLOG. You can check the regular syslog file (/var/log/syslog in my case) for any errors reported by RSYSLOG daemon and address any reported errors in the new configuration.