Secure Remote Logging to Central Log Server Using RSYSLOG on CentOS 6 / CentOS 7 and stunnel

Secure Remote Logging to Central Log Server via Encrypted stunnel Tunneling

Here is how I set up a secure remote rsyslog server on my VPS that accepts and records system logs from various clients on my home and cloud hobbyist networks.

The remote syslog server runs rsyslog daemon on CentOS 7, and the clients are a mixture of numerous operating systems, including Linux CentOS 6 and CentOS7, Windows, BSD, Routers running DD-WRT etc.

The basic idea is that local rsyslog daemons forward logs to stunnel also running locally. stunnel then forwards them to the remote server using a secure tunnel over TCP/IP. A peer stunnel daemon instance running on the remote server then forwards the incoming logs to the local rsyslog daemon on the remote server.

Server setup (CentOS 7)

In my setup, the server-side stunnel listens on port 65515 and forwards logs to rsyslog which listens on port 65514.

Install stunnel and add a script to start and stop it using the systemctl facility.

# yum install openssl stunnel

Add a file /usr/lib/systemd/system/stunnel.service with the following contents:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/kill -9 $(pgrep stunnel)

[Install]
WantedBy=multi-user.target

Add a file /etc/stunnel/stunnel.conf containing the following:

client = no
fips = no
cert = /etc/stunnel/stunnel.pem
[rsyslog]
accept = 65515
connect = 127.0.0.1:65514

Generate the server's certificate. The following will generate a certificate with an expiry of 100 years. Important: enter the fully qualified host-name of the server for the question "Common Name". Answer the questions the best you can or want to. I like to put in meaningful information for all the questions, including my real email address for the last question.

# cd /etc/stunnel

# openssl req -new -x509 -days 36500 -nodes -out stunnel.pem -keyout stunnel.pem

Generating a 2048 bit RSA private key

.............................+++

.+++

writing new private key to 'stunnel.pem'

-----

You are about to be asked to enter information that will be incorporated

into your certificate request.

What you are about to enter is what is called a Distinguished Name or a DN.

There are quite a few fields but you can leave some blank

For some fields there will be a default value,

If you enter '.', the field will be left blank.

-----

Country Name (2 letter code) [XX]:US

State or Province Name (full name) []:MD

Locality Name (eg, city) [Default City]:Germantown

Organization Name (eg, company) [Default Company Ltd]:Supratim Sanyal's Hobbyist Lab

Organizational Unit Name (eg, section) []:Basement Server

Common Name (eg, your name or your server's hostname) []:sanyalnet-cloud-vps2.freeddns.org

Email Address []: youremailhere@emaildomain.com

You should now see the newly generated certificate in the /etc/stunnel directory:

# ls -lrt

total 8

-rw------- 1 root root 105 Oct 26 20:54 stunnel.conf

-rw------- 1 root root 3278 Oct 26 23:04 stunnel.pem

Now configure the rsyslog system log daemon on this remote syslog server to listen to logs at the port indicated by connect line in the stunnel config file which you created above. Edit the file /etc/rsyslog.conf and look for, uncomment and adjust the following two lines (they were at around line 18 on my installation of CentOS 7):

# Provides TCP syslog reception

$ModLoad imtcp

$InputTCPServerRun 65514

The last configuration step is to open up the stunnel listening port to the external internet on the firewall. We also need rules to allow all traffic on the localhost (127.0.0.1) interface "lo" so that network traffic between stunnel and rsyslog flows internally over localhost. We then add firewall reject rules to reject everything else.

# iptables -A INPUT -i lo -j ACCEPT

# iptables -A OUTPUT -o lo -j ACCEPT

# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 65515 -j ACCEPT

# iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

Of course, to make these rules permanent, I recommend you edit and add them to the default iptables startup rules file /etc/sysconfig/iptables, adjusting for any existing other rules you may already have. In particular, for security, make sure you do not open up the local rsyslog listening port 65514 on your firewall; this is important since rsyslog binds to all interfaces and there is no easy way to force it to bind to the localhost (127.0.0.1) interface.

We are done configuring the rsyslog server, and can now enable and start up stunnel, and restart rsyslog, for the changes to take effect.

# systemctl enable stunnel

Created symlink from /etc/systemd/system/multi-user.target.wants/stunnel.service to /usr/lib/systemd/system/stunnel.service.
# systemctl start stunnel
# systemctl restart rsyslog

Perform some basic validations to check if everything is working as expected. Verify the stunnel daemon is up and running:

# systemctl status stunnel

● stunnel.service - SSL tunnel for network daemons

Loaded: loaded (/usr/lib/systemd/system/stunnel.service; enabled; vendor preset: disabled)

Active: active (exited) since Wed 2016-10-26 22:57:49 UTC; 18min ago

Main PID: 675 (code=exited, status=0/SUCCESS)

CGroup: /system.slice/stunnel.service

├─696 /bin/stunnel /etc/stunnel/stunnel.conf

├─697 /bin/stunnel /etc/stunnel/stunnel.conf

├─698 /bin/stunnel /etc/stunnel/stunnel.conf

├─699 /bin/stunnel /etc/stunnel/stunnel.conf

├─700 /bin/stunnel /etc/stunnel/stunnel.conf

└─701 /bin/stunnel /etc/stunnel/stunnel.conf

Oct 26 22:57:48 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Starting SSL tunnel for network daemons...

Oct 26 22:57:49 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started SSL tunnel for network daemons.

Oct 26 23:16:00 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started SSL tunnel for network daemons.

Verify rsyslog is working with the changed configuration:

# systemctl status rsyslog

● rsyslog.service - System Logging Service

Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)

Active: active (running) since Wed 2016-10-26 23:16:04 UTC; 2min 6s ago

Main PID: 2756 (rsyslogd)

CGroup: /system.slice/rsyslog.service

└─2756 /usr/sbin/rsyslogd -n

Oct 26 23:16:04 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Starting System Logging Service...

Oct 26 23:16:04 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started System Logging Service.

Verify that stunnel's startup activity is logged in the /var/log/secure logfile:

[root@sanyalnet-cloud-vps2 ~]# cat /var/log/secure

...
...
...

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: stunnel 4.56 on x86_64-redhat-linux-gnu platform

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Reading configuration from file /etc/stunnel/stunnel.conf

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: FIPS mode is disabled

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Configuration successful

Verify the correct server network ports are open:

# netstat -pan | egrep "stunnel|rsyslog"

tcp 0 0 0.0.0.0:65514 0.0.0.0:* LISTEN 2756/rsyslogd

tcp 0 0 0.0.0.0:65515 0.0.0.0:* LISTEN 701/stunnel

tcp 0 0 127.0.0.1:65514 127.0.0.1:42938 ESTABLISHED 2756/rsyslogd

tcp 0 0 127.0.0.1:42938 127.0.0.1:65514 ESTABLISHED 701/stunnel

...

Client setup (CentOS 6)

In this example client running CentOS 6, we will configure the rsyslog daemon to forward all logs to the remote syslog server created above over a secure encrypted network link implemented by stunnel.

Install stunnel and add a script to start and stop it using the CentOS 6 service management facility.

# yum install openssl stunnel

Add a file /etc/init.d/stunnel with the following contents:

#!/bin/sh

# $Id: stunnel.init.in,v 1.0 2016/10/26 21:11:58 mschimek Exp $

# Startup script for the stunnel daemon, RedHat style.

# See http://supratim-sanyal.blogspot.com/2016/10/secure-remote-logging-to-central-log.html

# chkconfig: - 90 10

# description: stunnel daemon to establish encrypted secure network channel between two hosts

# Supratim Sanyal <supratim at riseup dot net>

. /etc/init.d/functions

prefix=/usr

exec_prefix=/usr

[ -x /usr/bin/stunnel ] || exit 1

RETVAL=0

start(){

echo -n $"Starting stunnel secure sockets tunnel daemon: "

daemon stunnel

RETVAL=$?

echo

touch /var/lock/subsys/stunnel

return $RETVAL

}

stop(){

echo -n $"Stopping stunnel secure sockets tunnel daemon: "

killproc stunnel

echo

RETVAL=$?

rm -f /var/lock/subsys/stunnel

return $RETVAL

}

restart(){

stop

start

}

case "$1" in

start)

start

;;

stop)

stop

;;

restart)

restart

;;

status)

status stunnel

;;

condrestart)

[ -e /var/lock/subsys/stunnel ] && restart

;;

echo $"Usage: $0 {start|stop|status|restart|condrestart}"

RETVAL=1

esac

exit $RETVAL

Add the stunnel init script to the list of chkconfig managed init scripts, make it executable, and enable it, but do not start it yet.

# chmod +x /etc/init.d/stunnel
# chkconfig --add stunnel

# chkconfig stunnel on

# chkconfig --list stunnel

stunnel 0:off 1:off 2:on 3:on 4:on 5:on 6:off

# service stunnel status

stunnel is stopped

Add a file /etc/stunnel/stunnel.conf with the following contents, replacing <rsyslog-remote-server-hostname> with your actual fully qualified network-reachable remote syslog server hostname (or IP address):

client = yes

fips = no

[rsyslogd]

accept = 127.0.0.1:65514

connect = <rsyslog-remote-server-hostname>:65515

Note: you do not need a certificate on stunnel client side.

Edit the file /etc/rsyslog.conf and add the following at the bottom. If, however, you already have an existing forwarding rule for some other syslog server, insert the following lines below the existing $WorkDirectory but above the existing forwarding rule-sets.

$WorkDirectory /var/log # where to place spool files

# begin forwarding rule to stunnel port for passing on to VPS (see /etc/stunnel/stunnel.conf)

$ActionQueueType LinkedList

$ActionQueueFileName logs-spooled-4-VPS

$ActionResumeRetryCount -1

$ActionQueueSaveOnShutdown on

*.* @@127.0.0.1:65514

# ### end of the forwarding rule ###

Configuration is complete at this point. Start / restart the necessary daemons.

# service restart stunnel

# service restart rsyslog

Client setup (CentOS 7)

On a client running CentOS 7, we will configure the rsyslog daemon to forward all logs to the remote syslog server created above over a secure encrypted network link implemented by stunnel.

Install stunnel and add a script to start and stop it using the systemctl facility.

# yum install openssl stunnel

Add a file /usr/lib/systemd/system/stunnel.service with the following contents:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/kill -9 $(pgrep stunnel)

[Install]
WantedBy=multi-user.target

Enable the stunnel daemon (but do not start it yet):

# systemctl enable stunnel

Created symlink from /etc/systemd/system/multi-user.target.wants/stunnel.service to /usr/lib/systemd/system/stunnel.service.

client = yes

fips = no

[rsyslogd]

accept = 127.0.0.1:65514

connect = <rsyslog-remote-server-hostname>:65515

Note: you do not need a certificate on stunnel client side.

$WorkDirectory /var/log # where to place spool files

# begin forwarding rule to stunnel port for passing on to VPS (see /etc/stunnel/stunnel.conf)

$ActionQueueType LinkedList

$ActionQueueFileName logs-spooled-4-VPS

$ActionResumeRetryCount -1

$ActionQueueSaveOnShutdown on

*.* @@127.0.0.1:65514

# ### end of the forwarding rule ###

Configuration is complete at this point. Start / restart the necessary daemons.

# systemctl start stunnel

# systemctl restart rsyslog

Testing

Perform some basic validations to check if everything on the rsyslog and stunnel daemons on the client side are working as expected. Verify the stunnel and rsyslog daemons are up and running:

# service stunnel status

stunnel (pid 13781) is running...

# service rsyslog status

rsyslogd (pid 11285) is running...

Verify stunnel has connected at both the rsyslog port and peer stunnel on the remote server's port:

# cat /var/log/secure

Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: stunnel 4.29 on x86_64-redhat-linux-gnu with OpenSSL 1.0.1e-fips 11 Feb 2013

Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: Threading:PTHREAD SSL:ENGINE,FIPS Sockets:POLL,IPv6 Auth:LIBWRAP

Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: 500 clients allowed

Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: rsyslogd accepted connection from 127.0.0.1:44734

Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: connect_blocking: connected 64.137.228.122:65515

Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: rsyslogd connected remote server from 10.42.2.2:46658

Verify the network ports that you expecting are indeed in use:

# netstat -pan | egrep "rsyslog|stunnel"

tcp 0 0 127.0.0.1:65514 0.0.0.0:* LISTEN 13781/stunnel

tcp 0 0 127.0.0.1:49450 127.0.0.1:65514 ESTABLISHED 11285/rsyslogd

tcp 0 1 10.42.2.2:58520 64.137.228.85:65515 SYN_SENT 13781/stunnel

tcp 499 0 127.0.0.1:65514 127.0.0.1:49450 ESTABLISHED 13781/stunnel

Finally, send something to local logger and verify you can see it on the remote server. On the server:

# tail -f /var/log/messages

On the client host:

# logger "`date` tunnel test"

and verify on the server you see the "<date> tunnel test" line in the remote syslog file:

Oct 27 02:35:08 dormarth rumtuk: Thu Oct 27 02:35:08 UTC 2016 tunnel test

If you have the wireshark package installed, you may also want to run tshark on the stunnel port to verify the messages are indeed going across encrypted. For example, on the server, here is the traffic on port 65515 when the same logger command as above is repeated on the client:

# tshark -i ens33 | grep 65515

Running as user "root" and group "root". This could be dangerous.

Capturing on 'ens33'

80 64 5.467663041 108.31.178.247 -> 64.137.248.212 TCP 161 fjdmimgr > 65515 [PSH, ACK] Seq=1 Ack=1 Win=160 Len=107

65 5.467829924 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=108 Win=497 Len=0

69 6.097558428 108.31.178.247 -> 64.137.248.212 TCP 161 fjdmimgr > 65515 [PSH, ACK] Seq=108 Ack=1 Win=160 Len=107

70 6.097749638 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=215 Win=497 Len=0

199 190 19.123101437 108.31.178.247 -> 64.137.248.212 TCP 157 fjdmimgr > 65515 [PSH, ACK] Seq=215 Ack=1 Win=160 Len=103

191 19.123287145 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=318 Win=497 Len=0

405

Supratim Sanyal's Computing Blog | Wandering Digital Wastelands as a Geek

Wednesday, October 26, 2016