Search This Blog

Wednesday, October 26, 2016

Secure Remote Logging to Central Log Server Using RSYSLOG on CentOS 6 / CentOS 7 and stunnel

Secure Remote Logging to Central Log Server via Encrypted stunnel Tunneling
Here is how I set up a secure remote rsyslog server on my VPS that accepts and records system logs from various clients on my home and cloud hobbyist networks.

The remote syslog server runs rsyslog daemon on CentOS 7, and the clients are a mixture of numerous operating systems, including Linux CentOS 6 and CentOS7, Windows, BSD, Routers running DD-WRT etc.

The basic idea is that local rsyslog daemons forward logs to stunnel also running locally. stunnel then forwards them to the remote server using a secure tunnel over TCP/IP. A peer stunnel daemon instance running on the remote server then forwards the incoming logs to the local rsyslog daemon on the remote server.

Server setup (CentOS 7)

In my setup, the server-side stunnel listens on port 65515 and forwards logs to rsyslog which listens on port 65514.

Install stunnel and add a script to start and stop it using the systemctl facility.

# yum install openssl stunnel

Add a file /usr/lib/systemd/system/stunnel.service with the following contents:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/kill -9 $(pgrep stunnel)

[Install]
WantedBy=multi-user.target




Add a file /etc/stunnel/stunnel.conf containing the following:

client = no
fips = no
cert = /etc/stunnel/stunnel.pem
[rsyslog]
accept = 65515
connect = 127.0.0.1:65514

Generate the server's certificate. The following will generate a certificate with an expiry of 100 years. Important: enter the fully qualified host-name of the server for the question "Common Name". Answer the questions the best you can or want to. I like to put in meaningful information for all the questions, including my real email address for the last question.

# cd /etc/stunnel
# openssl req -new -x509 -days 36500 -nodes -out stunnel.pem -keyout stunnel.pem
Generating a 2048 bit RSA private key
.............................+++
.+++
writing new private key to 'stunnel.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:MD
Locality Name (eg, city) [Default City]:Germantown
Organization Name (eg, company) [Default Company Ltd]:Supratim Sanyal's Hobbyist Lab
Organizational Unit Name (eg, section) []:Basement Server
Common Name (eg, your name or your server's hostname) []:sanyalnet-cloud-vps2.freeddns.org
Email Address []: youremailhere@emaildomain.com

You should now see the newly generated certificate in the /etc/stunnel directory:

# ls -lrt
total 8
-rw------- 1 root root  105 Oct 26 20:54 stunnel.conf
-rw------- 1 root root 3278 Oct 26 23:04 stunnel.pem

Now configure the rsyslog system log daemon on this remote syslog server to listen to logs at the port indicated by connect line in the stunnel config file which you created above. Edit the file /etc/rsyslog.conf and look for, uncomment and adjust the following two lines (they were at around line 18 on my installation of CentOS 7):

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 65514

The last configuration step is to open up the stunnel listening port to the external internet on the firewall. We also need rules to allow all traffic on the localhost (127.0.0.1) interface "lo" so that network traffic between stunnel and rsyslog flows internally over localhost. We then add firewall reject rules to reject everything else.


# iptables -A INPUT -i lo -j ACCEPT

# iptables -A OUTPUT -o lo -j ACCEPT
# iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 65515 -j ACCEPT
# iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited

# iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited


Of course, to make these rules permanent, I recommend you edit and add them to the default iptables startup rules file /etc/sysconfig/iptables, adjusting for any existing other rules you may already have. In particular, for security, make sure you do not open up the local rsyslog listening port 65514 on your firewall; this is important since rsyslog binds to all interfaces and there is no easy way to force it to bind to the localhost (127.0.0.1) interface.

We are done configuring the rsyslog server, and can now enable and start up stunnel, and restart rsyslog, for the changes to take effect.

# systemctl enable stunnel
Created symlink from /etc/systemd/system/multi-user.target.wants/stunnel.service to /usr/lib/systemd/system/stunnel.service.
# systemctl start stunnel
# systemctl restart rsyslog

Perform some basic validations to check if everything is working as expected. Verify the stunnel daemon is up and running:

# systemctl status stunnel
stunnel.service - SSL tunnel for network daemons
   Loaded: loaded (/usr/lib/systemd/system/stunnel.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2016-10-26 22:57:49 UTC; 18min ago
 Main PID: 675 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/stunnel.service
           ├─696 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─697 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─698 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─699 /bin/stunnel /etc/stunnel/stunnel.conf
           ├─700 /bin/stunnel /etc/stunnel/stunnel.conf
           └─701 /bin/stunnel /etc/stunnel/stunnel.conf

Oct 26 22:57:48 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Starting SSL tunnel for network daemons...
Oct 26 22:57:49 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started SSL tunnel for network daemons.
Oct 26 23:16:00 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started SSL tunnel for network daemons.

Verify rsyslog is working with the changed configuration:

# systemctl status rsyslog
rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2016-10-26 23:16:04 UTC; 2min 6s ago
 Main PID: 2756 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─2756 /usr/sbin/rsyslogd -n

Oct 26 23:16:04 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Starting System Logging Service...
Oct 26 23:16:04 sanyalnet-cloud-vps2.freeddns.org systemd[1]: Started System Logging Service.

Verify that stunnel's startup activity is logged in the /var/log/secure logfile:

[root@sanyalnet-cloud-vps2 ~]# cat /var/log/secure
...
...
...

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: stunnel 4.56 on x86_64-redhat-linux-gnu platform

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Compiled/running with OpenSSL 1.0.1e-fips 11 Feb 2013

Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Threading:PTHREAD Sockets:POLL,IPv6 SSL:ENGINE,OCSP,FIPS Auth:LIBWRAP
Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Reading configuration from file /etc/stunnel/stunnel.conf
Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: FIPS mode is disabled
Oct 26 22:57:49 sanyalnet-cloud-vps2 stunnel: LOG5[675:140205220452416]: Configuration successful

Verify the correct server network ports are open:

# netstat -pan | egrep "stunnel|rsyslog"
tcp        0      0 0.0.0.0:65514           0.0.0.0:*               LISTEN      2756/rsyslogd
tcp        0      0 0.0.0.0:65515           0.0.0.0:*               LISTEN      701/stunnel
tcp        0      0 127.0.0.1:65514         127.0.0.1:42938         ESTABLISHED 2756/rsyslogd
tcp        0      0 127.0.0.1:42938         127.0.0.1:65514         ESTABLISHED 701/stunnel
...
...

Client setup (CentOS 6)

In this example client running CentOS 6, we will configure the rsyslog daemon to forward all logs to the remote syslog server created above over a secure encrypted network link implemented by stunnel.

Install stunnel and add a script to start and stop it using the CentOS 6 service management facility.

# yum install openssl stunnel

Add a file /etc/init.d/stunnel with the following contents:

#!/bin/sh
#
# $Id: stunnel.init.in,v 1.0 2016/10/26 21:11:58 mschimek Exp $
# Startup script for the stunnel daemon, RedHat style.
# See http://supratim-sanyal.blogspot.com/2016/10/secure-remote-logging-to-central-log.html
#
# chkconfig: - 90 10
# description: stunnel daemon to establish encrypted secure network channel between two hosts
#
# Supratim Sanyal <supratim at riseup dot net>
#

. /etc/init.d/functions

prefix=/usr
exec_prefix=/usr

[ -x /usr/bin/stunnel ]  || exit 1

RETVAL=0

start(){
        echo -n $"Starting stunnel secure sockets tunnel daemon: "
        daemon stunnel
        RETVAL=$?
        echo
        touch /var/lock/subsys/stunnel
        return $RETVAL
}

stop(){
        echo -n $"Stopping stunnel secure sockets tunnel daemon: "
        killproc stunnel
        echo
        RETVAL=$?
        rm -f /var/lock/subsys/stunnel
        return $RETVAL
}

restart(){
        stop
        start
}

case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                restart
                ;;
        status)
                status stunnel
                ;;
        condrestart)
                [ -e /var/lock/subsys/stunnel ] && restart
                ;;
        *)
                echo $"Usage: $0 {start|stop|status|restart|condrestart}"
                RETVAL=1
esac

exit $RETVAL


Add the stunnel init script to the list of chkconfig managed init scripts, make it executable, and enable it, but do not start it yet.

# chmod +x /etc/init.d/stunnel
# chkconfig --add stunnel
# chkconfig stunnel on
# chkconfig --list stunnel
stunnel         0:off   1:off   2:on    3:on    4:on    5:on    6:off
# service stunnel status
stunnel is stopped

Add a file /etc/stunnel/stunnel.conf with the following contents, replacing <rsyslog-remote-server-hostname> with your actual fully qualified network-reachable remote syslog server hostname (or IP address):

client = yes
fips = no
[rsyslogd]
accept = 127.0.0.1:65514
connect = <rsyslog-remote-server-hostname>:65515

Note: you do not need a certificate on stunnel client side.

Edit the file /etc/rsyslog.conf and add the following at the bottom. If, however, you already have an existing forwarding rule for some other syslog server, insert the following lines below the existing $WorkDirectory but above the existing forwarding rule-sets.

$WorkDirectory /var/log # where to place spool files
# begin forwarding rule to stunnel port for passing on to VPS (see /etc/stunnel/stunnel.conf)
$ActionQueueType LinkedList
$ActionQueueFileName logs-spooled-4-VPS
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @@127.0.0.1:65514
# ### end of the forwarding rule ###


Configuration is complete at this point. Start / restart the necessary daemons.

# service restart stunnel
# service restart rsyslog

Client setup (CentOS 7)

On a client running CentOS 7, we will configure the rsyslog daemon to forward all logs to the remote syslog server created above over a secure encrypted network link implemented by stunnel.
Install stunnel and add a script to start and stop it using the systemctl facility.

# yum install openssl stunnel

Add a file /usr/lib/systemd/system/stunnel.service with the following contents:

[Unit]
Description=SSL tunnel for network daemons
After=syslog.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/stunnel /etc/stunnel/stunnel.conf
ExecStop=/bin/kill -9 $(pgrep stunnel)

[Install]
WantedBy=multi-user.target



Enable the stunnel daemon (but do not start it yet):

# systemctl enable stunnel
Created symlink from /etc/systemd/system/multi-user.target.wants/stunnel.service to /usr/lib/systemd/system/stunnel.service.


Add a file /etc/stunnel/stunnel.conf with the following contents, replacing <rsyslog-remote-server-hostname> with your actual fully qualified network-reachable remote syslog server hostname (or IP address):

client = yes
fips = no
[rsyslogd]
accept = 127.0.0.1:65514
connect = <rsyslog-remote-server-hostname>:65515

Note: you do not need a certificate on stunnel client side.

Edit the file /etc/rsyslog.conf and add the following at the bottom. If, however, you already have an existing forwarding rule for some other syslog server, insert the following lines below the existing $WorkDirectory but above the existing forwarding rule-sets.

$WorkDirectory /var/log # where to place spool files
# begin forwarding rule to stunnel port for passing on to VPS (see /etc/stunnel/stunnel.conf)
$ActionQueueType LinkedList
$ActionQueueFileName logs-spooled-4-VPS
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @@127.0.0.1:65514
# ### end of the forwarding rule ###


Configuration is complete at this point. Start / restart the necessary daemons.

# systemctl start stunnel
# systemctl restart rsyslog


Testing

Perform some basic validations to check if everything on the rsyslog and stunnel daemons on the client side are working as expected. Verify the stunnel and rsyslog daemons are up and running:

# service stunnel status
stunnel (pid  13781) is running...
# service rsyslog status
rsyslogd (pid  11285) is running...

Verify stunnel has connected at both the rsyslog port and peer stunnel on the remote server's port:

# cat /var/log/secure
Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: stunnel 4.29 on x86_64-redhat-linux-gnu with OpenSSL 1.0.1e-fips 11 Feb 2013
Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: Threading:PTHREAD SSL:ENGINE,FIPS Sockets:POLL,IPv6 Auth:LIBWRAP
Oct 26 20:57:53 dormarth stunnel: LOG5[10866:139787722659776]: 500 clients allowed
Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: rsyslogd accepted connection from 127.0.0.1:44734
Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: connect_blocking: connected 64.137.228.122:65515
Oct 26 20:58:00 dormarth stunnel: LOG5[10872:139787722655488]: rsyslogd connected remote server from 10.42.2.2:46658

Verify the network ports that you expecting are indeed in use:

# netstat -pan | egrep "rsyslog|stunnel"
tcp        0      0 127.0.0.1:65514             0.0.0.0:*                   LISTEN      13781/stunnel
tcp        0      0 127.0.0.1:49450             127.0.0.1:65514             ESTABLISHED 11285/rsyslogd
tcp        0      1 10.42.2.2:58520             64.137.228.85:65515         SYN_SENT    13781/stunnel
tcp      499      0 127.0.0.1:65514             127.0.0.1:49450             ESTABLISHED 13781/stunnel

Finally, send something to local logger and verify you can see it on the remote server. On the server:

# tail -f /var/log/messages

On the client host:

# logger "`date` tunnel test"

and verify on the server you see the "<date> tunnel test" line in the remote syslog file:

Oct 27 02:35:08 dormarth rumtuk: Thu Oct 27 02:35:08 UTC 2016 tunnel test

If you have the wireshark package installed, you may also want to run tshark on the stunnel port to verify the messages are indeed going across encrypted. For example, on the server, here is the traffic on port 65515 when the same logger command as above is repeated on the client:

# tshark -i ens33 | grep 65515
Running as user "root" and group "root". This could be dangerous.
Capturing on 'ens33'

80  64 5.467663041 108.31.178.247 -> 64.137.248.212 TCP 161 fjdmimgr > 65515 [PSH, ACK] Seq=1 Ack=1 Win=160 Len=107
 65 5.467829924 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=108 Win=497 Len=0
 69 6.097558428 108.31.178.247 -> 64.137.248.212 TCP 161 fjdmimgr > 65515 [PSH, ACK] Seq=108 Ack=1 Win=160 Len=107
 70 6.097749638 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=215 Win=497 Len=0
199 190 19.123101437 108.31.178.247 -> 64.137.248.212 TCP 157 fjdmimgr > 65515 [PSH, ACK] Seq=215 Ack=1 Win=160 Len=103
191 19.123287145 64.137.248.212 -> 108.31.178.247 TCP 54 65515 > fjdmimgr [ACK] Seq=1 Ack=318 Win=497 Len=0
405

No comments:

Post a Comment

Recommended Products from Amazon