Mitigate OpenVMS Telnet Port Brute Force Intrusion Attacks from Botnets and Humans with Reporting to Blocklist.DE

Introduction

OpenVMS Remote Interactive Breakin Detection - Brute Force Attacks on TELNET port to QCOCAL

Like owners of every internet-facing DEC VAX server running Digital OpenVMS operating system accessible publicly via the Telnet port (TCP port 23), my pet SIMH VAXserver 3900 "QCOCAL" and SIMH VAX-11/780 "CLOUDY" are inundated with dictionary-based breaking attempts all the time from across the world. You can get a feel of the sheer volume of these attacks from a report I publish daily, generated by a DCL command procedure batch job, summarizing brute-force intrusion attempts at QCOCAL from the OpenVMS system using "ANALYZE/AUDIT/EVENT=(BREAKIN,LOGFAIL)/SINCE=-7-00:00:00 SYS$MANAGER:SECURITY.AUDIT$JOURNAL /output=dua2:[fal$server]intrusions.txt". This summary report is available publicly on the internet, served by WASD httpd web-server running on OpenVMS VAX 7.3, at the URL http://sanyalnet-openvms-vax.freeddns.org:82/falserver/intrusions.txt.

The telnet port intrusion malware problem has recently been exacerbated by the successful Mirai botnet which targets the IoT (Internet of Things ), looking for and installing itself in a mind-boggling number of devices like cameras, smart televisions, routers, switches and other internet-connected consumer and professional devices. The publication of the source code for Mirai has assured public server operators of similar and smarter derivatives continuing to challenge our servers. I publish a frequently-updated publicly available block-list of Mirai (and family) infected IP addresses that my systems are seeing actual brute force break-in attempts from. I use snort IDS/IPS software running on the 2nd of my 3-layer firewall for this. The URL of my Mirai blocklist is http://sanyalnet-cloud-vps.freeddns.org/mirai-ips.txt - feel free to implement a scheduled download of this file to your firewall.

However, the sheer number of sources of attack makes my attempts at denying access to servers based on just a dynamically-updated blacklist of little consequence. I needed something additional, and also wanted to report OpenVMS Telnet port 23 intrusion attempts to the blocklist.de service so that the ISPs of the offenders get notified of abuse, and the numerous users of the blocklist.de list also benefit.

In this post, I describe the rather obtuse way I achieved the goal of passing on intrusion information from OpenVMS logs to blocklist.de using the fail2ban tool running on a Linux intermediary. You can take a look at the results here.

1. Configure a central Unix-style syslog server and secure ssl tunnel for remote logging

One of the things I use my VPS sanyalnet-cloud-vps.freeddns.org located in a data center in Ontario, Canada for is as a central syslog repository which receives and logs entries from most of SANYALnet systems. I get around the issue of clear-text log transmission to the VPS over the internet by using a stunnel secure ssl tunnel at the endpoints. The local rsyslog and syslog-ng loggers forward logs to local stunnel endpoints. stunnel encrypts and transports them to the remote stunnel endpoint running on the remote VPS. The remote stunnel endpoint decrypts the log entries and sends them to the rsyslog daemon running locally on the same remote VPS.

secure remote unix-style central syslog using stunnel

For local servers that run the classic syslog daemon which is incapable of using ports other than the default syslog port (514), I point them to another rsyslog daemon running on a different node dormarth.sanyalnet.lan on the local LAN, which forwards them encrypted over stunnel to the remote unix-style syslog daemon on the VPS. Since stunnel works only for TCP logging, the same rsyslog daemon on the intermediate dormarth.sanyalnet.lan Linux box serves as a UDP syslog sink for UDP-only clients, whose logs are forwarded encrypted over stunnel to the remote cloud VPS.

I use a similar mechanism to forward OpenVMS logs to the central syslog server running on the remote VPS with dormarth.sanyalnet.lan as the intermediary. The flow of log data is OpenVMS OPERATOR LOG > OpenVMS SYSLOGD.EXE > dormarth.sanyalnet.lan > stunnel encryption > sanyalnet-cloud-vps.freeddns.org > stunnel  decryption > rsyslog daemon.

You can learn more about how I have set this up in my blog entry "Secure Remote Logging to Central Log Server Using RSYSLOG on CentOS 6 / CentOS 7 and stunnel".


2. Install and Configure SYSLOGD.EXE on OpenVMS

I slightly-modified the OpenVMS SYSLOG.EXE program by Doug O'Neal from Homewood Academic Computing at Johns Hopkins University to support a configurable port number. This modified SYSLOG.EXE runs on my OpenVMS VAX system QCOCAL, forwarding OPERATOR.LOG entries at real-time to dormarth.sanyalnet.lan, which in turn stunnels them to the remote VPS.

OpenVMS Logging to Unix-Style Linux SYSLOG

The important thing to note for the purposes of this article is the OpenVMS logs are, therefore, recorded by two syslog daemons - one at dormarth.sanyalnet.lan inside the local LAN, and the other at the remote sanyalnet-cloud-vps.freeddns.org VPS.

OpenVMS VAX Unix-Style SYSLOG logging to Linux RSYSLOG on LAN and Secure Tunnel to WAN

I have described OpenVMS to Unix-Style SYSLOG logging in detail in my blog entry "OpenVMS Log Files Remote Logging to Unix/Linux SYSLOG Facility RSYSLOG".


3. Parsing OpenVMS Logs on Linux and Reporting to Blocklist.DE via FAIL2BAN

I wrote a bash shell script "openvms-telnet-spam-blocklist.sh" which I execute periodically (currently every 12 hours) from cron on the intermediate (LAN) syslog server dormarth.sanyalnet.lan. This bash shell script does the following things at each execution:

- Find Remote interactive breakin detection entries logged remotely from the OpenVMS VAX server
- For each breakin atempt found, extract the information into a single line that looks like a Linux PAM authentication failure log entry
- Use the Linux logger to log the PAM auth failure equivalent
- Create a summary report listing the attack sources and details, one per line

Since I already have fail2ban configured on dormarth.sanyalnet.lan, the PAM auth failure entries logged by the bash script are picked up by fail2ban and reported straightaway to blocklist.de. This is ultimately what results in the public recording of the intrusion attempts warning others of the attack source, like you see on the internet abuse IP databases.

The script, listed below, basically looks for log entries in /var/log/messages like this

Dec  4 05:07:01 sanyalnet-vax [OPCOM] %%%%%%%%%%%  OPCOM   4-DEC-2016 05:07:01.97  %%%%%%%%%%%
Dec  4 05:07:01 sanyalnet-vax [OPCOM] Message from user INTERnet on QCOCAL
Dec  4 05:07:01 sanyalnet-vax [OPCOM] TELNET Login from Host: hycoon.lnk.telstra.net Port: 39793
Dec  4 05:07:03 sanyalnet-vax [OPCOM] %%%%%%%%%%%  OPCOM   4-DEC-2016 05:07:03.64  %%%%%%%%%%%
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Message from user AUDIT$SERVER on QCOCAL
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Security alarm (SECURITY) and security audit (SECURITY) on QCOCAL, system id: 1574
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Auditable event:          Remote interactive breakin detection
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Event time:                4-DEC-2016 05:07:03.64
Dec  4 05:07:03 sanyalnet-vax [OPCOM] PID:                      000008F1
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Process name:             _TNA1021:
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Username:                 ROOT
Dec  4 05:07:03 sanyalnet-vax [OPCOM] ort: 39793
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Remote node id:           1174439829
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Remote node fullname:     hycoon.lnk.telstra.net
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Remote username:          TELNET_95870046
Dec  4 05:07:03 sanyalnet-vax [OPCOM] Status:                   %LOGIN-F-NOSUCHUSER, no such user

And writes entries like this into /var/log/secure:

Dec 12 05:59:32 dormarth sshd[000010F1]: pam_unix(sshd:auth): authentication failure; logname=SYS$MANAGER:OPERATOR.LOG uid=0 euid=0 tty=TELNET_50EE4387:49926 ruser=ADMIN rhost=80.238.67.135 user=TELNET_50EE4387 rport=49926 time="12-DEC-2016 05:29:23.07 UTC" event="80-238-67-135.internetia.net.pl;%LOGIN-F-NOSUCHUSER, no such user;Auditable event: Remote interactive breakin detection" reporter="Digital-VAXserver-3900-OpenVMS-7.3;DECnet:QCOCAL(1.550);inet:http://sanyalnet-openvms-vax.freeddns.org:82/"

The offending IP address is reported to blocklist.de by fail2ban, resulting in it being listed in public abuse information, like this:

OpenVMS VAX Telnet Brute-force Intrusion Report at www.abuseipdb.com

Here is the bash shell script:


and here is the /etc/cron.d/openvms-telnet-spam-blocklist-update cron file (remember, it must not have write permissions for group and world, i.e. permissions on this file should be -rw-r--r-- ). It is to be noted here that my logrotate.conf file limits the system log to two days of retention, thus automatically limiting the flagged IP addresses to 2 days. And also fail2ban does not re-ban IPs that are already banned, thus avoiding duplicate notifications to blocklist.de within 48-hour periods, which is the default duration of blocklist.de blacklisted IP addresses.

# /etc/cron.d/openvms-telnet-spam-blocklist-update
47 3,15 * * * root nice -n 19 ionice -c3 /root/security/openvms-telnet-spam-blocklist.sh >/var/log/openvms-telnet-spam-blocklist.sh.log 2>&1