Search This Blog

Tuesday, April 4, 2017

pfSense pfBlockerNG: The Ultimate List of IP and DNSBL Blocklists for Home Internet Security Firewall and Gateway

Supratim Sanyal's Blog: pfSense Dashboard
pfSense Dashboard

The amazing pfSense Community Edition forms the first of my three-layer home internet security firewall and gateway. I have a dual-WAN setup with subscriptions to both Verizon FiOS and Comcast Xfinity, with the LAN side feeding into a Sophos UTM 9 which is further protected by ClearOS.

I run pfSense in a virtual machine. However, there are excellent dedicated firewall routers with pfSense preinstalled available that you can simply plug in between your WAN and LAN, like this one (includes my Amazon affiliate link):



I am a huge fan of blocklists and over the years settled down to a functional set of IP and DNSBL blocklists used with the wonderful pfBlockerNG package on my installation of pfSense Community open-source router firewall.

I have completely disabled IPv6; all of the following blocklists are for IPv4, and for DNSBL, domain names.

IP BLOCKLISTS


For the IP blocklists, the top-level blocklist groups are Level-1, Level-2, Level-3, Level-4 and SANYALnet.

Supratim Sanyal's Blog: pfBlockerNG on pfSense - top level IP (IPv4) blocklist groups
pfBlockerNG on pfSense - top level IP (IPv4) blocklist groups



Level-1 IP Blocklist

Level-1 Blocklist sources
pfBlockerNG Level-1 IP Blocklist sources
Incoming as well as outgoing connections from / to blocklisted IPs are blocked for these highest risk IP addresses. Of particular concern in modern times are the command-and-control (CNC) botnets particularly infecting digital security and surveillance systems, cameras, routers, televisions, DVD players and all sorts of devices making up the Internet of Things (IoT). The Level-1 IP BL is updated every hour, and the group members are:

  • https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset
  • https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt
  • https://rules.emergingthreats.net/blockrules/compromised-ips.txt
  • https://rules.emergingthreats.net/fwrules/emerging-Block-IPs.txt
  • http://www.abuseat.org/iotcc.txt


Level-2 IP Blocklist

Supratim Sanyal's Blog: pfSense pfBlockerNG Level-2 IP Blocklist sources
pfBlockerNG Level-2 IP Blocklist sources
In addition to IoT C&C botnets, the other primary threat today is from Ransomware. I only have Firehol Level 2,  Ransomware Tracker IP blacklists from abuse.ch including CryptoWall, Locky, TeslaCrypt, TorrentLocker C&C and Payment, and Zeus tracker and ci badguys IP deny blocklists at my level 2, which is also configured to block all outgoing as well as incoming connections. Level 2 IP blocklists are updated every 2 hours.

  • https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level2.netset
  • http://cinsscore.com/list/ci-badguys.txt
  • https://ransomwaretracker.abuse.ch/downloads/CW_PS_IPBL.txt
  • https://ransomwaretracker.abuse.ch/downloads/LY_PS_IPBL.txt
  • https://ransomwaretracker.abuse.ch/downloads/RW_IPBL.txt
  • https://ransomwaretracker.abuse.ch/downloads/TC_PS_IPBL.txt
  • https://ransomwaretracker.abuse.ch/downloads/TL_C2_IPBL.txt
  • https://ransomwaretracker.abuse.ch/downloads/TL_PS_IPBL.txt
  • https://zeustracker.abuse.ch/blocklist.php?download=badips


Level-3 IP Blocklist

Supratim Sanyal's Blog: pfSense pfBlockerNG Level-3 IP BL Blocklist sources
pfBlockerNG Level-3 IP Blocklist sources

IP addresses in my level 3 blocklist are denied on the incoming side only, i.e. I allow connections initiated from inside my home LAN out to these IPs to go through. The level 3 IP blacklist addresses are updated every 4 hours. The sources are:

  • https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level3.netset
  • http://danger.rulez.sk/projects/bruteforceblocker/blist.php
  • http://www.openbl.org/lists/base_7days.txt
  • https://lists.blocklist.de/lists/all.txt
  • http://malc0de.com/bl/IP_Blacklist.txt
  • https://feodotracker.abuse.ch/blocklist/?download=ipblocklist




Level-4 IP Blocklist

Supratim Sanyal's Blog: pfSense pfBlockerNG Level-3 IP Blocklist sources
pfBlockerNG Level-4 IP Blocklist sources


There are only a couple of blacklist sources for my level 4, including Firehol Level 4, and Malware Domain List IP addresses the equivalent domains of which are also included in my list of DNSBL lists separately. Level 4 is configured to block inbound connections only and updated every 8 hours.
  • https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level4.netset
  • http://www.malwaredomainlist.com/hostslist/ip.txt



SANYALnet IP Blocklist

Supratim Sanyal's Blog: pfSense pfBlockerNG SANYALnet IP Blocklist sources
pfBlockerNG SANYALnet IP Blocklist sources

The SANYALnet group is a collection of blocklists I maintain myself based on the brute force attacks and intrusion attempts logged by my own servers. This group is updated every hour to minimize on-going attacks.
  • http://sanyalnet-cloud-vps.freeddns.org/blocklist.txt
  • http://sanyalnet-cloud-vps.freeddns.org/mirai-ips.txt
  • http://sanyalnet-cloud-vps2.freeddns.org/blocklist.txt
  • http://wbri.duckdns.org/blocklist.txt
  • http://yiradio.duckdns.org/blocklist.txt
  • http://glewlwyd.duckdns.org/blocklist.txt

pfBlockerNG DNSBL Feeds

Supratim Sanyal's Blog: pfBlockerNG DNSBL Feeds DNS Groups
pfBlockerNG DNSBL Feeds DNS Groups
In addition to IP blocklists, I also extensively use pfBlockerNG's domain name blocklisting feature with publicly available domain blocklists.

The DNSBL configuration redirects domain name lookups for blocked domains to my own  "httpd410server" DNS sinkhole.

I have grouped the DNSBL feeds into three groups.


Zero-day Threat Domain Blocklist Group

Supratim Sanyal's Blog: pfBlockerNG DNSBL Zero-Day Threat Domain Blocklist
pfBlockerNG DNSBL Zero-Day Threat Domain Blocklist
I use the OpenPhish blocklist to block out emerging zero-day phishing and spear-phishing domains. Following advice from the pfSense forum, I use the "FLEX" as the State to retrieve feeds over https in cases where the usual "ON" state fails to retrieve them citing a peculiar curl error "SSL certificate problem: unable to get local issuer certificate" on pfSense. The feeds in this group are updated every hour.

  • https://openphish.com/feed.txt



General Domain Blocklist Group

pfBlockerNG DNSBL General Domain Blocklist Group

This group contains a collection of malware, ransomware, adware, spyware, tracker and generally undesirable domain blocklists updated once every day. This includes advertising services, thus making my pfSense firewall an effective ad blocker for all devices on my entire home network.

I turned the Eladkarako and Immortal Long Lived Malware Domains blocklists off because they were too generic and were blocking too many websites used by folks in my home. If you wish, you can turn them on for a more secure DNSBL at the cost of filtering out some websites that are otherwise useful.

  • https://ransomwaretracker.abuse.ch/downloads/RW_DOMBL.txt
  • http://www.malware-domains.com/files/justdomains.zip
  • https://isc.sans.edu/feeds/suspiciousdomains_Low.txt
  • https://isc.sans.edu/feeds/suspiciousdomains_Medium.txt
  • https://isc.sans.edu/feeds/suspiciousdomains_High.txt
  • https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
  • http://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
  • http://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
  • http://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
  • http://raw.githubusercontent.com/quidsup/notrack/master/trackers.txt
  • Use with care: http://raw.githubusercontent.com/eladkarako/hosts.eladkarako.com/master/_raw__hosts.txt
  • Use with care: http://mirror1.malwaredomains.com/files/immortal_domains.txt



Hosts File Format Blocklists

Supratim Sanyal's Blog: pfBlockerNG DNSBL General hosts File Format Blocklist Group
pfBlockerNG DNSBL General hosts File Format Blocklist Group
This group contains another long list of advertising domains, malware, ransomware, adware, spyware, tracker and generally undesirable domain blocklists updated daily. I like to keep blocklists formatted like the /etc/hosts file in a separate group.

  • https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts
  • http://avant.it-mate.co.uk/dl/Tools/hpHosts/HOSTS.zip
  • https://adaway.org/hosts.txt
  • https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&startdate%5Bday%5D=&startdate%5Bmonth%5D=&startdate%5Byear%5D=&mimetype=plaintext
  • http://someonewhocares.org/hosts/hosts
  • http://sysctl.org/cameleon/hosts
  • http://winhelp2002.mvps.org/hosts.txt
  • http://www.malekal.com/HOSTS_filtre/HOSTS.txt
  • http://www.malwaredomainlist.com/hostslist/hosts.txt
  • https://zeustracker.abuse.ch/blocklist.php?download=hostfile
  • http://www.hostsfile.org/Downloads/hosts.txt
  • http://www.securemecca.com/Downloads/hosts.txt
  • http://hosts-file.net/exp.txt
  • http://hosts-file.net/ad_servers.txt
  • http://hosts-file.net/emd.txt
  • http://hosts-file.net/hjk.txt
  • http://hosts-file.net/fsa.txt
  • http://hosts-file.net/grm.txt
  • http://hosts-file.net/psh.txt
  • http://hosts-file.net/mmt.txt
  • http://hosts-file.net/hfs.txt
  • http://hosts-file.net/pha.txt
  • http://hosts-file.net/wrz.txt
  • http://raw.githubusercontent.com/michaeltrimm/hosts-blocking/master/_hosts.txt


pfBlockerNG DNSBL Custom Domain Whitelist

Supratim Sanyal's Blog: pfSense pfBlockerNG DNSBL Custom Domain Whitelist
pfSense pfBlockerNG DNSBL Custom Domain Whitelist


Sometimes a domain blocklist included in pfSense pfBlockerNG DNSBL configuration will block URLs that you find useful and want to visit. Instead of digging through the logs to figure out which list is blocking your desired domain and disabling the entire list, you can simply add the domains that should not be blocked in the nifty Custom Domain Whitelist feature included as part of the DNSBL configuration.

Consolidated IP and DNSBL Blocklists

I make consolidated IP address and Domain Name blocklists available for free public use from my VPS at the following links; feel free to use them.

pfSense pfBlockerNG in Action

With the pfBlockerNG setup for IP and DNS Blocklists described above, I do see domains and IPs blocked all the time - here is a typical example of pfBlockerNG's "Alert" screen that shows the last 25 IP addresses and domains blocked at the time of writing:

Supratim Sanyal's Blog: pfSense pfBlockerNG Active Blocked IP Addresses and Domains
pfSense pfBlockerNG Active Blocked IP Addresses and Domains


A pfBlockerNG force reload log looks like this:




Hope you find this useful and please share the IP and domain blocklists you have found and use in comments below.




26 comments:

  1. Fantastic stuff man. Great work! This is really helpful

    ReplyDelete
    Replies
    1. Thank you Christopher, drop a line please if you find other interesting lists that I should also be using.

      Delete
  2. impressive list, i have a decent router that I will be testing this on,if it bothers me by excessive blocking or not I will keep you informed... anyway, thanx for the loads of work, its pretty hard accumulating such a database as I was building mine for 2 weeks but for now i scrapped it

    ReplyDelete
  3. The list is a too aggressive, it blocks too much that Windows it self sufferes some issues. I didnt use these two sources "Eladkarako or Immortal Long Lived Malware Domains".

    Nvidia experience fails upon searching for local games and I keep getting DistributedCOM "event id 10016" in the event viewer. at first it happened to me while I was building a similar list & had to format my PC twice and eventually I found that excessive list some of will do block essential services the PC relies on to function.There are multiple more even ids related to these lists, but the more prominent that affects the system is 10016.


    just a note, I'm trying to isolate the issue but its quit hard.

    ReplyDelete
  4. sorry for the headache, I found the culprit i think...

    https://raw.githubusercontent.com/quidsup/notrack/master/trackers.txt

    ReplyDelete
    Replies
    1. Good work, thanks for sharing. BTW, I also sometimes add the specific domains I need unblocked in the DNSBL white-list, which IMHO is preferable to disabling entire blocklists.

      Delete
  5. Be careful guys, the firehol level 1 just made my system unaccessible, internet completely down as it has my local IP address range in it. 192.168.0.0/16. Absolute unhappy, had to find a monitor to restore to last backup.

    ReplyDelete
    Replies
    1. Sorry to hear that tejal, glad to know you could recover. Just of curiosity, is your pfSense behind a NAT from your ISP? I ask because it may be your WAN address is a private address, which could happen if your pfSense is not talking to the internet directly.

      Delete
    2. No worries mate, the pfsense is behind my ISP. I was not able to manage my pfsense so basically locked myself out of web access.
      Thanks anyway, good write up.

      Delete
  6. Thank You very much Supratim Sanyal for sharing your list. I'm new to PfBlockerNg so it was a great help. All I had to do is whitelist a few names and I'm up and running. The only list that would not update was this one:
    https://isc.sans.edu/feeds/suspiciousdomains_Medium.txt

    Thanks, appreciate your generosity

    ReplyDelete
    Replies
    1. You are very welcome Blupie. Yes, some lists do get unreachable sporadically, but usually they come back in a few hours or days. Some providers also throttle access, i.e. if you fetch those too fast they will lock you out for a while. BTW https://isc.sans.edu/feeds/suspiciousdomains_Medium.txt is actually a mirror of http://www.dshield.org/feeds/suspiciousdomains_Medium.txt.

      Delete
  7. Supratim, any advise on how to block porn sites please. DNSBL does not seem to be doing so, can you help?

    ReplyDelete
  8. The simplest way is to point DNS servers in your router (or pfSense) to a free service like Norton ConnectSafe (https://dns.norton.com/configureRouter.html) or OpenDNS FamilyShield (https://support.opendns.com/hc/en-us/articles/228006487-FamilyShield-Router-Configuration-Instructions). The DNS server IP addresses for these two options are:
    Norton ConnectSafe Security+Pornography: 199.85.126.30, 199.85.127.30
    OpenDNS FamilyShield: 208.67.222.123, 208.67.220.123

    Personally I use a (free) OpenDNS account which allows much finer control over categories, block and whitelists; the ddclient script on a linux box in my LAN keeps my volatile residential DHCP IP address updated at OpenDNS (see https://support.opendns.com/hc/en-us/articles/227987727-Linux-IP-Updater-for-Dynamic-Networks). Also I run pfSense as a bridge into a Sophos UTM 9 security gateway that includes blocking porn among numerous awesome features; the Sophos UTM 9 is ahead of a ClearOS gateway that adds weighted phrase limit capability and more awesome things before distributing internet to local subnets in our house.

    Hope this helps, and good luck.

    ReplyDelete
  9. Thanks so much for your reply, another question if you dont mind. still i have implemented my pfsense as per your great write-up, facetime has stopped working for my users, any idea what i need to do to allow this, i cannot locate any log in the firewall at all.

    ReplyDelete
    Replies
    1. Can you try this and let me know if this resolves the facetime issue?
      - In DNSBL tab's "Custom Domain Whitelist" section, add the following two lines and save:
      apple.com
      icloud.com
      - In Update tab, check the "Reload" radio button for "Select 'Force' option" and click on the "Run" button, let the reload complete.

      For logs, make sure Logging in Enabled in the DNSBL tab, and also check the 'Global Logging' Option in the General Tab which overrides this setting as the description indicates.

      Please respond back with results of these steps. Thanks for not giving up on pfSense and patiently trying to work around problems!

      Delete
    2. Thanks mate, all done.

      Delete
  10. Hi Supratim,

    A Wonderful post, some great information in here.

    As someone else mentioned, the 192.168.0.0/16 is blocked by the firehol_level1.netset.

    This is a problem for me, so is there anyway i could use this list but ensure that the 192.168.0.0/16 is not affected?

    Any advice most welcome.

    And again thanks for such an informative post.

    ReplyDelete
  11. Hi Matty,

    I have not faced your situation (private IP on WAN), but suspect adding 192.168.0.0/16 to the pfBlockerNGSuppress alias should work. The pfBlockerNGSuppress alias should have been created by the package (check under Firewall -> Aliases) for the very purpose of white-listing some IPs.

    Please drop a line about using pfBlockerNGSuppress works if you will; I am curious.

    Thanks for your kind words.

    ReplyDelete
  12. Hi Subratim,

    This fixed my issue, although i did find it strange that by default the pfBlockerNGSuppress should remove RFC1918 addresses from all lists.

    After manually adding 192.168.10.0/24 , 192.168.20.0/24, and 192.168.30.0/24 Everything worked.

    I found that it was the deny outgoing that triggered the issues, maybe because i am 1:1 Natting many public IPs to private ips on those ranges?

    Not sure. But heh thanks to you it all works now :)

    Keep up the great work

    Mat.

    ReplyDelete
  13. Thank you for sharing this very useful information. I've attempted to used the white list technique you suggest. However, I'm still unable to access a few of the websites or their services. I've had to tweak the rule order in the general tab (PF_B pass/match, pfSense pass/match, PF_B block, pfSense block) in order to get everything to "sort of" work. Do you think that this rule selection weakens the strength of the overall desired protection? Additionally, I've had to completely disable the "Easy list" because I think that the ad block interferes when trying to play certain news videos. Finally, I've gotten several update download errors after CRON jobs. Any suggestions would be greatly appreciated.

    ReplyDelete
  14. Personally, my biggest worry is ransomware, followed by IoT targeted worms (think Mirai). As long as these two categories are carefully protected against, I am less worried about old-school virus/malware because our staple BitDefender and friends are sufficient protection for our computing devices. You are good if your modified rules do not let through the ransomware and IoT worms IMHO.

    Ads are at most an irritation, I am more of the "let me see if I can block them" type than an ad-hater. Yes, I have the same experience of broken video streaming, especially on the "interesting" video sites that claim no responsibility since they do not "host" the videos on their servers.

    Typically in the first few days of a new installation, rapid update requests cause a few of these list providers to lock you out, give it a couple of weeks and stop meddling, you should have all downloads working. I see a few fail here and there sporadically but usually they come back in a few hours/days/weeks.

    In short, if too many things are getting blocked, downsizing to ransomware and IoT malware blocking should be enough for a non-paranoid user.

    Hope this is helpful ?

    ReplyDelete
  15. hello, ive used the consolidated urls and i guess it failed on the dnsbl...below is the result

    ===[ DNSBL Process ]================================================
    Missing DNSBL stats and/or Unbound DNSBL conf file - Rebuilding

    [ dnsblsupratim ] Downloading update .. 200 OK
    No Domains Found

    Clearing all DNSBL Feeds... completed
    Validating database... completed [ 07/01/17 13:22:01 ]
    Reloading Unbound.... completed
    DNSBL update [ 0 | PASSED ]... completed

    ReplyDelete
    Replies
    1. Hi perhaps you are not running the latest pfSense firmware on your hardware firewall products? This bug seems to have been fixed a while ago - https://forum.pfsense.org/index.php?topic=86212.1005 - I have never faced this issue.

      Delete
  16. but was it right? i just pasted the consolidated dns blocklist url on the dnsbl feeds. iam using the 2.3.4

    ReplyDelete
    Replies
    1. I apologize, my bad; the consolidated domain name blocklist at http://sanyalnet-cloud-vps.freeddns.org/adblocklist.conf is geared towards the dnsmasq daemon, not pfBlockerNG - I updated the post to make that clear. The consolidate IP blocklist at http://sanyalnet-cloud-vps.freeddns.org/blocklist.txt should work with pfBlockerNG, though.

      Delete
    2. BTW if you are looking for one DNSBL here is a promising one: https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts
      This one includes adware + malware + fakenews + gambling + porn; more information at https://github.com/StevenBlack/hosts

      Delete

Recommended Products from Amazon