Here is a quick and easy way to block internet ads in your home. This also blocks various tracking services that collect information about your browsing (claiming to be to fine-tune ads for you) and makes internet browsing cleaner and faster. Everything I talk about in this post can be downloaded from a link at the bottom.
I use a ClearOS Community Edition server as the innermost of my three-layer onion internet security gateway (as in layers of an onion, nothing to do with the TOR project in this post, although I will describe my TOR gateway in another post later). Among the many excellent features of ClearOS, the fact that it is built on CentOS makes it instantly familiar to those of us at ease with CentOS who want to customize everything.
ClearOS comes with dnsmasq to serve DHCP and DNS requests. It provides IP addresses to DHCP hosts and forwards DNS requests to a higher-level peer DNS server. In my setup, I have three local area networks served by ClearOS: a dedicated 10.100.0.0 LAN for my hobby projects in the basement, a 10.200.0.0 "rest of the house" LAN and a host-only 10.42.0.0 LAN to provide internet to the DELL PowerEdge blade server itself that hosts all the virtual machines that make up the internet security onion gateway, and other VMs running on it.
As a side note, I named my internet safety gateway system "DORMARTH" after the great hound (also called "Dormarch") from Welsh mythology whose assistance to warriors chasing down the enemy and facilitating the passage of the dead to the other side is legendary. Most of my server hostnames have "dormarth" somewhere as a result.
Coming back to dnsmasq, before adding the ad-blocking feature, I checked on what was installed with CentOS and found the following primary configuration file.
Also inspecting /var/log/messages for logs from dnsmasq reveals everything is working as expected, which is also confirmed by looking at the log file /var/log/adblocker-dnsmasq.log:
Shutting down dnsmasq: [ OK ]
Starting dnsmasq: [ OK ]
Mon Jul 25 03:26:15 UTC 2016
/etc/dnsmasq.d/adblocklist.conf
---
address=/.../10.42.2.1
address=/.../10.42.2.1
I use a ClearOS Community Edition server as the innermost of my three-layer onion internet security gateway (as in layers of an onion, nothing to do with the TOR project in this post, although I will describe my TOR gateway in another post later). Among the many excellent features of ClearOS, the fact that it is built on CentOS makes it instantly familiar to those of us at ease with CentOS who want to customize everything.
 |
| Writing the Ad Server Blacklist script for Ad Blocker using dansmasq on ClearOS running on CentOS |
ClearOS comes with dnsmasq to serve DHCP and DNS requests. It provides IP addresses to DHCP hosts and forwards DNS requests to a higher-level peer DNS server. In my setup, I have three local area networks served by ClearOS: a dedicated 10.100.0.0 LAN for my hobby projects in the basement, a 10.200.0.0 "rest of the house" LAN and a host-only 10.42.0.0 LAN to provide internet to the DELL PowerEdge blade server itself that hosts all the virtual machines that make up the internet security onion gateway, and other VMs running on it.
As a side note, I named my internet safety gateway system "DORMARTH" after the great hound (also called "Dormarch") from Welsh mythology whose assistance to warriors chasing down the enemy and facilitating the passage of the dead to the other side is legendary. Most of my server hostnames have "dormarth" somewhere as a result.
Coming back to dnsmasq, before adding the ad-blocking feature, I checked on what was installed with CentOS and found the following primary configuration file.
The line "conf-dir=/etc/dnsmasq.d" was immediately interesting, telling me there is a whole directory that I can drop configuration files in for dnsmasq to pick up, without having to change anything in the main configuration file /etc/dnsmasq.conf at all.
Looking at the contents of /etc/dnsmasq.d, I found a single file /etc/dnsmasq.d/dhcp.conf which tells dnsmasq what interface to serve DHCP requests on, what IP address ranges to use for DHCP clients, how long DHCP sessions are valid before requiring renewal, etc.
The only other file I checked on was the resolver configuration file /etc/resolv-peerdns.conf referenced by /etc/dnsmasq.conf. This has one line, basically defining the peer DNS host to forward requests to. In my case, the ClearOS server forwards DNS requests to the 2nd of my three-layer onion security gateway (a Sophos UTM server):
The only other file I checked on was the resolver configuration file /etc/resolv-peerdns.conf referenced by /etc/dnsmasq.conf. This has one line, basically defining the peer DNS host to forward requests to. In my case, the ClearOS server forwards DNS requests to the 2nd of my three-layer onion security gateway (a Sophos UTM server):
[root@anubis-clearos ~]# cat /etc/resolv-peerdns.conf
nameserver 10.42.1.1
Now, we have enough information to add ad blocking to the dnsmasq configuration so that it stops known advertising websites from delivering ads to its DHCP and static clients.
I decided to use the great lists of advertising servers and URLs maintained by Yoyo Internet Services. The types of lists we can download from Yoyo can be chosen using selection drop-down lists and checkboxes at their "Ad blocking with ad server hostname and IP addresses" web page.
Among the numerous types of ad server list file formats, the two that I am interested in are the lists designed for dnsmasq using "address=" lines, and dnsmasq using "server=" lines. Selecting the corresponding items from the drop-down lists, choosing "no links back to this page" and checking the "view list as plain text" gives us the following two lists that we can simply dump into directory /etc/dnsmasq.d via a cron job and restart dnsmasq.
I wrote a basic shell script to download these ad server lists and drop them into /etc/dnsmasq.d, and saved it into /root/adblocker/adblocker-dnsmasq.sh. Here is adblocker-dnsmasq.sh
The original adserver list file adblocklist.conf downloaded by wget contains the IP address of 127.0.0.1 that all the domains and URLs resolve to. However, I run a little custom HTTP server that responds with a HTTP code of 410 (GONE) to all requests while logging the request. The purpose of this minimal server (I have described it in detail with the source code here) is to log all advertising server requests that dnsmasq blocks based on the block-list. To do that, the 127.0.0.1 addresses in the block-list need to be changed to the IP 10.42.2.1 of the server running the small HTTP 502 responder. This is accomplished by the line "sed -i 's/127.0.0.1/10.42.2.1/g' /etc/dnsmasq.d/adblocklist.conf" in the above script. I will write separately about the little server, but there is an example of what it produces later in this post.
Another subtlety of the above script is that wget preserves the time of the original file it gets from the remote site. The folks at yoyo update the lists last on July 21, and today is July 25. When I ran the script today, both adblocklist.conf and adblockserverlist.conf were fetched successfully by wget with July 21 dates on the files, but the sed command ran on adblocklist.conf, thus changing the file time of adblocklist.conf to the current. adblockserverlist.conf is not manipulated at all and the original file date is preserved.
The final piece is adding the adblocker-dnsmasq.sh script to cron so that the lists get updated automagically. I chose a weekly update schedule because advertising servers do not pop up or disappear very frequently. For the cron job, I created the file update-adblocker-dnsmasq in /etc/cron.d/ containing a directive to crond to run the ad server list updater on a schedule:
To test, I ran the updater command line directly and verified that the ad server block lists were indeed making it to the correct directory /etc/dnsmasq.d:
[root@anubis-clearos adblocker]# ls -l /etc/dnsmasq.d
total 148
-rw-r--r-- 1 root root 84187 Jul 25 03:26 adblocklist.conf
-rw-r--r-- 1 root root 60357 Jul 21 09:38 adblockserverlist.conf
-rw-r--r-- 1 root root 523 Aug 2 2015 dhcp.conf
[root@anubis-clearos adblocker]#
[root@anubis-clearos adblocker]# date
Mon Jul 25 03:30:31 UTC 2016
[root@anubis-clearos adblocker]# ls -l /etc/dnsmasq.d
total 148
-rw-r--r-- 1 root root 84187 Jul 25 03:26 adblocklist.conf
-rw-r--r-- 1 root root 60357 Jul 21 09:38 adblockserverlist.conf
-rw-r--r-- 1 root root 523 Aug 2 2015 dhcp.conf
[root@anubis-clearos adblocker]# ls -l /var/log/adblocker-dnsmasq.log
-rw-r--r-- 1 root root 144817 Jul 25 03:26 /var/log/adblocker-dnsmasq.log
[root@anubis-clearos adblocker]# ls -l /tmp/adblock*
-rw-r--r-- 1 root root 84187 Jul 25 03:07 /tmp/adblocklist.conf.bak
-rw-r--r-- 1 root root 60357 Jul 25 03:14 /tmp/adblockserverlist.conf.bak
I decided to use the great lists of advertising servers and URLs maintained by Yoyo Internet Services. The types of lists we can download from Yoyo can be chosen using selection drop-down lists and checkboxes at their "Ad blocking with ad server hostname and IP addresses" web page.
Among the numerous types of ad server list file formats, the two that I am interested in are the lists designed for dnsmasq using "address=" lines, and dnsmasq using "server=" lines. Selecting the corresponding items from the drop-down lists, choosing "no links back to this page" and checking the "view list as plain text" gives us the following two lists that we can simply dump into directory /etc/dnsmasq.d via a cron job and restart dnsmasq.
- Ad Server Block List configuration file for dnsmasq using "address="
- Ad Server Block List configuration file for dnsmasq using "server="
I wrote a basic shell script to download these ad server lists and drop them into /etc/dnsmasq.d, and saved it into /root/adblocker/adblocker-dnsmasq.sh. Here is adblocker-dnsmasq.sh
The original adserver list file adblocklist.conf downloaded by wget contains the IP address of 127.0.0.1 that all the domains and URLs resolve to. However, I run a little custom HTTP server that responds with a HTTP code of 410 (GONE) to all requests while logging the request. The purpose of this minimal server (I have described it in detail with the source code here) is to log all advertising server requests that dnsmasq blocks based on the block-list. To do that, the 127.0.0.1 addresses in the block-list need to be changed to the IP 10.42.2.1 of the server running the small HTTP 502 responder. This is accomplished by the line "sed -i 's/127.0.0.1/10.42.2.1/g' /etc/dnsmasq.d/adblocklist.conf" in the above script. I will write separately about the little server, but there is an example of what it produces later in this post.
Another subtlety of the above script is that wget preserves the time of the original file it gets from the remote site. The folks at yoyo update the lists last on July 21, and today is July 25. When I ran the script today, both adblocklist.conf and adblockserverlist.conf were fetched successfully by wget with July 21 dates on the files, but the sed command ran on adblocklist.conf, thus changing the file time of adblocklist.conf to the current. adblockserverlist.conf is not manipulated at all and the original file date is preserved.
The final piece is adding the adblocker-dnsmasq.sh script to cron so that the lists get updated automagically. I chose a weekly update schedule because advertising servers do not pop up or disappear very frequently. For the cron job, I created the file update-adblocker-dnsmasq in /etc/cron.d/ containing a directive to crond to run the ad server list updater on a schedule:
To test, I ran the updater command line directly and verified that the ad server block lists were indeed making it to the correct directory /etc/dnsmasq.d:
[root@anubis-clearos adblocker]# ls -l /etc/dnsmasq.d
total 148
-rw-r--r-- 1 root root 84187 Jul 25 03:26 adblocklist.conf
-rw-r--r-- 1 root root 60357 Jul 21 09:38 adblockserverlist.conf
-rw-r--r-- 1 root root 523 Aug 2 2015 dhcp.conf
[root@anubis-clearos adblocker]#
[root@anubis-clearos adblocker]# date
Mon Jul 25 03:30:31 UTC 2016
[root@anubis-clearos adblocker]# ls -l /etc/dnsmasq.d
total 148
-rw-r--r-- 1 root root 84187 Jul 25 03:26 adblocklist.conf
-rw-r--r-- 1 root root 60357 Jul 21 09:38 adblockserverlist.conf
-rw-r--r-- 1 root root 523 Aug 2 2015 dhcp.conf
[root@anubis-clearos adblocker]# ls -l /var/log/adblocker-dnsmasq.log
-rw-r--r-- 1 root root 144817 Jul 25 03:26 /var/log/adblocker-dnsmasq.log
[root@anubis-clearos adblocker]# ls -l /tmp/adblock*
-rw-r--r-- 1 root root 84187 Jul 25 03:07 /tmp/adblocklist.conf.bak
-rw-r--r-- 1 root root 60357 Jul 25 03:14 /tmp/adblockserverlist.conf.bak
Also inspecting /var/log/messages for logs from dnsmasq reveals everything is working as expected, which is also confirmed by looking at the log file /var/log/adblocker-dnsmasq.log:
Shutting down dnsmasq: [ OK ]
Starting dnsmasq: [ OK ]
Mon Jul 25 03:26:15 UTC 2016
/etc/dnsmasq.d/adblocklist.conf
---
address=/.../10.42.2.1
address=/.../10.42.2.1
address=/.../10.42.2.1
...
...
address=/.../10.42.2.1
address=/.../10.42.2.1
---
/etc/dnsmasq.d/adblockserverlist.conf:
---
server=/.../
server=/.../
server=/.../
---
dnsmasq (pid 11431) is running...
That is all folks.
Now log into any computer or mobile device served by this ad-filtering installation of dnsmasq and browse some ad-heavy web-sites. The sites will load faster and cleaner without the advertising, not to mention your browsing security increases since there is far less amount of third-party websites tracking your behavior. Here is a sample of the ads blocked by dnsmasq as logged by my little 502 server to which the blocked domains and URLs are redirected:
No comments:
Post a Comment
"SEO" link builders: move on, your spam link will not get posted.
Note: Only a member of this blog may post a comment.