net/adblock/files/README.md

   1 # dns based ad/abuse domain blocking
   2
   3 ## Description
   4 A lot of people already use adblocker plugins within their desktop browsers, but what if you are using your (smart) phone, tablet, watch or any other wlan gadget...getting rid of annoying ads, trackers and other abuse sites (like facebook ;-) is simple: block them with your router. When the dns server on your router receives dns requests, you will sort out queries that ask for the resource records of ad servers and return a simple 'NXDOMAIN'. This is nothing but **N**on-e**X**istent Internet or Intranet domain name, if domain name is unable to resolved using the dns server, a condition called the 'NXDOMAIN' occurred.
   5
   6 ## Main Features
   7 * support of the following domain block list sources (free for private usage, for commercial use please check their individual licenses):
   8     * [adaway](https://adaway.org)
   9     * => infrequent updates, approx. 400 entries (enabled by default)
  10     * [adguard](https://adguard.com)
  11     * => numerous updates on the same day, approx. 12.000 entries
  12     * [blacklist]()
  13     * => static local blacklist, located by default in '/etc/adblock/adblock.blacklist'
  14     * [disconnect](https://disconnect.me)
  15     * => numerous updates on the same day, approx. 6.500 entries (enabled by default)
  16     * [dshield](http://dshield.org)
  17     * => daily updates, approx. 4.500 entries
  18     * [feodotracker](https://feodotracker.abuse.ch)
  19     * => daily updates, approx. 0-10 entries
  20     * [hphosts](https://hosts-file.net)
  21     * => monthly updates, approx. 50.000 entries
  22     * [malwaredomains](http://malwaredomains.com)
  23     * => daily updates, approx. 16.000 entries
  24     * [malwaredomainlist](http://www.malwaredomainlist.com)
  25     * => daily updates, approx. 1.500 entries
  26     * [openphish](https://openphish.com)
  27     * => numerous updates on the same day, approx. 1.800 entries
  28     * [ransomware tracker](https://ransomwaretracker.abuse.ch)
  29     * => daily updates, approx. 150 entries
  30     * [reg_cn](https://easylist-downloads.adblockplus.org/easylistchina+easylist.txt)
  31     * => regional blocklist for China, daily updates, approx. 1.600 entries
  32     * [reg_pl](http://adblocklist.org)
  33     * => regional blocklist for Poland, daily updates, approx. 50 entries
  34     * [reg_ro](https://easylist-downloads.adblockplus.org/rolist+easylist.txt)
  35     * => regional blocklist for Romania, weekly updates, approx. 600 entries
  36     * [reg_ru](https://code.google.com/p/ruadlist)
  37     * => regional blocklist for Russia, weekly updates, approx. 2.000 entries
  38     * [securemecca](http://www.securemecca.com)
  39     * => infrequent updates, approx. 25.000 entries
  40     * [shallalist](http://www.shallalist.de) (categories "adv" "costtraps" "spyware" "tracker" "warez" enabled by default)
  41     * => daily updates, approx. 32.000 entries (a short description of all shallalist categories can be found [online](http://www.shallalist.de/categories.html))
  42     * [spam404](http://www.spam404.com)
  43     * => infrequent updates, approx. 5.000 entries
  44     * [sysctl/cameleon](http://sysctl.org/cameleon)
  45     * => weekly updates, approx. 21.000 entries
  46     * [whocares](http://someonewhocares.org)
  47     * => weekly updates, approx. 12.000 entries
  48     * [winhelp](http://winhelp2002.mvps.org)
  49     * => infrequent updates, approx. 15.000 entries
  50     * [winspy](https://github.com/crazy-max/WindowsSpyBlocker)
  51     * => infrequent updates, approx. 120 entries
  52     * [yoyo](http://pgl.yoyo.org/adservers)
  53     * => weekly updates, approx. 2.500 entries (enabled by default)
  54     * [zeus tracker](https://zeustracker.abuse.ch)
  55     * => daily updates, approx. 440 entries
  56 * zero-conf like automatic installation & setup, usually no manual changes needed
  57 * simple but yet powerful adblock engine: adblock does not use error prone external iptables rulesets, http pixel server instances and things like that
  58 * automatically selects dnsmasq or unbound as dns backend
  59 * automatically selects uclient-fetch or wget as download utility (other tools like curl or aria2c are supported as well)
  60 * support http only mode (without installed ssl library) for all non-SSL blocklist sources
  61 * automatically supports a wide range of router modes, even AP modes are supported
  62 * full IPv4 and IPv6 support
  63 * supports tld compression (top level domain compression), this feature removes thousands of needless host entries from the block lists and lowers the memory footprint for the dns backends
  64 * each block list source will be updated and processed separately
  65 * block list source parsing by fast & flexible regex rulesets
  66 * overall duplicate removal in separate block lists
  67 * additional whitelist for manual overrides, located by default in /etc/adblock/adblock.whitelist
  68 * quality checks during block list update to ensure a reliable dns backend service
  69 * minimal status & error logging to syslog, enable debug logging to receive more output
  70 * procd based init system support (start/stop/restart/reload/suspend/resume/query/status)
  71 * procd based hotplug support, the adblock start will be solely triggered by network interface triggers
  72 * suspend & resume adblock actions temporarily without block list reloading
  73 * runtime information available via LuCI & via 'status' init command
  74 * query function to quickly identify blocked (sub-)domains, e.g. for whitelisting
  75 * optional: force dns requests to local resolver
  76 * optional: force overall sort / duplicate removal for low memory devices (handle with care!)
  77 * optional: 'manual mode' to re-use blocklist backups during startup, get fresh lists only via manual reload or restart action
  78 * optional: automatic block list backup & restore, they will be used in case of download errors or during startup in manual mode
  79 * optional: add new adblock sources on your own via uci config
  80
  81 ## Prerequisites
  82 * [LEDE project](https://www.lede-project.org), tested with latest stable release (LEDE 17.01) and with current LEDE snapshot
  83 * a usual setup with an enabled dns backend at minimum - dump AP modes without a working dns backend are _not_ supported
  84 * a download utility:
  85     * to support all blocklist sources a full version (with ssl support) of 'wget', 'uclient-fetch' with one of the 'libustream-*' ssl libraries, 'aria2c' or 'curl' is required
  86     * for limited devices with real memory constraints, adblock provides also a plain http option and supports wget-nossl and uclient-fetch (without libustream-ssl), too
  87     * for more configuration options see examples below
  88
  89 ## LEDE trunk Installation & Usage
  90 * install 'adblock' (_opkg install adblock_) and that's it - the adblock start will be automatically triggered by procd interface trigger
  91 * control the adblock service manually with _/etc/init.d/adblock_ start/stop/restart/reload/suspend/resume/status or use the LuCI frontend
  92 * enable/disable your favored block list sources in _/etc/config/adblock_ - 'adaway', 'disconnect' and 'yoyo' are enabled by default
  93
  94 ## LuCI adblock companion package
  95 * for easy management of the various block list sources and all other adblock options you can also use a nice & efficient LuCI frontend
  96 * install 'luci-app-adblock' (_opkg install luci-app-adblock_)
  97 * the application is located in LuCI under 'Services' menu
  98
  99 ## Tweaks
 100 * **runtime information:** the adblock status is available via _/etc/init.d/adblock status_ (see example below)
 101 * **debug logging:** for script debugging please set the config option 'adb\_debug' to '1' and check the runtime output with _logread -e "adblock"_
 102 * **storage expansion:** to process and store all block list sources at once it might helpful to enlarge your temp directory with a swap partition => see [openwrt wiki](https://wiki.openwrt.org/doc/uci/fstab) for further details
 103 * **add white- / blacklist entries:** add domain white- or blacklist entries to always-allow or -deny certain (sub) domains, by default both lists are empty and located in _/etc/adblock_. Please add one domain per line - ip addresses, wildcards & regex are _not_ allowed (see example below)
 104 * **backup & restore block lists:** enable this feature, to restore automatically the latest compressed backup of your block lists in case of any processing error (e.g. a single block list source is not available during update). Please use an (external) solid partition and _not_ your volatile router temp directory for this
 105 * **scheduled list updates:** for a scheduled call of the adblock service add an appropriate crontab entry (see example below)
 106 * **restrict procd interface trigger:** restrict the procd interface trigger to a (list of) certain interface(s) (default: wan). To disable it at all, remove all entries
 107 * **suspend & resume adblocking:** to quickly switch the adblock service 'on' or 'off', simply use _/etc/init.d/adblock [suspend|resume]_
 108 * **domain query:** to query the active block lists for a specific domain, please run _/etc/init.d/adblock query `<DOMAIN>`_ (see example below)
 109 * **add new list sources:** you could add new block list sources on your own via uci config, all you need is a source url and an awk one-liner (see example below)
 110 * **disable active dns probing in windows 10:** to prevent a yellow exclamation mark on your internet connection icon (which wrongly means connected, but no internet), please change the following registry key/value from "1" to "0" _HKLM\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet\EnableActiveProbing_
 111
 112 ## Further adblock config options
 113 * usually the pre-configured adblock setup works quite well and no manual config overrides are needed, all listed options apply to the 'global' config section:
 114     * adb\_enabled => main switch to enable/disable adblock service (default: '1', enabled)
 115     * adb\_debug => enable/disable adblock debug output (default: '0', disabled)
 116     * adb\_iface => set the procd interface trigger to a (list of) lan / wan interface(s) (default: 'wan')
 117     * adb\_fetch => full path to a different download utility, see example below (default: not set, use wget)
 118     * adb\_fetchparm => options for the download utility, see example below (default: not set, use wget options)
 119     * adb\_triggerdelay => additional trigger delay in seconds before adblock processing starts (default: '2')
 120     * adb\_forcedns => force dns requests to local resolver (default: '0', disabled)
 121     * adb\_forcesrt => force overall sort on low memory devices with less than 64 MB RAM (default: '0', disabled)
 122     * adb\_manmode => do not automatically update blocklists during startup, use blocklist backups instead (default: '0', disabled)
 123
 124 ## Examples
 125 **change default dns backend to 'unbound':**
 126 <pre><code>
 127 Adblock detects the presence of an active unbound dns backend and the block lists will be automatically pulled in by unbound.
 128 The adblock script deposits the sorted and filtered block lists in '/var/lib/unbound' where unbound can find them in its jail.
 129 If you use manual configuration for unbound, then just include the following line in your 'server:' clause:
 130
 131   include: "/var/lib/unbound/adb_list.*"
 132 </code></pre>
 133
 134 **configuration for different download utilities:**
 135 <pre><code>
 136 wget (default):
 137   option adb_fetch="/usr/bin/wget"
 138   option adb_fetchparm="--quiet --no-cache --no-cookies --max-redirect=0 --timeout=10 --no-check-certificate -O"
 139
 140 aria2c:
 141   option adb_fetch '/usr/bin/aria2c'
 142   option adb_fetchparm '-q --timeout=10 --allow-overwrite=true --auto-file-renaming=false --check-certificate=false -o'
 143
 144 uclient-fetch:
 145   option adb_fetch '/bin/uclient-fetch'
 146   option adb_fetchparm '-q --timeout=10 --no-check-certificate -O'
 147
 148 curl:
 149   option adb_fetch '/usr/bin/curl'
 150   option adb_fetchparm '-s --connect-timeout 10 --insecure -o'
 151 </code></pre>
 152
 153 **receive adblock runtime information:**
 154 <pre><code>
 155 root@blackhole:~# /etc/init.d/adblock status
 156 ::: adblock runtime information
 157  status          : active
 158  adblock_version : 2.6.0
 159  blocked_domains : 113711
 160  fetch_info      : wget (built-in)
 161  dns_backend     : dnsmasq
 162  last_rundate    : 12.04.2017 13:08:26
 163  system          : LEDE Reboot SNAPSHOT r3900-399d5cf532
 164 </code></pre>
 165
 166 **cronjob for a regular block list update (/etc/crontabs/root):**
 167 <pre><code>
 168 0 06 * * *    /etc/init.d/adblock reload
 169 </code></pre>
 170
 171 **blacklist entry (/etc/adblock/adblock.blacklist):**
 172 <pre><code>
 173 ads.example.com
 174
 175 This entry blocks the following (sub)domains:
 176   http://ads.example.com/foo.gif
 177   http://server1.ads.example.com/foo.gif
 178   https://ads.example.com:8000/
 179
 180 This entry does not block:
 181   http://ads.example.com.ua/foo.gif
 182   http://example.com/
 183 </code></pre>
 184
 185 **whitelist entry (/etc/adblock/adblock.whitelist):**
 186 <pre><code>
 187 here.com
 188
 189 This entry removes the following (sub)domains from the block lists:
 190   maps.here.com
 191   here.com
 192
 193 This entry does not remove:
 194   where.com
 195   www.adwhere.com
 196 </code></pre>
 197
 198 **query active block lists for a certain (sub-)domain, e.g. for whitelisting:**
 199 <pre><code>
 200 /etc/init.d/adblock query example.www.doubleclick.net
 201 ::: distinct results for domain 'example.www.doubleclick.net'
 202  no match
 203 ::: distinct results for domain 'www.doubleclick.net'
 204  adb_list.sysctl      : www.doubleclick.net
 205 ::: distinct results for domain 'doubleclick.net'
 206  adb_list.adaway      : ad-g.doubleclick.net
 207  adb_list.securemecca : 1168945.fls.doubleclick.net
 208  adb_list.sysctl      : 1435575.fls.doubleclick.net
 209  adb_list.whocares    : 3ad.doubleclick.net
 210
 211 The query function checks against the submitted (sub-)domain and recurses automatically to the upper top level domain(s).
 212 For every domain it returns the overall count plus a distinct list of active block lists with the first relevant result.
 213 In the example above whitelist "www.doubleclick.net" to free the submitted domain.
 214 </code></pre>
 215
 216 **add a new block list source:**
 217 <pre><code>
 218 1. the easy way ...
 219 example: https://easylist-downloads.adblockplus.org/rolist+easylist.txt
 220 adblock already supports an easylist source, called 'ruadlist'. To add the additional local easylist
 221 as a new source, copy the existing config source 'ruadlist' section and change only
 222 the source name, the url and the description - that's all!
 223
 224 config source 'rolist'
 225   option enabled '0'
 226   option adb_src 'https://easylist-downloads.adblockplus.org/rolist+easylist.txt'
 227   option adb_src_rset '{FS=\"[|^]\"} \$0 ~/^\|\|([A-Za-z0-9_-]+\.){1,}[A-Za-z]+\^$/{print tolower(\$3)}'
 228   option adb_src_desc 'focus on romanian ad related domains plus generic easylist additions, weekly updates, approx. 600 entries'
 229
 230 2. a bit harder ...
 231 to add a really new source with different domain/host format you have to write a suitable
 232 awk one-liner on your own, so basic awk skills are needed. As a starting point check the already
 233 existing awk strings (adb_src_rset) in adblock config, maybe you need only small changes for your individual list.
 234 Download the desired list and test your new awk string locally with:
 235   cat new.list | awk 'fs__individual search__search core__result'
 236   'fs' => field separator (optional)
 237   'individual search' => individual search part to filter out needless list information
 238   'search core' => always '([A-Za-z0-9_-]+\.){1,}[A-Za-z]+', this is part of all list sources and should be unchanged
 239   'result' => always '{print tolower(\$n)}', only the output column 'n' may vary
 240 the output result should be a sequential list with one domain/host per line - nothing more.
 241
 242 If your awk one-liner works quite well, add a new source section in adblock config and test your new source
 243 </code></pre>
 244
 245 ## Support
 246 Please join the adblock discussion in this [forum thread](https://forum.lede-project.org/t/adblock-2-x-support-thread/507) or contact me by mail <dev@brenken.org>
 247
 248 ## Removal
 249 * stop all adblock related services with _/etc/init.d/adblock stop_
 250 * optional: remove the adblock package (_opkg remove adblock_)
 251
 252 Have fun!
 253 Dirk