* added HTTP ETag or entity tag support to download only ressources that have been updated on the server side,
to save bandwith and speed up banIP reloads
* added 4 new feeds: binarydefense, bruteforceblock, etcompromised, ipblackhole (see readme)
* updated the readme
Signed-off-by: Dirk Brenken <dev@brenken.org>
include $(TOPDIR)/rules.mk
PKG_NAME:=banip
-PKG_VERSION:=0.8.8
-PKG_RELEASE:=2
+PKG_VERSION:=0.8.9
+PKG_RELEASE:=1
PKG_LICENSE:=GPL-3.0-or-later
PKG_MAINTAINER:=Dirk Brenken <dev@brenken.org>
| antipopads | antipopads IPs | | | x | [Link](https://github.com/dibdot/banIP-IP-blocklists) |
| asn | ASN IPs | | | x | [Link](https://asn.ipinfo.app) |
| backscatterer | backscatterer IPs | x | x | | [Link](https://www.uceprotect.net/en/index.php) |
+| binarydefense | binary defense banlist | x | x | | [Link](https://iplists.firehol.org/?ipset=bds_atif) |
| bogon | bogon prefixes | x | x | | [Link](https://team-cymru.com) |
+| bruteforceblock | bruteforceblocker IPs | x | x | | [Link](https://danger.rulez.sk/index.php/bruteforceblocker/) |
| country | country blocks | x | x | | [Link](https://www.ipdeny.com/ipblocks) |
| cinsscore | suspicious attacker IPs | x | x | | [Link](https://cinsscore.com/#list) |
| darklist | blocks suspicious attacker IPs | x | x | | [Link](https://darklist.de) |
| drop | spamhaus drop compilation | x | x | | [Link](https://www.spamhaus.org) |
| dshield | dshield IP blocklist | x | x | | [Link](https://www.dshield.org) |
| edrop | spamhaus edrop compilation | x | x | | [Link](https://www.spamhaus.org) |
+| etcompromised | ET compromised hosts | x | x | | [Link](https://iplists.firehol.org/?ipset=et_compromised) |
| feodo | feodo tracker | x | x | x | [Link](https://feodotracker.abuse.ch) |
| firehol1 | firehol level 1 compilation | x | x | | [Link](https://iplists.firehol.org/?ipset=firehol_level1) |
| firehol2 | firehol level 2 compilation | x | x | | [Link](https://iplists.firehol.org/?ipset=firehol_level2) |
| greensnow | suspicious server IPs | x | x | | [Link](https://greensnow.co) |
| iblockads | Advertising IPs | | | x | [Link](https://www.iblocklist.com) |
| iblockspy | Malicious spyware IPs | x | x | | [Link](https://www.iblocklist.com) |
+| ipblackhole | blackhole IPs | x | x | | [Link](https://ip.blackhole.monster) |
| ipthreat | hacker and botnet TPs | x | x | | [Link](https://ipthreat.net) |
| myip | real-time IP blocklist | x | x | | [Link](https://myip.ms) |
| nixspam | iX spam protection | x | x | | [Link](http://www.nixspam.org) |
* Per feed it can be defined whether the wan-input chain, the wan-forward chain or the lan-forward chain should be blocked (default: all chains)
* Automatic blocklist backup & restore, the backups will be used in case of download errors or during startup
* Automatically selects one of the following download utilities with ssl support: aria2c, curl, uclient-fetch or full wget
-* Supports an 'allowlist only' mode, this option restricts internet access from/to a small number of secure websites/IPs
+* Provides HTTP ETag or entity tag support to download only ressources that have been updated on the server side, to save bandwith and speed up banIP reloads
+* Supports an 'allowlist only' mode, this option restricts internet access from/to a given number of secure websites/IPs
* Deduplicate IPs accross all Sets (single IPs only, no intervals)
* Provides comprehensive runtime information
* Provides a detailed Set report
## Prerequisites
* **[OpenWrt](https://openwrt.org)**, latest stable release or a snapshot with nft/firewall 4 and logd/logread support
-* A download utility with SSL support: 'aria2c', 'curl', full 'wget' or 'uclient-fetch' with one of the 'libustream-*' SSL libraries
+* A download utility with SSL support: 'aria2c', 'curl', full 'wget' or 'uclient-fetch' with one of the 'libustream-*' SSL libraries, the latter one doesn't provide support for ETag HTTP header
* A certificate store like 'ca-bundle', as banIP checks the validity of the SSL certificates of all download sites by default
* For E-Mail notifications you need to install and setup the additional 'msmtp' package
| ban_autoblocklist | option | 1 | add suspicious attacker IPs and resolved domains automatically to the local blocklist (not only to the Sets) |
| ban_autoblocksubnet | option | 0 | add entire subnets to the blocklist Sets based on an additional RDAP request with the suspicious IP |
| ban_autoallowuplink | option | subnet | limit the uplink autoallow function to: 'subnet', 'ip' or 'disable' it at all |
-| ban_allowlistonly | option | 0 | restrict the internet access from/to a small number of secure websites/IPs |
+| ban_allowlistonly | option | 0 | restrict the internet access from/to a given number of secure websites/IPs |
| ban_basedir | option | /tmp | base working directory while banIP processing |
| ban_reportdir | option | /tmp/banIP-report | directory where banIP stores the report files |
| ban_backupdir | option | /tmp/banIP-backup | directory where banIP stores the compressed backup files |
Furthermore, you can reference external Allowlist URLs with additional IPv4 and IPv6 feeds (see 'ban_allowurl').
Both local lists also accept domain names as input to allow IP filtering based on these names. The corresponding IPs (IPv4 & IPv6) will be extracted and added to the Sets. You can also start the domain lookup separately via /etc/init.d/banip lookup at any time.
+**allowlist-only mode**
+banIP supports an "allowlist only" mode. This option restricts the internet access from/to a small number of secure MACs, IPs or domains, and block access from/to the rest of the internet. All IPs and Domains which are _not_ listed in the allowlist (plus the external Allowlist URLs) are blocked.
+
**MAC/IP-binding**
banIP supports concatenation of local MAC addresses with IPv4/IPv6 addresses, e.g. to enforce dhcp assignments. Following notations in the local allow and block lists are allowed:
```
C8:C2:9B:F7:80:12 => this will be populated to v6MAC-Set with the IP-wildcard ::/0
```
-**allowlist-only mode**
-banIP supports an "allowlist only" mode. This option restricts the internet access from/to a small number of secure MACs, IPs or domains, and block access from/to the rest of the internet. All IPs and Domains which are _not_ listed in the allowlist are blocked.
-
**redirect Asterisk security logs to lodg/logread**
banIP only supports logfile scanning via logread, so to monitor attacks on Asterisk, its security log must be available via logread. To do this, edit '/etc/asterisk/logger.conf' and add the line 'syslog.local0 = security', then run 'asterisk -rx reload logger' to update the running Asterisk configuration.
ban_fetchinsecure=""
ban_fetchretry="5"
ban_rdapparm=""
+ban_etagparm=""
ban_cores=""
ban_memory=""
ban_packages=""
[ "${ban_fetchinsecure}" = "1" ] && insecure="--check-certificate=false"
ban_fetchparm="${ban_fetchparm:-"${insecure} --timeout=20 --retry-wait=10 --max-tries=${ban_fetchretry} --max-file-not-found=${ban_fetchretry} --allow-overwrite=true --auto-file-renaming=false --log-level=warn --dir=/ -o"}"
ban_rdapparm="--timeout=5 --allow-overwrite=true --auto-file-renaming=false --dir=/ -o"
+ ban_etagparm="--timeout=5 --allow-overwrite=true --auto-file-renaming=false --dir=/ --dry-run --log -"
;;
"curl")
[ "${ban_fetchinsecure}" = "1" ] && insecure="--insecure"
- ban_fetchparm="${ban_fetchparm:-"${insecure} --connect-timeout 20 --retry-delay 10 --retry ${ban_fetchretry} --retry-all-errors --fail --silent --show-error --location -o"}"
+ ban_fetchparm="${ban_fetchparm:-"${insecure} --connect-timeout 20 --retry-delay 10 --retry ${ban_fetchretry} --retry-max-time $((ban_fetchretry * 20)) --retry-all-errors --fail --silent --show-error --location -o"}"
ban_rdapparm="--connect-timeout 5 --silent --location -o"
+ ban_etagparm="--connect-timeout 5 --silent --location --head"
;;
- "uclient-fetch")
+ "wget")
[ "${ban_fetchinsecure}" = "1" ] && insecure="--no-check-certificate"
- ban_fetchparm="${ban_fetchparm:-"${insecure} --timeout=20 -O"}"
+ ban_fetchparm="${ban_fetchparm:-"${insecure} --no-cache --no-cookies --timeout=20 --waitretry=10 --tries=${ban_fetchretry} --retry-connrefused -O"}"
ban_rdapparm="--timeout=5 -O"
+ ban_etagparm="--timeout=5 --spider --server-response"
;;
- "wget")
+ "uclient-fetch")
[ "${ban_fetchinsecure}" = "1" ] && insecure="--no-check-certificate"
- ban_fetchparm="${ban_fetchparm:-"${insecure} --no-cache --no-cookies --timeout=20 --waitretry=10 --tries=${ban_fetchretry} --retry-connrefused -O"}"
+ ban_fetchparm="${ban_fetchparm:-"${insecure} --timeout=20 -O"}"
ban_rdapparm="--timeout=5 -O"
;;
esac
- f_log "debug" "f_getfetch ::: auto/update: ${ban_autodetect}/${update}, cmd: ${ban_fetchcmd:-"-"}, fetch_parm: ${ban_fetchparm:-"-"}, rdap_parm: ${ban_rdapparm:-"-"}"
+ f_log "debug" "f_getfetch ::: auto/update: ${ban_autodetect}/${update}, cmd: ${ban_fetchcmd:-"-"}, fetch_parm: ${ban_fetchparm:-"-"}, rdap_parm: ${ban_rdapparm:-"-"}, etag_parm: ${ban_etagparm:-"-"}"
}
# get wan interfaces
for ip in ${ban_uplink}; do
if ! "${ban_grepcmd}" -q "${ip} " "${ban_allowlist}"; then
if [ "${update}" = "0" ]; then
- "${ban_sedcmd}" -i '/# uplink added on /d' "${ban_allowlist}"
+ "${ban_sedcmd}" -i "/# uplink added on /d" "${ban_allowlist}"
fi
printf "%-42s%s\n" "${ip}" "# uplink added on $(date "+%Y-%m-%d %H:%M:%S")" >>"${ban_allowlist}"
f_log "info" "add uplink '${ip}' to local allowlist"
done
ban_uplink="${ban_uplink%%?}"
elif [ "${ban_autoallowlist}" = "1" ] && [ "${ban_autoallowuplink}" = "disable" ]; then
- "${ban_sedcmd}" -i '/# uplink added on /d' "${ban_allowlist}"
+ "${ban_sedcmd}" -i "/# uplink added on /d" "${ban_allowlist}"
update="1"
fi
[ -s "${file}" ] && printf "%s" "elements={ $("${ban_catcmd}" "${file}" 2>/dev/null) };"
}
+# handle etag http header
+#
+f_etag() {
+ local http_head http_code etag_id etag_rc out_rc="4" feed="${1}" feed_url="${2}" feed_suffix="${3}"
+
+ if [ -n "${ban_etagparm}" ]; then
+ [ ! -f "${ban_backupdir}/banIP.etag" ] && : >"${ban_backupdir}/banIP.etag"
+ http_head="$("${ban_fetchcmd}" ${ban_etagparm} "${feed_url}" 2>&1)"
+ http_code="$(printf "%s" "${http_head}" | "${ban_awkcmd}" 'tolower($0)~/^http\/[0123\.]+ /{printf "%s",$2}')"
+ etag_id="$(printf "%s" "${http_head}" | "${ban_awkcmd}" '{FS="\""}tolower($0)~/^[[:space:]]*etag: /{printf "%s",$2}')"
+ etag_rc="${?}"
+
+ if [ "${http_code}" = "404" ] || { [ "${etag_rc}" = "0" ] && [ -n "${etag_id}" ] && "${ban_grepcmd}" -q "^${feed}${feed_suffix}.*${etag_id}\$" "${ban_backupdir}/banIP.etag"; }; then
+ out_rc="0"
+ elif [ "${etag_rc}" = "0" ] && [ -n "${etag_id}" ] && ! "${ban_grepcmd}" -q "^${feed}${feed_suffix}.*${etag_id}\$" "${ban_backupdir}/banIP.etag"; then
+ "${ban_sedcmd}" -i "/^${feed}${feed_suffix}/d" "${ban_backupdir}/banIP.etag"
+ printf "%-20s%s\n" "${feed}${feed_suffix}" "${etag_id}" >>"${ban_backupdir}/banIP.etag"
+ out_rc="2"
+ fi
+ fi
+
+ f_log "debug" "f_etag ::: feed: ${feed}, suffix: ${feed_suffix:-"-"}, http_code: ${http_code:-"-"}, etag_id: ${etag_id:-"-"} , etag_rc: ${etag_rc:-"-"}, rc: ${out_rc}"
+ return "${out_rc}"
+}
+
# build initial nft file with base table, chains and rules
#
f_nftinit() {
feed_rc="${?}"
f_log "debug" "f_nftinit ::: devices: ${ban_dev}, priority: ${ban_nftpriority}, policy: ${ban_nftpolicy}, loglevel: ${ban_nftloglevel}, rc: ${feed_rc:-"-"}, log: ${feed_log:-"-"}"
- return ${feed_rc}
+ return "${feed_rc}"
}
# handle downloads
#
f_down() {
- local log_input log_forwardwan log_forwardlan start_ts end_ts tmp_raw tmp_load tmp_file split_file ruleset_raw handle
+ local log_input log_forwardwan log_forwardlan start_ts end_ts tmp_raw tmp_load tmp_file split_file ruleset_raw handle rc etag_rc="0"
local cnt_set cnt_dl restore_rc feed_direction feed_rc feed_log feed="${1}" proto="${2}" feed_url="${3}" feed_rule="${4}" feed_flag="${5}"
start_ts="$(date +%s)"
} >"${tmp_flush}"
fi
- # restore local backups during init
+ # restore local backups
#
- if { [ "${ban_action}" != "reload" ] || [ "${feed_url}" = "local" ]; } && [ "${feed%v*}" != "allowlist" ] && [ "${feed%v*}" != "blocklist" ]; then
- f_restore "${feed}" "${feed_url}" "${tmp_load}"
- restore_rc="${?}"
- feed_rc="${restore_rc}"
+ if { [ "${ban_action}" != "reload" ] || [ "${feed_url}" = "local" ] || [ -n "${ban_etagparm}" ]; } && [ "${feed%v*}" != "allowlist" ] && [ "${feed%v*}" != "blocklist" ]; then
+ if [ -n "${ban_etagparm}" ] && [ "${feed_url}" != "local" ]; then
+ if [ "${feed%v*}" = "country" ]; then
+ for country in ${ban_country}; do
+ f_etag "${feed}" "${feed_url}${country}-aggregated.zone" ".${country}"
+ rc="${?}"
+ [ "${rc}" = "4" ] && break
+ etag_rc="$((etag_rc + rc))"
+ done
+ elif [ "${feed%v*}" = "asn" ]; then
+ for asn in ${ban_asn}; do
+ f_etag "${feed}" "${feed_url}AS${asn}" ".{asn}"
+ rc="${?}"
+ [ "${rc}" = "4" ] && break
+ etag_rc="$((etag_rc + rc))"
+ done
+ else
+ f_etag "${feed}" "${feed_url}"
+ etag_rc="${?}"
+ fi
+ fi
+ if [ "${etag_rc}" = "0" ] || [ "${ban_action}" != "reload" ] || [ "${feed_url}" = "local" ]; then
+ f_restore "${feed}" "${feed_url}" "${tmp_load}" "${etag_rc}"
+ restore_rc="${?}"
+ feed_rc="${restore_rc}"
+ fi
fi
# prepare local allowlist
"gz")
feed_log="$("${ban_fetchcmd}" ${ban_fetchparm} "${tmp_raw}" "${feed_url}" 2>&1)"
feed_rc="${?}"
- if [ "${feed_rc}" = "0" ]; then
- "${ban_zcatcmd}" "${tmp_raw}" 2>/dev/null >"${tmp_load}"
- feed_rc="${?}"
- fi
+ [ "${feed_rc}" = "0" ] && "${ban_zcatcmd}" "${tmp_raw}" 2>/dev/null >"${tmp_load}"
rm -f "${tmp_raw}"
;;
esac
rm -f "${tmp_split}" "${tmp_nft}"
end_ts="$(date +%s)"
- f_log "debug" "f_down ::: name: ${feed}, cnt_dl: ${cnt_dl:-"-"}, cnt_set: ${cnt_set:-"-"}, split_size: ${ban_splitsize:-"-"}, time: $((end_ts - start_ts)), rc: ${feed_rc:-"-"}, log: ${feed_log:-"-"}"
+ f_log "debug" "f_down ::: feed: ${feed}, cnt_dl: ${cnt_dl:-"-"}, cnt_set: ${cnt_set:-"-"}, split_size: ${ban_splitsize:-"-"}, time: $((end_ts - start_ts)), rc: ${feed_rc:-"-"}, log: ${feed_log:-"-"}"
}
# backup feeds
gzip -cf "${feed_file}" >"${ban_backupdir}/banIP.${feed}.gz"
backup_rc="${?}"
- f_log "debug" "f_backup ::: name: ${feed}, source: ${feed_file##*/}, target: banIP.${feed}.gz, rc: ${backup_rc}"
- return ${backup_rc}
+ f_log "debug" "f_backup ::: feed: ${feed}, file: banIP.${feed}.gz, rc: ${backup_rc}"
+ return "${backup_rc}"
}
# restore feeds
#
f_restore() {
- local tmp_feed restore_rc="1" feed="${1}" feed_url="${2}" feed_file="${3}" feed_rc="${4:-"0"}"
+ local tmp_feed restore_rc="4" feed="${1}" feed_url="${2}" feed_file="${3}" in_rc="${4}"
- [ "${feed_rc}" != "0" ] && restore_rc="${feed_rc}"
[ "${feed_url}" = "local" ] && tmp_feed="${feed%v*}v4" || tmp_feed="${feed}"
- if [ -f "${ban_backupdir}/banIP.${tmp_feed}.gz" ]; then
+ if [ -s "${ban_backupdir}/banIP.${tmp_feed}.gz" ]; then
"${ban_zcatcmd}" "${ban_backupdir}/banIP.${tmp_feed}.gz" 2>/dev/null >"${feed_file}"
restore_rc="${?}"
fi
- f_log "debug" "f_restore ::: name: ${feed}, source: banIP.${tmp_feed}.gz, target: ${feed_file##*/}, in_rc: ${feed_rc}, rc: ${restore_rc}"
- return ${restore_rc}
+ f_log "debug" "f_restore ::: feed: ${feed}, file: banIP.${tmp_feed}.gz, in_rc: ${in_rc:-"-"}, rc: ${restore_rc}"
+ return "${restore_rc}"
}
# remove disabled Sets
"descr": "backscatterer IPs",
"flag": "gz"
},
+ "binarydefense":{
+ "url_4": "https://iplists.firehol.org/files/bds_atif.ipset",
+ "rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$/{printf \"%s,\\n\",$1}",
+ "descr": "binary defense banlist"
+ },
"bogon":{
"url_4": "https://www.team-cymru.org/Services/Bogons/fullbogons-ipv4.txt",
"url_6": "https://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt",
"rule_6": "/^(([0-9A-f]{0,4}:){1,7}[0-9A-f]{0,4}:?(\\/(1?[0-2][0-8]|[0-9][0-9]))?)$/{printf \"%s,\\n\",$1}",
"descr": "bogon prefixes"
},
+ "bruteforceblock":{
+ "url_4": "https://danger.rulez.sk/projects/bruteforceblocker/blist.php",
+ "rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)[[:space:]]/{printf \"%s,\\n\",$1}",
+ "descr": "bruteforceblocker IPs"
+ },
"cinsscore":{
"url_4": "https://cinsscore.com/list/ci-badguys.txt",
"rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$/{printf \"%s,\\n\",$1}",
"rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)[[:space:]]/{printf \"%s,\\n\",$1}",
"descr": "spamhaus edrop compilation"
},
+ "etcompromised":{
+ "url_4": "https://iplists.firehol.org/files/et_compromised.ipset",
+ "rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$/{printf \"%s,\\n\",$1}",
+ "descr": "ET compromised hosts"
+ },
"feodo":{
"url_4": "https://feodotracker.abuse.ch/downloads/ipblocklist.txt",
"rule_4": "BEGIN{RS=\"\\r\\n\"}/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$/{printf \"%s,\\n\",$1}",
"descr": "malicious spyware IPs",
"flag": "gz"
},
+ "ipblackhole":{
+ "url_4": "https://ip.blackhole.monster/blackhole-today",
+ "rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)$/{printf \"%s,\\n\",$1}",
+ "descr": "blackhole IP blocklist"
+ },
"ipthreat":{
"url_4": "https://lists.ipthreat.net/file/ipthreat-lists/threat/threat-30.txt",
"rule_4": "/^(([0-9]{1,3}\\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])(\\/(1?[0-9]|2?[0-9]|3?[0-2]))?)[-[:space:]]?/{printf \"%s,\\n\",$1}",
projects / feed / packages.git / commitdiff