Using Spamhaus DROP lists on Internet-facing systems

[jggimi]

In a thread about mail hosting, I'd written, in part:

Originally Posted by jggimi

...I use their Don't Route or Peer (DROP) lists in PF on all internet-facing servers. My PF logs show constant probes from these "evil" netblocks....

The DROP lists consist of 3 text files that do not change frequently. They are served by BGP or by HTTP/HTTPS. Spamhaus.org requires refreshing them at a maximum of once per hour, and at minimum once per day. I use the following script, called by daily.local(5) for a once-per-day refresh. The 3 files are downloaded via HTTPS into a temporary directory, they are concatenated together, semicolons denoting comments are converted to pound signs for compatibility with PF, the combined file is stored in the /root directory, and the PF <spamhaus> table is updated.

#!/bin/sh
#
# this is normally run once per day via /etc/daily.local.
#
echo updating Spamhaus DROP lists:
TMPDIR=`mktemp -d` || exit 1
( cd $TMPDIR
ftp https://www.spamhaus.org/drop/drop.txt 
ftp https://www.spamhaus.org/drop/edrop.txt 
ftp https://www.spamhaus.org/drop/dropv6.txt 
cat drop.txt edrop.txt dropv6.txt | sed "s/;/#/" > /root/drop.txt
)
rm -r $TMPDIR
pfctl -t spamhaus -T replace -f /root/drop.tx

My pf.conf(5) files on my servers have the following rules included very early in each ruleset, typically immediately following the default block. On any boot, the most recently received and edited combined DROP lists are read from the file in the /root directory.

# Spamhaus DROP list:
table <spamhaus> persist file "/root/drop.txt"
block drop in log quick from <spamhaus>

[jggimi]

Yeah, I should probably move the file from /root to /var/db.

[e1-531g]

Originally Posted by jggimi

In a thread about mail hosting, I'd written, in part: [URL="https://www.spamhaus.org/drop/"]Spamhaus.org requires refreshing them at a maximum of once per hour, and at minimum once per day.

Actually even less frequently may be fine:

Please DO NOT auto-fetch the DROP / EDROP list more than once per hour!

The DROP list changes quite slowly. There is no need to update cached data more than once per hour, in fact once per day is more than enough in most cases. Automated downloads must be at least one hour apart. Excessive downloads may result in your IP being firewalled from the Spamhaus website.

I used Firefox to see DROP list and HTTP header says:
last-modified: Sun, 31 May 2020 15:45:06 GMT
For eDROP:
last-modified: Tue, 12 May 2020 10:42:24 GMT
DropV6 doesn't sent that header, but third line of file says:
; Last-Modified: Thu, 30 Jan 2020 05:16:30 GMT

Anyway I really like that OpenBSD's pf allows to load table directly from file. In Linux's ipset command closest you can get is "ipset restore < file" command, but file must include actual ipset commands instead of only CIDRs.

[jggimi]

While I could call the script with weekly.local(5) as the lists don't change frequently, even hourly downloads are acceptable use to Spamhaus.

The pfctl(8) -T replace messages are informative. The cron output shows, as examples, "no changes." or "2 addresses added."


---


Edited to add: Spamhaus marks the lists with an expiration, and recommends one-per-day as the minimum refresh rate.

[e1-531g]

Originally Posted by jggimi

The pfctl(8) -T replace messages are informative. The cron output shows, as examples, "no changes." or "2 addresses added."

Another small thing that makes PF shine.

[e1-531g]

Little suggestion for script in post #1
I was experimenting with bash and wget lately, but I think this should going to work for Korn Shell and ftp too. Execute frp commands in a group. Instead of writing to three files make them send output to stdout and pipe directly to sed:

#!/bin/sh
#
# this is normally run once per day via /etc/daily.local.
#
echo updating Spamhaus DROP lists:
( 
  { ftp -o - https://www.spamhaus.org/drop/drop.txt && \
    ftp -o - https://www.spamhaus.org/drop/edrop.txt && \
    ftp -o - https://www.spamhaus.org/drop/dropv6.txt ; \
  } 2>/dev/null | sed "s/;/#/" > /root/drop.txt
)
pfctl -t spamhaus -T replace -f /root/drop.txt

[jggimi]

Yeah, that's another way to do it, thanks! (And I did move the file to /var/db/ on all my public-facing systems, after years in /root.)