Thoughts on running small volume mail servers

[jggimi]

A couple of days ago in another thread, I was asked:

Originally Posted by fvgit

How much time & effort do you spend maintaining your mailserver? I've been remotely considering setting sth. like that up myself every once in a while. But every time Peter Hansteen posts one of his spam-related blogposts I tell myself: 'Nah, maybe not...'

I'd like to use this thread to discuss the effort, rather than all of the provisioning bits. For provisioning guidance, see Gilles Chehade's excellent blog post, Setting up a mail server with OpenSMTPD, Dovecot and Rspamd. Even if you select completely different software tools, there's plenty of good advice there about all of the fiddly bits that don't have much to do with your specific software choices, such as (prior) IP address and domain reputation, and setting up all the outgoing authentication protocols needed to participate in modern acceptable email transfer.

I perceive 2 key administrative tasks for operating mail servers:

1. Governing outbound SMTP traffic
   
   This is absolutely key to prevent both a decline in reputation and being added to blacklists.
   
   To avoid spam-pumps -- including from any IoT devices in your local networks -- unauthenticated internal mail should be blocked or rejected -- and you should either scan logs or be notified about these attempts, so you can find any spam-pumps and correct the compromised device.
   
   Authenticated mail should be monitored for volume, as a spam-pump could possibly have a valid authentication key or userid/password pair.
   
   I use a simple cron script which scans /var/log/maillog.0.gz and outputs a count of outbound emails. I can then review logs manually if I have any concerns. My mailservers block (and send to pflog) any attempts to send unauthenticated mail by internal network devices, and I check pflog files with tcpdump(8) every so often. So far, no internal spam-pumps of either kind.
2. Filtering or blocking incoming spam.
   
   This is always where most effort is expended.We want to stop all spam, but let non-spam through. There are many technologies that we can choose from to automate, or semi-automate blocking and/or filtering. But technologies shift, and we need to keep up or shift our focus. Years ago I used greylisting as a way to address incoming spam-bots, but with the advent of major mailer pools greylisting became less effective -- even harmful. In the last several years SPF analysis as a part of greylisting can make it useful once more.
   
   I use a combination of blocking and filtering. Any Email that gets through blocklists will be analyzed and tagged for sorting into an Inbox or Spam folder.
   
   Until quite recently I was using one automated blocklist (bgp-spamd), but it recently went defunct and I removed it. For years, I've been manually managing two local blocklists (short and long term). But with the recent re-addition of filters into my mail server software of choice, I've turned off all manual blocklists and have returned to using DNS-based blocklists as I used to do many years ago.

[fvgit]

Thanks a lot for the write-up! Since you've been doing this for years, have you ever run into any 'Oh shit!'-moments, especially with regard to your server's reputation or did yours even land on any blacklists?

I think that would be my biggest worry, running into such a scenario.

[jggimi]

I inherited an IP address with a poor reputation so I needed to contact several block lists to request review.

Provisioning DKIM, DMARC, and SPF took some time to get right.

[e1-531g]

Originally Posted by jggimi

I inherited an IP address with a poor reputation so I needed to contact several block lists to request review.

Provisioning DKIM, DMARC, and SPF took some time to get right.

Do you suggest any tools for testing server before sending and receiving e-mails from major e-mail services providers?

[jggimi]

As I recall, there were two external services that were helpful: first mxtoolbox.com, and then the Spam folder in a Gmail account.

[e1-531g]

Is it possible to get permanently blocklisted domain via some misconfiguration?
Maybe it is safer to get some free, temporary domain to practice and test server configuration and then switch to paid, personal domain.

[jggimi]

Originally Posted by e1-531g

Is it possible to get permanently blocklisted domain via some misconfiguration?

First, don't confuse two separate reputations, IP address and Domain name.

1. IP Address
   
   There are two types of address reputations to consider: individual address, and the netblock in which the address resides. For the latter, it is helpful to consider netblocks as a tree structure. Typically, netblocks are allocated by the Regional Internet Registries and then sub-allocated from larger to smaller service providers. This means that a single IP address might be a member of several netblocks, from smaller-to-larger, and the reputation of a larger "upstream" netblock will affect downstream reputations.
   
   Netblock reputation can be improved, but only if the service providers -- downstream and upstream are no longer doing evil. Netblock reputation improvement takes time, though, as changing "ownership" of evil netblocks is often done through front companies owned by the same people.
2. Domain Name
   
   Consider any "dangerous to click" link at a website or in an HTML email, such as in a phishing spam. The domain names used in these things get poor reputations, often very quickly, and are always considered permanent.

Maybe it is safer to get some free, temporary domain to practice and test server configuration and then switch to paid, personal domain.

Second, operating a mail service to receive incoming mail requires at minimum a static IP address on the Internet and an authoritative domain name with an MX domain record.

And, any outbound mail from that service will often be rejected or will be marked as spam outright unless additional DNS-based authentication protocols such as SPF and DKIM are deployed.

[e1-531g]

Last resort, but still practical solution for poor IP address reputation is to just switch ISP or VPS provider.
I mentioned domain, because when I like a domain name I don't want to abandon/lose/waste it. That is probably my biggest concern that badly configured DKIM or some other e-mail measure is going to blocklist my domain permanently.

[jggimi]

Originally Posted by e1-531g

That is probably my biggest concern that badly configured DKIM or some other e-mail measure is going to blocklist my domain permanently.

Domain reputations become poor through misuse -- that is, what is generally recognized as "evil" activity. Misconfiguration of mail servers that turn them into open relays for spam pumps (from internal or external sources) will usually not not carry the mail server's domain name in their traffic and will not affect that domain's reputation, only the emitting IP address. If there are multiple emitters from the same netblock, it can affect the netblock's reputation. DKIM/SPF/DMARC misconfiguration happens, and generally does not effect reputation over the short term.

Poor domain reputation is more likely to be associated with a compromised web server.

For more on domain reputation, see the FAQ for Spamhaus.org's Domain Block List -- it may be helpful: https://www.spamhaus.org/faq/section/Spamhaus%20DBL

[jggimi]

Since I mentioned the Spamhaus Domain Block List (DBL), I should note that its one of their advisory services that I don't actually utilize.

I use their Don't Route or Peer (DROP) lists in PF on all internet-facing servers. My PF logs show constant probes from these "evil" netblocks. I see probes for responses from common ports like ntp, smtp, http, https, as well as probes for responses from a vast array of high port numbers for "bot" command and control. (This doesn't eliminate the constant probing from the rest of the Internet, but there's no reason to permit any traffic from hijacked RIR netblocks or other recognized criminal-controlled netblocks.)

On my external-facing mail server, I also use the Spamhous consolidated ZEN block list as one of several DNSBL filters.

[e1-531g]

Thanks for the response jjgimi. I think I just needed some reassurance.
Thanks for the names of different block lists. It is good starting point.

[jggimi]

I just checked MX toolbox, and it currently reports on 95 domain blacklists and 87 IP blacklists. Which is a lot.

[jggimi]

I just happened to review mail logs logs today, and noticed that yesterday a spambot (at an apparent residential ISP) attempted to connect 27 times within a single 4-second period. No idea why it stopped at 27. It came, it hammered, it left.

I filter with three different DNSBLs -- Spamhaus Zen, GBUdb, and SpamCop -- and the IP address was listed at GDUdb. It also happened to be the first time since I enabled filtering that GBUdb showed a positive listing while the other two blacklists did not.

[e1-531g]

How do you deal with ICMP packets?
I've read about it and it seems that rate-limiting is best for ICMP packets, because in the past it was quite easy to DoS different OSes with ICMP packets. Blocking is another option, but it probably isn't the most RFC compliant way.
I don't know how much should I rate-limit them and where. ICMP rate-limiting on ingress only should prevent DoS, but may be not enough to prevent sending large number of ICMP packets (packet reflection), because they may be generated by OS rejecting UDP packets. On the other hand rate-limiting on egress only does not prevent against DoS and I can just not reject UDP packets and drop them instead. I lean towards rate-limiting on both ingress and egress.

[jggimi]

ICMPv6 is an operational requirement for IPv6, and my main mail server is IPv6/IPv4. I don't block ICMPv6 and I have never had any issues with ICMP (v4). I would look into rate limiting ICMP it if my server is ever attacked.

[jggimi]

Currently, my DNSBL filter is working very well, and spam is at very manageable levels.

I now only report spam to the responsible ISP ("LART") when the sending IP address has a neutral or positive Mailspike reputation, or no reputation at all. If it has a negative reputation, then I assume my report will bounce or be ignored, and I will not bother to send an abuse report. SpamAssassin provides this reputation level as part of its analysis reporting, so it makes it easy for me to determine whether to LART or not. When LARTing, I use a simple script that automates much of the report generation, minimizing my effort when I do report.

LART is an acronym which these days just means "Report to the responsible ISP." Hstorically it meant bringing the problem of spam to the attention of those who are unaware but with awareness might be able to do something about it: "Loser Attitude Readjustment Tool."

[e1-531g]

Originally Posted by jggimi

ICMPv6 is an operational requirement for IPv6, and my main mail server is IPv6/IPv4. I don't block ICMPv6 and I have never had any issues with ICMP (v4). I would look into rate limiting ICMP it if my server is ever attacked.

Thanks for the response.
I can enable IPv6 for my VPS, but my ISP and many other ISPs in Poland does not support IPv4 and I recall Gmail also have not accepted mail sent by other server (different domain) via IPv6 protocol (fix for Postfix), so I operate IPv4 server and I am mainly interested in ICMP(v4).
When it comes to packet reflection some Paul Vixie's article was eye opening for me. It is probably hard to completely defend against being taken advantage of packet reflection, but at least I will try to limit it. Not only to defend my VPS, but also to not help DDoS others services.

[jggimi]

Originally Posted by e1-531g

...I recall Gmail also have not accepted mail sent by other server (different domain) via IPv6 protocol...

Most of my incoming Gmail arrives via IPv6. They appear to prefer it to IPv4 when available.

Both ICMPs are state-managed by PF, so stateful tracking options are available to rate limit them, such as:

pass proto {icmp icmp6} keep state (max 5000)

I haven't needed this, so far.