DaemonForums  

Go Back   DaemonForums > Miscellaneous > General software and network

General software and network General OS-independent software and network questions, X11, MTA, routing, etc.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 22nd May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default Thoughts on running small volume mail servers

A couple of days ago in another thread, I was asked:
Quote:
Originally Posted by fvgit View Post
How much time & effort do you spend maintaining your mailserver? I've been remotely considering setting sth. like that up myself every once in a while. But every time Peter Hansteen posts one of his spam-related blogposts I tell myself: 'Nah, maybe not...'
I'd like to use this thread to discuss the effort, rather than all of the provisioning bits. For provisioning guidance, see Gilles Chehade's excellent blog post, Setting up a mail server with OpenSMTPD, Dovecot and Rspamd. Even if you select completely different software tools, there's plenty of good advice there about all of the fiddly bits that don't have much to do with your specific software choices, such as (prior) IP address and domain reputation, and setting up all the outgoing authentication protocols needed to participate in modern acceptable email transfer.

I perceive 2 key administrative tasks for operating mail servers:
  1. Governing outbound SMTP traffic

    This is absolutely key to prevent both a decline in reputation and being added to blacklists.

    To avoid spam-pumps -- including from any IoT devices in your local networks -- unauthenticated internal mail should be blocked or rejected -- and you should either scan logs or be notified about these attempts, so you can find any spam-pumps and correct the compromised device.

    Authenticated mail should be monitored for volume, as a spam-pump could possibly have a valid authentication key or userid/password pair.

    I use a simple cron script which scans /var/log/maillog.0.gz and outputs a count of outbound emails. I can then review logs manually if I have any concerns. My mailservers block (and send to pflog) any attempts to send unauthenticated mail by internal network devices, and I check pflog files with tcpdump(8) every so often. So far, no internal spam-pumps of either kind.
  2. Filtering or blocking incoming spam.

    This is always where most effort is expended.We want to stop all spam, but let non-spam through. There are many technologies that we can choose from to automate, or semi-automate blocking and/or filtering. But technologies shift, and we need to keep up or shift our focus. Years ago I used greylisting as a way to address incoming spam-bots, but with the advent of major mailer pools greylisting became less effective -- even harmful. In the last several years SPF analysis as a part of greylisting can make it useful once more.

    I use a combination of blocking and filtering. Any Email that gets through blocklists will be analyzed and tagged for sorting into an Inbox or Spam folder.

    Until quite recently I was using one automated blocklist (bgp-spamd), but it recently went defunct and I removed it. For years, I've been manually managing two local blocklists (short and long term). But with the recent re-addition of filters into my mail server software of choice, I've turned off all manual blocklists and have returned to using DNS-based blocklists as I used to do many years ago.
Reply With Quote
  #2   (View Single Post)  
Old 22nd May 2020
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 314
Default

Thanks a lot for the write-up! Since you've been doing this for years, have you ever run into any 'Oh shit!'-moments, especially with regard to your server's reputation or did yours even land on any blacklists?

I think that would be my biggest worry, running into such a scenario.
Reply With Quote
  #3   (View Single Post)  
Old 23rd May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I inherited an IP address with a poor reputation so I needed to contact several block lists to request review.

Provisioning DKIM, DMARC, and SPF took some time to get right.
Reply With Quote
  #4   (View Single Post)  
Old 23rd May 2020
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Quote:
Originally Posted by jggimi View Post
I inherited an IP address with a poor reputation so I needed to contact several block lists to request review.

Provisioning DKIM, DMARC, and SPF took some time to get right.
Do you suggest any tools for testing server before sending and receiving e-mails from major e-mail services providers?
__________________
Signature: Furthermore, I consider that systemd must be destroyed.
Based on Latin oratorical phrase
Reply With Quote
  #5   (View Single Post)  
Old 23rd May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

As I recall, there were two external services that were helpful: first mxtoolbox.com, and then the Spam folder in a Gmail account.
Reply With Quote
  #6   (View Single Post)  
Old 25th May 2020
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Is it possible to get permanently blocklisted domain via some misconfiguration?
Maybe it is safer to get some free, temporary domain to practice and test server configuration and then switch to paid, personal domain.
__________________
Signature: Furthermore, I consider that systemd must be destroyed.
Based on Latin oratorical phrase
Reply With Quote
  #7   (View Single Post)  
Old 25th May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
Originally Posted by e1-531g View Post
Is it possible to get permanently blocklisted domain via some misconfiguration?
First, don't confuse two separate reputations, IP address and Domain name.

  1. IP Address

    There are two types of address reputations to consider: individual address, and the netblock in which the address resides. For the latter, it is helpful to consider netblocks as a tree structure. Typically, netblocks are allocated by the Regional Internet Registries and then sub-allocated from larger to smaller service providers. This means that a single IP address might be a member of several netblocks, from smaller-to-larger, and the reputation of a larger "upstream" netblock will affect downstream reputations.

    Netblock reputation can be improved, but only if the service providers -- downstream and upstream are no longer doing evil. Netblock reputation improvement takes time, though, as changing "ownership" of evil netblocks is often done through front companies owned by the same people.
  2. Domain Name

    Consider any "dangerous to click" link at a website or in an HTML email, such as in a phishing spam. The domain names used in these things get poor reputations, often very quickly, and are always considered permanent.
Quote:
Maybe it is safer to get some free, temporary domain to practice and test server configuration and then switch to paid, personal domain.
Second, operating a mail service to receive incoming mail requires at minimum a static IP address on the Internet and an authoritative domain name with an MX domain record.

And, any outbound mail from that service will often be rejected or will be marked as spam outright unless additional DNS-based authentication protocols such as SPF and DKIM are deployed.
Reply With Quote
  #8   (View Single Post)  
Old 25th May 2020
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Last resort, but still practical solution for poor IP address reputation is to just switch ISP or VPS provider.
I mentioned domain, because when I like a domain name I don't want to abandon/lose/waste it. That is probably my biggest concern that badly configured DKIM or some other e-mail measure is going to blocklist my domain permanently.
__________________
Signature: Furthermore, I consider that systemd must be destroyed.
Based on Latin oratorical phrase
Reply With Quote
  #9   (View Single Post)  
Old 25th May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
Originally Posted by e1-531g View Post
That is probably my biggest concern that badly configured DKIM or some other e-mail measure is going to blocklist my domain permanently.
Domain reputations become poor through misuse -- that is, what is generally recognized as "evil" activity. Misconfiguration of mail servers that turn them into open relays for spam pumps (from internal or external sources) will usually not not carry the mail server's domain name in their traffic and will not affect that domain's reputation, only the emitting IP address. If there are multiple emitters from the same netblock, it can affect the netblock's reputation. DKIM/SPF/DMARC misconfiguration happens, and generally does not effect reputation over the short term.

Poor domain reputation is more likely to be associated with a compromised web server.

For more on domain reputation, see the FAQ for Spamhaus.org's Domain Block List -- it may be helpful: https://www.spamhaus.org/faq/section/Spamhaus%20DBL
Reply With Quote
Old 26th May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Since I mentioned the Spamhaus Domain Block List (DBL), I should note that its one of their advisory services that I don't actually utilize.

I use their Don't Route or Peer (DROP) lists in PF on all internet-facing servers. My PF logs show constant probes from these "evil" netblocks. I see probes for responses from common ports like ntp, smtp, http, https, as well as probes for responses from a vast array of high port numbers for "bot" command and control. (This doesn't eliminate the constant probing from the rest of the Internet, but there's no reason to permit any traffic from hijacked RIR netblocks or other recognized criminal-controlled netblocks.)

On my external-facing mail server, I also use the Spamhous consolidated ZEN block list as one of several DNSBL filters.

Last edited by jggimi; 26th May 2020 at 03:34 PM. Reason: clarity
Reply With Quote
Old 26th May 2020
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Thanks for the response jjgimi. I think I just needed some reassurance.
Thanks for the names of different block lists. It is good starting point.
__________________
Signature: Furthermore, I consider that systemd must be destroyed.
Based on Latin oratorical phrase
Reply With Quote
Old 27th May 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I just checked MX toolbox, and it currently reports on 95 domain blacklists and 87 IP blacklists. Which is a lot.
Reply With Quote
Old 8th June 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I just happened to review mail logs logs today, and noticed that yesterday a spambot (at an apparent residential ISP) attempted to connect 27 times within a single 4-second period. No idea why it stopped at 27. It came, it hammered, it left.

I filter with three different DNSBLs -- Spamhaus Zen, GBUdb, and SpamCop -- and the IP address was listed at GDUdb. It also happened to be the first time since I enabled filtering that GBUdb showed a positive listing while the other two blacklists did not.
Reply With Quote
Old 20th June 2020
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

How do you deal with ICMP packets?
I've read about it and it seems that rate-limiting is best for ICMP packets, because in the past it was quite easy to DoS different OSes with ICMP packets. Blocking is another option, but it probably isn't the most RFC compliant way.
I don't know how much should I rate-limit them and where. ICMP rate-limiting on ingress only should prevent DoS, but may be not enough to prevent sending large number of ICMP packets (packet reflection), because they may be generated by OS rejecting UDP packets. On the other hand rate-limiting on egress only does not prevent against DoS and I can just not reject UDP packets and drop them instead. I lean towards rate-limiting on both ingress and egress.
__________________
Signature: Furthermore, I consider that systemd must be destroyed.
Based on Latin oratorical phrase
Reply With Quote
Old 20th June 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

ICMPv6 is an operational requirement for IPv6, and my main mail server is IPv6/IPv4. I don't block ICMPv6 and I have never had any issues with ICMP (v4). I would look into rate limiting ICMP it if my server is ever attacked.
Reply With Quote
Old 21st June 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Currently, my DNSBL filter is working very well, and spam is at very manageable levels.

I now only report spam to the responsible ISP ("LART") when the sending IP address has a neutral or positive Mailspike reputation, or no reputation at all. If it has a negative reputation, then I assume my report will bounce or be ignored, and I will not bother to send an abuse report. SpamAssassin provides this reputation level as part of its analysis reporting, so it makes it easy for me to determine whether to LART or not. When LARTing, I use a simple script that automates much of the report generation, minimizing my effort when I do report.

LART is an acronym which these days just means "Report to the responsible ISP." Hstorically it meant bringing the problem of spam to the attention of those who are unaware but with awareness might be able to do something about it: "Loser Attitude Readjustment Tool."

Last edited by jggimi; 21st June 2020 at 03:20 PM. Reason: clarity
Reply With Quote
Old 21st June 2020
e1-531g e1-531g is offline
ISO Quartermaster
 
Join Date: Mar 2014
Posts: 628
Default

Quote:
Originally Posted by jggimi View Post
ICMPv6 is an operational requirement for IPv6, and my main mail server is IPv6/IPv4. I don't block ICMPv6 and I have never had any issues with ICMP (v4). I would look into rate limiting ICMP it if my server is ever attacked.
Thanks for the response.
I can enable IPv6 for my VPS, but my ISP and many other ISPs in Poland does not support IPv4 and I recall Gmail also have not accepted mail sent by other server (different domain) via IPv6 protocol (fix for Postfix), so I operate IPv4 server and I am mainly interested in ICMP(v4).
When it comes to packet reflection some Paul Vixie's article was eye opening for me. It is probably hard to completely defend against being taken advantage of packet reflection, but at least I will try to limit it. Not only to defend my VPS, but also to not help DDoS others services.
__________________
Signature: Furthermore, I consider that systemd must be destroyed.
Based on Latin oratorical phrase
Reply With Quote
Old 21st June 2020
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
Originally Posted by e1-531g View Post
...I recall Gmail also have not accepted mail sent by other server (different domain) via IPv6 protocol...
Most of my incoming Gmail arrives via IPv6. They appear to prefer it to IPv4 when available.

Both ICMPs are state-managed by PF, so stateful tracking options are available to rate limit them, such as:
Code:
pass proto {icmp icmp6} keep state (max 5000)
I haven't needed this, so far.

Last edited by jggimi; 21st June 2020 at 03:11 PM. Reason: typo
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Co-Opted Small Business Servers shep News 0 11th June 2016 04:13 PM
HTTPS-crippling attack threatens tens of thousands of Web and mail servers J65nko News 8 22nd May 2015 06:50 AM
Virtual domains on multiple mail servers running Exim4 as MTA satimis General software and network 10 27th November 2008 02:42 PM
NFS your thoughts rex FreeBSD General 4 24th September 2008 03:32 AM
MTA thoughts cajunman4life General software and network 37 8th June 2008 07:37 PM


All times are GMT. The time now is 06:43 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick