New mail server setup

Michael K. Smith - Adhost mksmith at
Thu Sep 17 16:09:50 UTC 2009

Hello Steve:

I'll try to answer your questions in line.  

> > Another approach would be a cluster of Postfix servers and Dovecot
> > servers behind PF load balancers.  We have 3 "POP" servers
> (IMAP/POP), 9
> > Mail Servers, 2 Defer servers and 5 Filter servers that process over
> 20
> > million messages a day without a blip.  We can take individual
> servers
> > out of the pool for maintenance, etc.  Everything is fed to a set of
> > redundant NAS for the data storage and common configuration files.
> Thanks Mike,
> I'm interested to learn a little more about your setup. I was going to
> take it off-list, but if you can provide some further details, it
> probably add long-term value to keep it here.
> So, a couple of questions:
> - can your PF load balancers 'sense' when one of the Postfix/Dovecot
> units are down, or is this a manual change in config to prevent any
> time-out conditions?

Not natively.  When we initially implemented this setup, ifstated wasn't
up to snuff, so we wrote some PERL scripts that make connections to the
required ports and, if no connection is established, pull the server
from the table and send us an alarm.  We also have scripts so that we
can pull servers out when we're doing maintenance.
> I like this load balancer idea. In my environment, it would be trivial
> to set up a couple of them, throw Quagga on them, and integrate them
> directly into our iBGP setup. On the other side, I could use VRRP or
> the
> like to ensure redundancy from front to back.

We use two PF boxes and CARP with PFSync for failover, so no dynamic
protocols are needed.  
> - do the Postfix/Dovecot servers communicate with each other, or are
> they simply stand-alone units that don't know/care that they have
> peers helping with the workload?
They are standalone.  All of the user authentication is handled from a
centralized database, so there are no local credentials stored on the

> - are your filter servers in front of, or behind the load balancers
> (iow, is all of your inbound email passed through the balancers, and
> then filtered/processed/delivered in behind them)?

They are behind the PF boxes.  We have other hooks in PF that we use to
block SPAM in PF, including Cloudmark and some custom stuff that looks
for multiple mails to non-existent addresses.  We also use the overload
tables for abusive connections.
> - how do all of the pieces communicate with the NAS...NFS?

Yes.  Originally we used TCP but we found performance to be much better
with UDP.  NFSv3 by the way.
> - could you share a small snip of your PF config in relation to
> load-balancing, so I can get a bit of a better understanding config-
> wise
> on how that piece hangs together? (I've never used PF, only IFPW ;)

That might be difficult because it's about 720 lines.  :-)  Here are
some highlights, though.

1) Our customers use for everything - SMTP, POP and
IMAP.  We use redirects in PF so that traffic coming in on the
associated ports goes to the appropriate servers.
2) We have our load-balanced DNS servers behind the same PF boxes so we
localize the tons of DNS queries related to mail.
3) We do a lot of our rejecting in PF, including Spamhaus, Cloudmark,
check scripts for Phishing, Porn and Viruses, as well as our own list of
"Nefarious" IP's culled from various sources.  When traffic matches
these originators, we send them to mail reject servers that send out a
550 message with the group name so we can find false positives more
4) Because 3 does have false positives, we have a whitelist that we can
add to that will pass traffic to the mail servers before they match
against any of the tables in 3.
5) We use POP before SMTP, so once we authenticate a user to send, their
IP address is also added to an allow table.
6) The filter servers are load balanced to and from the mail servers so
we can take them in and out of their pool for maintenance.

If you have a particular scenario you're thinking about I could help you
with the rules to make it work.



More information about the freebsd-questions mailing list