Building a Scalable Mail System?

clusteredMail asks: "I work for a small ISP that up until now has survived with single servers for most critical roles, including the mail server. We are planning to introduce multiple mail servers (primarily for email collection via POP3 and IMAP) and want to put in place the most scalable, resistant to failure system that we can manage. Everything is currently running on one or another flavour of Linux. In my mind, the ultimate scenario would be to have some sort of distributed/clustered file system between the multiple machines, so that any user could log onto any server, and the loss of a single server would not cause downtime for any group of users. Has anyone in the Slashdot community had to put together a system like this using Linux and Open Source Software? If so, how did you fare and what were the major stumbling blocks?" "So far, the plan is to split up the mail accounts between multiple servers and use some sort of connection proxy to sort out which account logs into which server but this seems like a rough approach. The disadvantage to this setup: if one server fails all the users who have accounts on that machine will be in the dark, email-wise."

Comments

Check out Perdition

[Matts]

If it's IMAP scalability you want then you should look into Perdition, particularly their article on clustered mail server farms. This is in use in a lot of high performance, high scaling environments.

Re:Check out Perdition

[Bronster]

Alternatively, check out nginx. Sure you have to wade through the Russian, but the configuration syntax is pretty simple and it's easy enough to build.

It uses epoll. We replaced a perdition proxy that was seriously loading two servers with a single 8 process nginx instance that's not even breaking a sweat. It's amazing what the change from 32000 process down to 8 processes can do on a busy site! The two frontend machines are now configured with heartbeat to get full failover of IP addresses. Downtime appears to be on the order of 1-2 seconds with an orderly cutover and probably about 10 seconds for a total host failure.

Cyrus supports replication now, which is a good way to handle the backends. I'd say more about it, but I haven't actually finished configuring the full failover system yet for this - lots of gating logic required to make sure two machines don't both believe they're master for a bit!

Er, but why would I help you anyway, you're the competition ;)

(I work for FastMail.FM btw)

You mean, like Cyrus?

[Just Some Guy]

Cyrus is the most amazingly flexible POP/IMAP server I've run, and it supports clustering (a "Murder") out of the box.

It's not for the faint of heart, but only takes a couple of "a-ha!" moments to go from lost to competent. Good luck!

Re:You mean, like Cyrus?

[vitroth]

A Cyrus IMAP Murder isn't a clustered system, its exactly the multiple servers with a proxy that the original post was describing. (Note: I work for the CMU IT department, I'm familiar with the way this works, but I don't work for the email group.)

However if you used the Murder as your frontend for clients, and applied fairly standard high availability tactics to the individual backends you could achieve clustering. Make each backend server a redundant load balanced virtual server, then make the Murder know about the mailbox locations on the virutal systems.

I'm sure it could be done, but its definitely not something that Cyrus does out of the box.

In practice the multiple servers w/ proxy has been good enough for CMU. With good hardware for the backend servers, and good RAID arrays, hardware failures are rare.

The trick is...

[Quaoar]

...to make the mail modular so that people of different sizes can wear the armor without the need for re-forging.

There is always

[jellomizer]

You can Point you MX record to google gmail service and modify the pop addresses to point to google mail too.

I am in the process of doing this...

[charlesnw]

I run an open source project that is building an exchange replacement. http://www.thewybles.com/~charles/oser is the project homepage. It will be highly available (supporting both hardware (cisco/webmux load balancers) and software based load balancing. Along with a whole host of other groupware functionality. I have done high availability e-mail solution deployments. I am in the SoCal area but am willing to travel if necessary. There are others who can help you as well. Your choice. My blog covers a lot of the progress of the project and details. I would be happy to work with you to complete this task. Just e-mail me and we can work out an arrangement.

For starters...

[metamatic]

...don't touch mbox format. Whatever software you choose, make sure it uses Maildir.

Re:For starters...

[subreality]

...don't touch mbox format. Whatever software you choose, make sure it uses Maildir.

You shouldn't state this as an absolute, because it's not. You also need to give reasons WHY to use maildir.

An example exception case: We had an application where thousands of very small emails needed to be delivered to a single mailbox every minute. They all get picked up every minute by POP, and all messages are deleted every cycle. mbox is *vastly* better in this scenario, because you don't have to create all the files, move them around a few times, stat large dirs every time POP runs, etc. With mbox, all the delivery threads become sequential, so you cut down seek overhead, and the POP read becomes a single large file read, which is far faster. You also cut way down on metadata updates, and caching works better.

mbox shines in this scenario, and it's not that uncommon. Many customer service apps work like this.

In the situation of handling many users's email in a scalable system Maildir is usually better (NFS-safe, concurrent delivery, efficient individual message deletion, etc), but you've not even considered the other range of things available. MH and database backends come to mind. Each has their good and bad points.

open SMTP relays

[JeanBaptiste]

cheap, reliable, scalable, and there doesn't seem to be any shortage of them.

now if I could only figure out how to receive...

My solution

[subreality]

It's been a while since I built a mid-size email system, but the last time I did it I used:

Data stores were maildirs on NetApps
SMTP servers running Postfix
IMAP servers running Courier IMAP
Logins via NIS
IMAP and SMTP failover by means of load balancers

The SMTP and IMAP servers get NIS-distributed automounter tables, so everyone's homedir is available everywhere. The load balancers distribute the load out to the SMTP and IMAP servers, and work around any that fail. Mail comes into the SMTP servers, and Postfix delivers to maildirs in the users' homedirs. Any SMTP server can deliver to any user. Users log in with IMAP on the Courier IMAP servers. Again, all homedirs are everywhere, so it doesn't matter which server they hit.

Adding capacity at any point is easy - you just add more servers of the appropriate type when you need more. IMAP and SMTP are fully redundant. Load balancers usually only operate in failover pairs, but you can add more A records in DNS for more LB pairs if you need it.

The one sticky point is the data stores on the NFS servers. Adding capacity is easy (just add more servers). but there's no easy way to make this fully redundant. See notes for more.

So there you have it. That'll scale to a pretty large system, and it's simple to implement. It's not THE MOST scalable system, but if you have to ask, this is probably sufficient for your needs.

Notes:

You must use maildirs, not mbox. Maildirs perform very well even on NFS, because there can be multiple simultaneous readers and writers. mbox requires locking.

With NetApp, or Red Hat Cluster Server, or any other cluster NFS server, you can make the head end redundant, so your disk shelf becomes the last single point of failure. If you run RAID 1+0, you can have all the disks mirrored across two shelves, so at least the hardware is completely redundant. However, there are still rare, but possible failure modes. STONITH is, ultimately, a problem that has no perfect solution. (Look it up if you're not familiar with STONITH.)

NetApp makes very reliable NFS servers. Even in single head configurations my uptime experience has been incredibly good. Dual head is even better. But they're god awful expensive. There are other ones you can buy at all different price points. Clustered file systems like Coda sound really sexy, but they're still half baked. Lustre http://www.lustre.org/ might work well, but it wasn't available when I last did this, so I can't say. Choose what's appropriate to your needs and budget.

I used NIS. These days LDAP is more fashionable. Make your LDAP server redundant of course.

You need redundant networks. In the simplest case, put half of each type of servers (IMAP, SMTP, LB, NFS) on two different switches.

I never bothered with POP, but you can get POP servers for maildirs, too.

Configure your load balancers to balance per session - IE, if a user creates multiple IMAP connections, they all go to the same server. This helps keep down the number of NFS mounts, LDAP requests, etc.

Software opinions: I like Postfix and Courier. They're simple, robust, flexible enough for most situations, and perform very well. Cyrus also has a good following in the large-scale arena, but does things different. Qmail's non-OSS license prevents people from releasing versions that strip out djb's quirky way of doing things, which is why I left it for Postfix (and never looked back). Sendmail doesn't suck as much as it used to, but I haven't really seen why I SHOULD use it these days either. Any of these can be made to work, though, so use whatever you're comfortable with.

Tip for any email system: outright reject (IE, don't accept at all, don't send to someone's spam folder) as much spam as you can. If 90% of your mail is spam, and you reject the 90% most-likely-spam (delivering the other 10% more questionable stuff to a spam folder), you've just increased your mail performance and disk space by > 5x.

Good luck!

A Real-world Big Design

[jjgm]

I recently designed and built a mail system for a six-digit ISP userbase.

Before I feed you the design, let me tell you a *crucial* concept that you must carry with you at all times.

EMAIL SYSTEMS ARE PROTOCOL SPEAKERS BETWEEN USER DIRECTORIES AND STORAGE.

Read that and inwardly digest it before you even start to design your system.

For the design, first, I'm going to proselytize a particular piece of software.

DOVECOT IS THE FREE POP/IMAP SERVER OF THE FUTURE. It leaves the Cyrus codebase rotting in the slime. It already kicks Courier's butt in performance and ease of deployment. It's beautifully coded; it has the most elegant authentication architecture; it's exceptionally fast. It isn't complete yet but it's featureful and stable enough that I have successfully deployed 1.0-betas into production. http://www.dovecot.org/ for the last IMAP server you'll ever need.

Here is the design:

1 x OpenLDAP 2.3 master server
2 x OpenLDAP 2.3 read-only replicas
2 x world-facing mail servers running Postfix 2.3
4 x mail scanning servers running amavisd-new 2.3.3, ClamAV, SpamAssassin, Sophos SAVI and Sophos PMX-ENGINE. LMTP in from the mail front-ends; ESMTP out to the mail storage.
2 x mail storage front-ends running Postfix 2.3 and Dovecot IMAP/POP3 1.0-beta. These servers also run mysql for amavisd-new quarantine and squirrelmail user options. Actual storage is over NFS to the NetApps. Using Dovecot's Sieve-based delivery agent for server-side filtering.
2 x Squirrelmail webmail servers. We have our own skin, and our own sqm plugins as the user interface to our various system options - which are all in LDAP. We have integrated MailZu into sqm as a quarantine view/release interface.
2 x NetApp FAS3020c heads w/4TB NFS storage allocated to mail.

Everything is load-balanced using foundry hardware LBs. It's very high-throughput and very reliable. It's also easy to monitor (we're using Nagios).

Base OS is Debian Sarge with applicable backports. I'd prefer FreeBSD but this happens to be a Debian shop, and I wasn't out to change their world, just their mail system.

Probably the most borderline item is mysql's performance as a quarantine DB; however much RAM and index/query tuning we throw at it, I'm yet to be satisfied with InnoDB's performance on this 100GB+ INSERT-heavy database.

If I could change one thing about it, it'd be to use the extremely pretty and surprisingly good value @mail (a commercial choice) rather than SquirrelMail. I'd also consider Fedora Directory Server over OpenLDAP, but it wasn't looking ready for this design at the time.

I have to say there is some bad advice in this thread; now for the hatchet:

Cyrus: difficult to configure, doesn't support shared storage, horribly ugly codebase, and has some nasty-ass failure modes.
Qmail: stale, poorly integrated MTA software from the bitchiest developer in town.
Sendmail: doesn't scale. Even the developers think so, which is why Sendmail X is a rip-off of postfix.
Communigate Pro: if I don't get to futz with the source for integration and value-add, I'm not interested.
GFS/GPFS: you don't need the complexity or interesting failure modes of shared-block-storage filesystems. Stay away.
Linux NFS: isn't reliable enough. We've had problems with data corruption to Linux NFS, both kernel and userland. Right now the only NFS server implementations I trust are NetApp's and Solaris's. No doubt the Linux one can/will improve, or already has, but trust is a hard thing to build :). Plus NetApps are shiny, marvelously reliable, and I love their support.