Slashdot Mirror

← Back to Stories (view on slashdot.org)

Building a Scalable Mail System?

Posted by ryuzaki0 on from the multiple-mail-servers dept.

clusteredMail asks: "I work for a small ISP that up until now has survived with single servers for most critical roles, including the mail server. We are planning to introduce multiple mail servers (primarily for email collection via POP3 and IMAP) and want to put in place the most scalable, resistant to failure system that we can manage. Everything is currently running on one or another flavour of Linux. In my mind, the ultimate scenario would be to have some sort of distributed/clustered file system between the multiple machines, so that any user could log onto any server, and the loss of a single server would not cause downtime for any group of users. Has anyone in the Slashdot community had to put together a system like this using Linux and Open Source Software? If so, how did you fare and what were the major stumbling blocks?" "So far, the plan is to split up the mail accounts between multiple servers and use some sort of connection proxy to sort out which account logs into which server but this seems like a rough approach. The disadvantage to this setup: if one server fails all the users who have accounts on that machine will be in the dark, email-wise."

1 of 109 comments (clear)

  1. My solution by subreality · · Score: 5, Informative

    It's been a while since I built a mid-size email system, but the last time I did it I used:

    Data stores were maildirs on NetApps
    SMTP servers running Postfix
    IMAP servers running Courier IMAP
    Logins via NIS
    IMAP and SMTP failover by means of load balancers

    The SMTP and IMAP servers get NIS-distributed automounter tables, so everyone's homedir is available everywhere. The load balancers distribute the load out to the SMTP and IMAP servers, and work around any that fail. Mail comes into the SMTP servers, and Postfix delivers to maildirs in the users' homedirs. Any SMTP server can deliver to any user. Users log in with IMAP on the Courier IMAP servers. Again, all homedirs are everywhere, so it doesn't matter which server they hit.

    Adding capacity at any point is easy - you just add more servers of the appropriate type when you need more. IMAP and SMTP are fully redundant. Load balancers usually only operate in failover pairs, but you can add more A records in DNS for more LB pairs if you need it.

    The one sticky point is the data stores on the NFS servers. Adding capacity is easy (just add more servers). but there's no easy way to make this fully redundant. See notes for more.

    So there you have it. That'll scale to a pretty large system, and it's simple to implement. It's not THE MOST scalable system, but if you have to ask, this is probably sufficient for your needs.

    Notes:

    You must use maildirs, not mbox. Maildirs perform very well even on NFS, because there can be multiple simultaneous readers and writers. mbox requires locking.

    With NetApp, or Red Hat Cluster Server, or any other cluster NFS server, you can make the head end redundant, so your disk shelf becomes the last single point of failure. If you run RAID 1+0, you can have all the disks mirrored across two shelves, so at least the hardware is completely redundant. However, there are still rare, but possible failure modes. STONITH is, ultimately, a problem that has no perfect solution. (Look it up if you're not familiar with STONITH.)

    NetApp makes very reliable NFS servers. Even in single head configurations my uptime experience has been incredibly good. Dual head is even better. But they're god awful expensive. There are other ones you can buy at all different price points. Clustered file systems like Coda sound really sexy, but they're still half baked. Lustre http://www.lustre.org/ might work well, but it wasn't available when I last did this, so I can't say. Choose what's appropriate to your needs and budget.

    I used NIS. These days LDAP is more fashionable. Make your LDAP server redundant of course.

    You need redundant networks. In the simplest case, put half of each type of servers (IMAP, SMTP, LB, NFS) on two different switches.

    I never bothered with POP, but you can get POP servers for maildirs, too.

    Configure your load balancers to balance per session - IE, if a user creates multiple IMAP connections, they all go to the same server. This helps keep down the number of NFS mounts, LDAP requests, etc.

    Software opinions: I like Postfix and Courier. They're simple, robust, flexible enough for most situations, and perform very well. Cyrus also has a good following in the large-scale arena, but does things different. Qmail's non-OSS license prevents people from releasing versions that strip out djb's quirky way of doing things, which is why I left it for Postfix (and never looked back). Sendmail doesn't suck as much as it used to, but I haven't really seen why I SHOULD use it these days either. Any of these can be made to work, though, so use whatever you're comfortable with.

    Tip for any email system: outright reject (IE, don't accept at all, don't send to someone's spam folder) as much spam as you can. If 90% of your mail is spam, and you reject the 90% most-likely-spam (delivering the other 10% more questionable stuff to a spam folder), you've just increased your mail performance and disk space by > 5x.

    Good luck!