Slashdot Mirror
Building a Scalable Mail System?
clusteredMail asks: "I work for a small ISP that up until now has survived with single servers for most critical roles, including the mail server. We are planning to introduce multiple mail servers (primarily for email collection via POP3 and IMAP) and want to put in place the most scalable, resistant to failure system that we can manage. Everything is currently running on one or another flavour of Linux. In my mind, the ultimate scenario would be to have some sort of distributed/clustered file system between the multiple machines, so that any user could log onto any server, and the loss of a single server would not cause downtime for any group of users. Has anyone in the Slashdot community had to put together a system like this using Linux and Open Source Software? If so, how did you fare and what were the major stumbling blocks?"
"So far, the plan is to split up the mail accounts between multiple servers and use some sort of connection proxy to sort out which account logs into which server but this seems like a rough approach. The disadvantage to this setup: if one server fails all the users who have accounts on that machine will be in the dark, email-wise."
If it's IMAP scalability you want then you should look into Perdition, particularly their article on clustered mail server farms. This is in use in a lot of high performance, high scaling environments.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
MIT, an organization which you'd think would have a handle on this sort of thing, simply has a bunch of independent servers and assigns accounts to a specific one. One user might use po10.mit.edu for POP/IMAP, another might use po2.
Of course, that might just be because the IT department at MIT does not take advice from the faculty and students, and just generally sucks.
It's not for the faint of heart, but only takes a couple of "a-ha!" moments to go from lost to competent. Good luck!
Dewey, what part of this looks like authorities should be involved?
...to make the mail modular so that people of different sizes can wear the armor without the need for re-forging.
I'll form my OWN solar system! With blackjack! And hookers!
You can Point you MX record to google gmail service and modify the pop addresses to point to google mail too.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
I would recommend deploying an LDAP-enabled directory server as the foundation for everything else. Almost every other service can leverage the directory for pulling various information about each user. You can really help yourself down the road by making your directory server _the_ single authoritative source of email-related customer information.
Distributed authentication system (nis, kerberos, ldap, mysql, whatever), and storage (netapp. don't even think about using anything else, just get a damn netapp). have multiple everything and redundancy redundancy redundancy.
don't forget testing it. get a system up and running, then see what happens if you power off the master. Hint: it shouldn't change anything.
http://www.nrg4u.com/
From commercial experience: Multiple MTA machines fine. Multiple MUA (IMAP, POP, Webmail) machines fine. Don't use a clustered filesystem; use NFS. All the software of note use locking which plays well with NFS.
Scaling can be done easily by adding more NFS boxes and managing the directory structure with links or whatnot.
I run an open source project that is building an exchange replacement. http://www.thewybles.com/~charles/oser is the project homepage. It will be highly available (supporting both hardware (cisco/webmux load balancers) and software based load balancing. Along with a whole host of other groupware functionality. I have done high availability e-mail solution deployments. I am in the SoCal area but am willing to travel if necessary. There are others who can help you as well. Your choice. My blog covers a lot of the progress of the project and details. I would be happy to work with you to complete this task. Just e-mail me and we can work out an arrangement.
Charles Wyble System Engineer
See e.g. article about a university system. Also Cyrus supports load balancing and failover via murder, although backends (the actual file stores) still remain single points of failure.
...don't touch mbox format. Whatever software you choose, make sure it uses Maildir.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Multiple servers all looking at the same files? That's what shared storage is all about.. Fibre Channel, iSCSI, etc... Look at GFS from RedHat or any of the others like IBM's recently announced GPFS. Fedora or CentOS for RedHat's GFS on a more reasonable budget.
cheap, reliable, scalable, and there doesn't seem to be any shortage of them.
now if I could only figure out how to receive...
Scales beyond anything you can be doing, and has the features you're going to want. Check it out: http://stalker.com/
Run it on Linux, it just works...
there are many elements to this problem
o when a user points their mail client at the server do you want one address ?
if yes
then you want to invest in equipment/software load balance the IMAP/POP3/HTTP sessions across mail servers And shared require all serve out same same client data
and
you want several/all mail servers to be able serve out all the accounts then you want a shared stroage backend
(either file system OR database wise)
else
simple option many mail servers all acting exactly the same with exactly the same config only differant servers used by differant clients and maybe a backup
lotus notes has had options like this for a while so has exchange to a lesser degree these solutions base it around IMAP/POP3 multiplexing but exchange and notes Aggregate as well if needed
if you want THE unix mailserver talk to Sun and their IMAP (java mail server system or whatever they are calling it now...) works
regards
John Jones
They have this thing called DNS now, where you can create 'aliases' for all of your servers when someone tries to connect to 'mail.isp.net'. It's kind of interesting stuff, one of these days it might take off.
I want to delete my account but Slashdot doesn't allow it.
I have been running CommuniGatePro mail servers for years. They have all the features you're looking for and more. The main thing I love about CommuniGate is the fact that I have one application that is the MTA, IMAP server, POP Server, and WebMail all in one. No dependencies. No futzing around with PHP or config files for half-a-dozen different applications. And it also supports clustering. A low-end cluster would consist of 2 machines with a NFS or CIFS/SMB backend for the storage. They also support so many operating systems it's not even funny. Solaris, Windows, Linux....even OS/2! You can grab a fully-functional trial off their site and have it up and running in minutes. Check it out, you won't be sorry. Their stuff scales from small systems to really huge ones with millions of accounts. For example - UC Berkeley's mail system is CGP. You can check 'em out at http://stalker.com/.
... I like to keep an open mind, but not so open that my brains fall out. - Judge Harry Stone, Night Court
This page might have some ideas.. Also see this page for a howto on setting up the mail server.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
It costs, but since I've been looking into mail servers lately I can let you know Scalix has an enterprise edition that run on multiple servers.
Please go here instead:
Scalix Comparison
and does it shift 500 light users onto a differant server from the 1 heavy user is hogging all the cpu or the box has crashed/error ?
If you have a lot of data, then you can choose a scalable system like IBRIX and then use stateful load balancers between each of the POP3/IMAP servers. When you get to multiple nodes on the same filesystem, you have two problems: synchronization between nodes and locking.
Note that the Oracle Clustered Filesystem v2 has now been merged with the mainline kernel.
I would recommend RadWare http://radware.com/ or f5 http://f5.com/ to load balance the traffic to multiple IMAP and/or POP back end servers.
You can even cluster the load balancers...
Cybie! aka Ralph Bonnell
The best solution I have found is to put generic smtp/imap/pop servers behind layer 4 switching.
Using this the only thing your servers need in common is backend storage that you can easily mount off NFS etc.
A properly configured, customized, qmail/vpopmail cluster is a beauty to behold. Unfortuantely, it takes the better part of a month to get up to speed on how the system works, and it will be many months overall before you really feel "comfortable" with how it works (longer if you're coming solely from a sendmail background).
That being said, it's also rock-solid, extremely fast when properly configured, and more flexible than you can imagine.
We currently use a single RAID-10 NFS and MySQL DB system handling the backend, with 5 cluster servers in front of it, each of them able to perform any number of roles. (We had a load balancer in front of them at one point, but it actually more just got in the way than anything else.) A sixth box handles all DNS requests for the servers, and we'll be bringing a 7th up soon to offload some of the spam processing from the three that currently run our asnychronous processing code. The cluster boxes are cheap MicroATX Athlon XP 3000+ machines with 2 GB of RAM. I've seen each box take well over a 100 simultanous SMTP connections without CPU being noticably affected. Current 1 does webmail, 1 does incoming MX, 1 does POP3/IMAP, 1 is for development and servers IMAP to the webmail box, and 1 is running SMTP, 587, and SMTP-SSL.
When properly administered, I think it beats anything out there. However, if you can't afford the time and 3am-bang-your-head-against-your-monitor agony, I'd suggest one of the other solutions people have mentioned here.
My $.02
Hire a Linux system administrator, systems engineer,
It's been a while since I built a mid-size email system, but the last time I did it I used:
Data stores were maildirs on NetApps
SMTP servers running Postfix
IMAP servers running Courier IMAP
Logins via NIS
IMAP and SMTP failover by means of load balancers
The SMTP and IMAP servers get NIS-distributed automounter tables, so everyone's homedir is available everywhere. The load balancers distribute the load out to the SMTP and IMAP servers, and work around any that fail. Mail comes into the SMTP servers, and Postfix delivers to maildirs in the users' homedirs. Any SMTP server can deliver to any user. Users log in with IMAP on the Courier IMAP servers. Again, all homedirs are everywhere, so it doesn't matter which server they hit.
Adding capacity at any point is easy - you just add more servers of the appropriate type when you need more. IMAP and SMTP are fully redundant. Load balancers usually only operate in failover pairs, but you can add more A records in DNS for more LB pairs if you need it.
The one sticky point is the data stores on the NFS servers. Adding capacity is easy (just add more servers). but there's no easy way to make this fully redundant. See notes for more.
So there you have it. That'll scale to a pretty large system, and it's simple to implement. It's not THE MOST scalable system, but if you have to ask, this is probably sufficient for your needs.
Notes:
You must use maildirs, not mbox. Maildirs perform very well even on NFS, because there can be multiple simultaneous readers and writers. mbox requires locking.
With NetApp, or Red Hat Cluster Server, or any other cluster NFS server, you can make the head end redundant, so your disk shelf becomes the last single point of failure. If you run RAID 1+0, you can have all the disks mirrored across two shelves, so at least the hardware is completely redundant. However, there are still rare, but possible failure modes. STONITH is, ultimately, a problem that has no perfect solution. (Look it up if you're not familiar with STONITH.)
NetApp makes very reliable NFS servers. Even in single head configurations my uptime experience has been incredibly good. Dual head is even better. But they're god awful expensive. There are other ones you can buy at all different price points. Clustered file systems like Coda sound really sexy, but they're still half baked. Lustre http://www.lustre.org/ might work well, but it wasn't available when I last did this, so I can't say. Choose what's appropriate to your needs and budget.
I used NIS. These days LDAP is more fashionable. Make your LDAP server redundant of course.
You need redundant networks. In the simplest case, put half of each type of servers (IMAP, SMTP, LB, NFS) on two different switches.
I never bothered with POP, but you can get POP servers for maildirs, too.
Configure your load balancers to balance per session - IE, if a user creates multiple IMAP connections, they all go to the same server. This helps keep down the number of NFS mounts, LDAP requests, etc.
Software opinions: I like Postfix and Courier. They're simple, robust, flexible enough for most situations, and perform very well. Cyrus also has a good following in the large-scale arena, but does things different. Qmail's non-OSS license prevents people from releasing versions that strip out djb's quirky way of doing things, which is why I left it for Postfix (and never looked back). Sendmail doesn't suck as much as it used to, but I haven't really seen why I SHOULD use it these days either. Any of these can be made to work, though, so use whatever you're comfortable with.
Tip for any email system: outright reject (IE, don't accept at all, don't send to someone's spam folder) as much spam as you can. If 90% of your mail is spam, and you reject the 90% most-likely-spam (delivering the other 10% more questionable stuff to a spam folder), you've just increased your mail performance and disk space by > 5x.
Good luck!
The typical way to set this up is:
High availability redundant NFS servers for storing the mailbox data and user information.
One or more machines mounting this file-system for handling POP, IMAP, and SMTP from accounts and mailfolders off the NFS server.
Webmail can be tricky because you need to make sure that either users always hit the same machine for webmail during a session, or session information is shared among the cluster. LVS systems can handle either of these scenarios, so it's not a problem, just something you have to be aware of.
LVS systems up front, again running High Availability which do load-balancing and automatic removal of failed servers. These are the machines that have the IPs which your customers contact, and then get spread across the real machines in the middle layer above.
This sort of solution works really well, and we have deployed it for customers of ours with good results. You can get started for only $5k to $10k worth of hardware and if you're building this from scratch it will probably only take you around 100 hours. If you have experience with this sort of setup it can take as little as 10 to 20.
If $5k to $10k for hardware is out of your budget, you probably shouldn't be looking at this sort of solution. Individual stand-alone servers or even a single pointy box, possibly with high availability, is probably where you want to be in that case.
linux-ha.org is the place to go for High Availability software on Linux.
Sean
Scalemail looks like its a good start. http://www.inoi.fi/open/trac/scalemail/ "A scalable (but not fully highly available, atleast not yet) virtual domain system for handling mail for many users, based on Postfix, LDAP and Courier-IMAP."
I recently designed and built a mail system for a six-digit ISP userbase.
:). Plus NetApps are shiny, marvelously reliable, and I love their support.
Before I feed you the design, let me tell you a *crucial* concept that you must carry with you at all times.
EMAIL SYSTEMS ARE PROTOCOL SPEAKERS BETWEEN USER DIRECTORIES AND STORAGE.
Read that and inwardly digest it before you even start to design your system.
For the design, first, I'm going to proselytize a particular piece of software.
DOVECOT IS THE FREE POP/IMAP SERVER OF THE FUTURE. It leaves the Cyrus codebase rotting in the slime. It already kicks Courier's butt in performance and ease of deployment. It's beautifully coded; it has the most elegant authentication architecture; it's exceptionally fast. It isn't complete yet but it's featureful and stable enough that I have successfully deployed 1.0-betas into production. http://www.dovecot.org/ for the last IMAP server you'll ever need.
Here is the design:
1 x OpenLDAP 2.3 master server
2 x OpenLDAP 2.3 read-only replicas
2 x world-facing mail servers running Postfix 2.3
4 x mail scanning servers running amavisd-new 2.3.3, ClamAV, SpamAssassin, Sophos SAVI and Sophos PMX-ENGINE. LMTP in from the mail front-ends; ESMTP out to the mail storage.
2 x mail storage front-ends running Postfix 2.3 and Dovecot IMAP/POP3 1.0-beta. These servers also run mysql for amavisd-new quarantine and squirrelmail user options. Actual storage is over NFS to the NetApps. Using Dovecot's Sieve-based delivery agent for server-side filtering.
2 x Squirrelmail webmail servers. We have our own skin, and our own sqm plugins as the user interface to our various system options - which are all in LDAP. We have integrated MailZu into sqm as a quarantine view/release interface.
2 x NetApp FAS3020c heads w/4TB NFS storage allocated to mail.
Everything is load-balanced using foundry hardware LBs. It's very high-throughput and very reliable. It's also easy to monitor (we're using Nagios).
Base OS is Debian Sarge with applicable backports. I'd prefer FreeBSD but this happens to be a Debian shop, and I wasn't out to change their world, just their mail system.
Probably the most borderline item is mysql's performance as a quarantine DB; however much RAM and index/query tuning we throw at it, I'm yet to be satisfied with InnoDB's performance on this 100GB+ INSERT-heavy database.
If I could change one thing about it, it'd be to use the extremely pretty and surprisingly good value @mail (a commercial choice) rather than SquirrelMail. I'd also consider Fedora Directory Server over OpenLDAP, but it wasn't looking ready for this design at the time.
I have to say there is some bad advice in this thread; now for the hatchet:
Cyrus: difficult to configure, doesn't support shared storage, horribly ugly codebase, and has some nasty-ass failure modes.
Qmail: stale, poorly integrated MTA software from the bitchiest developer in town.
Sendmail: doesn't scale. Even the developers think so, which is why Sendmail X is a rip-off of postfix.
Communigate Pro: if I don't get to futz with the source for integration and value-add, I'm not interested.
GFS/GPFS: you don't need the complexity or interesting failure modes of shared-block-storage filesystems. Stay away.
Linux NFS: isn't reliable enough. We've had problems with data corruption to Linux NFS, both kernel and userland. Right now the only NFS server implementations I trust are NetApp's and Solaris's. No doubt the Linux one can/will improve, or already has, but trust is a hard thing to build
I started a free email service to compete with Gmail about 2 months after Gmail launched. For those interested, the name is Nerdshack.com.
/. about it.
At first I used Postfix and Cyrus, but I found it to be a nightmare when your talking about more than 50k accounts.
What I wanted was an email platform that integrated with ClamAV, DSPAM, supported SPF, Greylisting/Blacklisting/Whitelisting, and was all controlled from a MySQL database. I also wanted it to support SSL, and clustering.
Frankly I didn't find anything. So I wrote my own. This may not be your cup of tea, so if not I reccomend looking at DB Mail (www.dbmail.org) and Cyrus (asg.web.cmu.edu/cyrus). Both are compotent mail servers, can be built to support a large user base. The problem I had was expanding their feature sets.
As has been mentioned numerous times above, getting stock open source software to support a large user base is a huge pain. Combine that with trying to add in things like DSPAM, SPF and ClamAV, and your going to be faced with a nightmare. The system you end up with will be a kluge of hacks, custom scripts, and chewing gum. To me that seemed to much like a house cards. On top of which, most open source sytems do not handle large quota accounts very well. Run benchmarks against your favorite mail server using a 10 to 20 gig mail store for a single user. You will quickly find that even maildir struggles with that many files. (Hint, make sure you use ReiserFS at least.)
So I went the route of Gmail, Yahoo, and Hotmail. I wrote my own, and after a few early bumps in the road, its pretty solid. I've had 100% uptime for over 300 days, util last month when I moved datacenters.
Basically how I set things up is I have an Alteon AD4 load balancer that balances traffic amongst my application cluster. This app cluster runs my custom code which speaks SMTP and POP (IMAP is about 75% done), and interfaces with DSPAM, ClamAV, libspf, OpenSSL, LZW (for compression), and supports a host of other features. I even support using public key encryption (ECC) to store messages on disk with your public key, and then encrypt your private key with AES256 using your password as the key. Its seemless to the end user, but guarantees privacy while the mail is on my servers. I even created a point system to allow me to automatically block IP addresses that attempt dictionary attacks, etc (though its disabled at the moment). Each server caches everything it can to reduce database load, and uses a connection pool for retrieving messages and running queries. I wrote my server in C, so its very, very fast.
These app servers store user information, preferences, etc, in a MySQL database. The actual messages are stored on message storage servers using a custom algorithim, and protocol for speed. Every message is stored on two servers, with both locations stored in the database. Needless to say my system rocks. Each Dell 1650 server can support close to 1000 simultaneous connections while using less than 10% of the CPU.
What I'm working on now is the IMAP server, integration with Memcached, and moving configuration settings into an XML file. Right now config settings are DEFINE parameters, which means changing anything requires a recompile. I've also found that my database is the bottleneck, so I want to offload as much as I can to Memcached. Checking whether an email address exists, or using my custom point system with the database is too inefficient, so I hope Memcached will help.
I've thought about releasing the source under the GPL, but I don't think its quite ready. I want to at least get config settings into an XML file first. I'd also like to find a company to sponser my development, but that hasn't happened yet. (I still have a day job.)
Executive summary. Its always more fun to write your own, and then post to
The best way to do it is to buy a NAS box and use Maildir to store the mail (so you don't have to worry about NFS file locking issues) then use replicated LDAP servers (for redundancy) to store the account details, one DNS entry for your SMTP server (which you can later use round-robin entries to scale to multple servers) and another DNS entry for your POP/IMAP connections so you can split them off onto a separate machine. I'd use postfix for the SMTP server and Courier IMAP for POP/IMAP. And of course, you need SpamAssassin and probably clam-av or some other anti-virus solution.
The nice part about this is that you can originally start this all on a single machine, and migrate the different components off as the system load increases. You probably want to start with a dedicated NAS box though, even if it's just a regular Linux box running software RAID and NFS, as any new machines are going to need heavy remote disk access to the mailspools.
Wow, that's really interesting. I work at FastMail.FM and we've built our system on Cyrus as the backend, which has lots of advantages but also some big disadvantages.
We use open source software throughout our system and contribute back most of our changes (where they actually have some utility outside our little world, 50 line perl programs that just query out database for status information need not apply - and we wouldn't want to inflict our web framework on the world. It certainly doesn't need another web framework with a steep learning curve and funky special cases!)
Right as I type this (or at least when I stop typing and get my arse back to working on what I'm being paid to do) we're setting up a replicated environment with pairs of Cyrus servers:
* Dual Xeon with hyperthreading
* 8Gb RAM
* 12 SATA drives configured as 4 arrays:
[73Gb RAID 1] [73Gb RAID 1] [1.2Tb RAID5] [1.2Tb RAID5]
Each array is then split in half, with the first partition holding an active Cyrus partition and the second half holding a replicated set from its "pair" server. This spreads IO evenly.
The small RAID1 sets are faster disks and they hold the metadata partitions which get most of the IO.
We don't have these things in production yet (still huge IBM monster machines with 6Gb memory and terabytes of attached SATA storage. They're much more reliable but don't have replication, so it's a tradeoff that makes an array failure much more painful to recover from).
Every so often I do wish I had the time to build a full IMAP server from scratch with a modern indexed database engine (boo hiss in the direction of Berkeley DB) and the capability to store multiple copies of files. I've already done something like that with our virtual filesystem that provides DAV and FTP access to files on the servers, as well as websites viewing parts of your filespace - it's shiny and I can reboot any server in the VFS backend set without worrying about impacting production.
Think of redundancy. What is it supposed to achieve? One goes down the other keeps going...
Now 3 servers would be a waste. Think about it. What are the chances a casing would FAIL?
So lets put 3 servers in one box. Data has to go onto each of the 3 disks. Instantly. Theres so much IO involved. Should each email coming in have to go through the tcpip stack, through the kernel API levels, through the HAL out the driver, out the network card, through the switch and all the way back down to the disk? Using too much makes things less stable.
So lets put the 3 disks together with one chip and lets call the chip servraid. Data is copied across the disks immediately by the chip in hardware, and the OS doesnt see it. Offloading work from the CPU and OS. You can do the same for the CPU and memory, and it will resemble the IBM xSeries 236, or countless others from Sun, HP, Dell etc. Medium level servers from these companies have redundant disks, power supplies, multiple redundant CPUs and NICs and even redundant memory slots. The only common part is the chipset and the motherboard, and of course the metal chassis. Both have extremely low instances of failure. About the same as the switch that connects the 3 servers, and the mechanism on the public side that is supposed to round-robin or otherwise load balance the servers.
Just get something like the x236 (or the equivalent Opteron system from Sun). Use qmail on FreeBSD and you will not lose email.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
These multi-server solutions will increase complexity and increase the amount of sysadmin work. You could try switching to a single scalable Sun box. If you need more resources, throw in more cpus, ram, or disks. You have real reliability and scalabilty on a single machine. I've used Suns for over 10 years in a banking environment. You can sleep at night with a single critical Sun system.
Simplicity, less work, reliability, scalability.
Buy a Sun.
http://www.hserus.net/mailboxes-srs-inboxevent2004 .ppt
w ww.hserus.net/mailboxes-srs-inboxevent2004.ppt+inb oxevent2004.ppt+site:hserus.net&hl=en&ct=clnk&cd=1
http://72.14.203.104/search?q=cache:v5XWBwgqXQcJ:
That system currently handles over 41 million users, serves up POP3, IMAP, Webmail, spam and virus filtering for paying customers, and deals with over half a billion messages per day.
Every service is on physically separate hardware: MX, outbound MTAs, content filters, frontends....
I can throw myself at the ground, and miss.
Most of them are on windows. Just use a virus to re-configure.
Just use gmail for domain.
Definitely more scalable than anything that you can come up with.
you should check out mail toaster from tnpi - it's open source - built on/against FreeBSD - but a creative soul can pretty easily get it going on Linux (if Linux is that important to you).
http://www.tnpi.biz/internet/mail/toaster/
it's qmail/imap based and scales quite well in my experience.
This changes when you have multiple LDAs and MUAs sharing an NFS store. Locking matters a lot. On mbox, if you have a user that has a 500MB mailbox, and they delete the first message, the whole file has to be rewritten. While that's happening, all your SMTP servers have to queue mail for that user, because the mbox is locked. When you have a user sitting there incrementally deleting messages one by one off the top of their mail, at the same time as spammers are delivering mail to the end of the file, that can make stuff backlog pretty bad.
:)
Issue #2: some mail clients create multiple concurrent connections to the server when you're reading mail. With mbox, this hurts.
Aside from the locking issue, this creates a ton of IO on your NFS server, due to constant mbox rewrites, as well as simply having to read the whole file every time a user connects. The sheer volume of IO for mbox is really the bigger problem.
http://www-uxsup.csx.cam.ac.uk/~fanf2/hermes/doc/t alks/2004-02-ukuug/
is how the University of Cambridge do it....
lots of nice details in there
We'll be using a few front-line MX boxes running Postfix, SpamAssassin, and PureMessage (not FOSS), delivering via LMTP to several Cyrus machines, which will share SAN storage via Lustre. We'll also have a small cluster of boxes running the Horde suite for a web UI. All of this is authenticating against a four-way multi-master cluster of Fedora DS machines. Not only will that save us a bundle of cash, it will be far more stable than the breakage-prone JSMS, and, to be perfectly honest, support will be better from a bunch of 1337 h4x0r5 on a mailing list than from Sun.
Another one bites the dust
overview
Dbmail is as scalable as the database system that is used for the mail storage. In theory millions of accounts can be managed using dbmail. One could, for example, run 4 different servers with the pop3 daemon each connecting to the same database (cluster) server.
Dbmail is based upon a database. Dbmail can be managed by changing settings in the database (f.e. using PHP/Perl/SQL), without needing shell access.
Dbmail uses very efficient, database specific queries for retrieving mail information. This is much faster then parsing a filesystem.
Dbmail has got nothing to do with the filesystem or interaction with other programs in the Unix environment which need special permissions. Dbmail is as secure as the database it's based upon.
Changes on a Dbmail system (adding of users, changing passwords etc.) are effective immediately.
Other thoughts:
Running the mail store on NFS has been an unmitigated disaster. Lost mail, nfs lock problems, slow response and other issues.
Speculation about GFS and LUSTRE are just that. I have never met anyone successfully using a gfs/iscsi/god knows what in anything remotely resembling a production environment.
It would be terrific to be able to backup the customer's mail. There is no effective way to do this with nfs/maildir/what have you. A database supports this natively
I am a little disappointed that dovecot was not the base for the imap server in dbmail, but the provided imap server seems pretty full featured.
We've been using Sun's Java Enterprise System (formerly SunONE formerly iPlanet formerly Netscape blah blah) Messaging Server for the last few years for 25,000+ (and growing) users. It's proven to be very reliable and is extremely flexible. For scalability and added reliability look into their Messaging Multiplexor and High Availability options. Other significant advantages are the availability of well-written and thorough documentation and Sun's support team.
Cyrus machines, which will share SAN storage via Lustre.
Have you tested that this provides good performance? AFAIK Lustre is designed and optimized for massively parallel applications doing sequential IO on huge files (e.g. HPC apps using the MPI-IO API). Sounds like maildir (lots of tiny files) is the exact opposite.
I used to do this quite a bit. Save yourself a lot of work later by putting a bit of extra dns work up front when you create users. The nice exchange feature of being able to have the client find its mailbox when you move it to a different server is a tremendous help when scaling systems out -- and it's very easily replicated with simple dns.
For each user create two records, one for retrieving mail and one for sending it. They may initially point to the same server, but as you scale up/out they can point to other servers, requiring no client change.
Organize these as subdomains. So for every user you'd have, for example:
bob.mailin.domain.com IN CNAME imapserver1.domain.com
bob.mailout.domain.com IN CNAME smtpserver4.domain.com
Then if you need to move bob's mail spool later you just move it - no need to update the client. This approach, while not required by some client/server packages (notably outlook/exchange) is useful on almost all of them.
Zimbra is Open Source includes Perdition, Postfix and a niffy AJAX web UI. Check out the online demo. www.zimbra.com
First, Sun Java Entrprise System Messaging Server is free as part of the Solaris Enterprise System. You just have to pay for support if you want it and the mailing list that you referenced is better than paid support anyways since you are talking to the engineering team directly.
Second, stop spreading FUD. The gripes on the mailing list are mostly if not all dealing with the web end user interface (Communications Express or Unified Web Client - UWC). The LDAP/MTA/Message Store is ROCK solid and the virtual domain capibility is very simple to do which would be a BIG win for an ISP deployment.
Third, JESMS is very much role/service based so that value-added services are easy to do on a per-user and/or per-domain basis. For example, you want everyone to have their mail go through a basic anti-virus/anti-spam product, but you also can offer different levels such that one level gets run through additional scanning and a third level goes through a completely different AVAS system.
This then can extend to potentially offering calendar and IM (JABBER) based services with the installation of those servers. (Once you have the LDAP backend it becomes pretty painless to role out those services.)
Take a look at the Deployment Planning Guide for example on how it can be deployed to fit the OP requirements.
I will say that using tcpserver is a hit/miss proposition. If you don't get the memory requirements just right, you can easily take down a server with too many processes.
My primary incoming MX is a $200 machine from WAL-MART with maybe 768MB. I've eaten a few hundred thousand joe-jobs without it going down - just had to wait for the queue to empty after figuring out what account the spammer had created... *grumble*
Rick
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)