Howdy, I've worked for an ISP that was growing rapidly. We shot from 20,000 to 30,000 customers in a very short period of time. Mail broke almost over nite. There was no warning sign. One day we had lots of capacity, the next... nothing. And we had a fairly beefy server: dual proc Sun Ultra Enterprise 2/300 with 1 gig of ram and gobs of disk. When it had it, the load would shoot up over 20 for hours (and we had 20 set as our sendmail throttle limit). We tuned and screwed around with the pop server and fixing sendmail, but it usually only bought us a week or so. In the end we rearcitected. The solution we came to is highly scalable, very reliable, and fairly easy to do. It was inspired by what Earthlink did a few years back (if you're a USENIX member see her e.)
Basically, we nfs mounted our users mailfiles so they were accessable across multiple machines. The biggest problem you run into here is file locking, but NFS will handle most of it for you... no need to implement.locking or anything like that. Contention is very low for user files. Each box runs cucipop for POP and sendmail for mail, each using very stock configs. We played with IMAP for a bit, and UW-IMAP is about the best around (in the free world, anyway).
We added 2 more similarly equipped mailservers. This gave us N+1 redundancy. So, if a server fails, we still have enough capacity that we don't have any interuptions... we just take the bad server out of the round robin in DNS. (BTW: _all_ servers in this must have an A record for mailhost.domain.com... no aliases.. read the RFC). The NFS server was running on a RAID, so the only real single point of failure was the NFS server itself, not disks or mailserver. To get NFS redundancy, you pay a LOT of money. But, a good NFS server should run for years with a decent load.
So the end result is a care free mail architecture that should run forever and we can scale by just dropping in a new machine when we run out of capacity. Use something like scp and CVS to keep all your conf files the same and you're off to bigger and better projects.
Slashdot Mirror
Howdy, I've worked for an ISP that was growing rapidly. We shot from 20,000 to 30,000 customers in a very short period of time. Mail broke almost over nite. There was no warning sign. One day we had lots of capacity, the next... nothing. And we had a fairly beefy server: dual proc Sun Ultra Enterprise 2/300 with 1 gig of ram and gobs of disk. When it had it, the load would shoot up over 20 for hours (and we had 20 set as our sendmail throttle limit). We tuned and screwed around with the pop server and fixing sendmail, but it usually only bought us a week or so. In the end we rearcitected. The solution we came to is highly scalable, very reliable, and fairly easy to do. It was inspired by what Earthlink did a few years back (if you're a USENIX member see her e.)
.locking or anything like that. Contention is very low for user files. Each box runs cucipop for POP and sendmail for mail, each using very stock configs. We played with IMAP for a bit, and UW-IMAP is about the best around (in the free world, anyway).
Basically, we nfs mounted our users mailfiles so they were accessable across multiple machines. The biggest problem you run into here is file locking, but NFS will handle most of it for you... no need to implement
We added 2 more similarly equipped mailservers. This gave us N+1 redundancy. So, if a server fails, we still have enough capacity that we don't have any interuptions... we just take the bad server out of the round robin in DNS. (BTW: _all_ servers in this must have an A record for mailhost.domain.com... no aliases.. read the RFC). The NFS server was running on a RAID, so the only real single point of failure was the NFS server itself, not disks or mailserver. To get NFS redundancy, you pay a LOT of money. But, a good NFS server should run for years with a decent load.
So the end result is a care free mail architecture that should run forever and we can scale by just dropping in a new machine when we run out of capacity. Use something like scp and CVS to keep all your conf files the same and you're off to bigger and better projects.
Questions? pls feel free to contact me. Later