Slashdot Mirror
Ask Slashdot: Building a Large Email Service
Rewd asks:
"I'm looking at implementing a large scale email server (cluster) to handle POP3 and IMAP4 for about 25000 people, including a lot of attachments. I'd like to go for an Open Source solution, but a lot of people around here want to go for Microsoft Exchange on NT.
Has anyone here successfully built anything like this? Can you recommend any combinations and components which are particularly
efficent, capable, secure and reliable?"
I'm familiar with one setup that used Netscape's messaging systems and they can become a total nightmare. There are lots of wierd quirks about the way it handles aliases, etc, that can be a real giant pain.
I've never tried any of the open source products on any large scale so I don't know how they perform. I wouldn't go with Exchange if you are looking to do good scalability. It would be nice to run the service on a big beefy box, and Sun boxes are far better than Intels.
Sendmail isn't the part that doesn't scale, it's POP3 when dealing with Sendmail files.
----
----
Open mind, insert foot.
And I suppose yours does?
How about stupidity?
/cgi-bin directory to IIS, Hotmail is still running FreeBSD and Apache.
n
He was talking about the user interface, not the mail server. The mail server is a hacked Qmail running on Solaris. The web servers executing the mod_perl code are FreeBSD.
And what does hotmail being owned by Microsoft have anything to do with it? Microsoft bought Hotmail after it had already been established, after all. Unless Microsoft has suddenly added a
bash-2.03$ telnet www.hotmail.com 80
Trying 216.33.151.7...
Connected to www.hotmail.com.
Escape character is '^]'.
HEAD / HTTP/1.0
HTTP/1.1 302 Found
Date: Fri, 30 Jul 1999 02:26:01 GMT
Server: Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b
Location: http://lc3.law5.hotmail.passport.com/cgi-bin/logi
Connection: close
Content-Type: text/html
-E
Send mail here if you want to reach me.
So: Microsoft Sales says that you should use NT because someone who doesn't know what they're doing can successfully configure and maintain it, thus saving on expensive Unix system administrators.
Then it crashes, and the Microsoft apologists say that it's because it takes an expert to install, configure, and maintain an NT installation, and of course it's going to crash if you have the janitor maintaining it.
Typical. Just typical.
My question: If you need a skilled system administrator in the first place, regardless of the operating system, where's the TCO benefit for the Microsoft software?
-E
Send mail here if you want to reach me.
I think it depends upon what you're doing and how you're doing it. POP is straight file slingin', little CPU involved. IMAP is, as you mention, a bit of a hog.
IMAP for 30,000 users on Linux or *BSD would require a cluster of machines for the front end, but you'd still only need one machine feeding the data into the Netapps on the back end. But the big iron Solaris solution definitely has the cojones to handle this situation without clustering, and will be tremendously easier to configure and maintain than the cluster. And also quite a bit more expensive hardware and software-wise. So it is a tradeoff, and it depends upon how much in-house talent you have. If you have a lot of inhouse talent, the Linux cluster will save quite a bit of money. If you don't, the Solaris solution will require less expensive consultant time to set up and configure, meaning it will be the more cost effective solution. Wanted a simple answer? The only simple answer I can give is "Don't use NT, at least not with Exchange" (grin).
-E
Send mail here if you want to reach me.
Performance problems at the outset:
1) syslog - for this much volume, syslog will slow your system a lot.
2) Qpopper requires a read through the entire mail file for that user each time mail is checked. For a user with a couple of megabytes of crap (think attachments) this can be a few seconds worth of activity just to get the first 5 lines of each message. Solution? Use the maildir format, which gives each message a file. Don't use MH Mail file format. Why? Because mh will do ungodly amounts of rename() calls each time the user deletes a message from the middle of their mailbox. Maildir is much more efficient.
3) Sendmail takes a lot of tuning to meet this sort of demand. Sendmail also has a large footprint. Using a mail server like qmail (my pref) or postfix (others' pref) will buy you a lot of performance for a one time learning curve of about a week's time, without having to guess at how to get high-capacity out of the system.
4) Linux is good, but unfortunately if you're going to do this on a local file system for a system with 25,000 users you need to have a lot of space. I think a Journaled (sp?) filesystem is called for here. Currently for supported tools that really means a commercial unix. I've used solaris and veritas' filesystems a lot, and I know that for a mail queue and for mail delivery veritas does amazing this. In addition, it makes recovery in the case of a system crash amazingly fast, and its snapshot facility allows you options to backup that are better then that is usually available on a mail system (i.e. minimal to no downtime to perform a backup of a stable image of the filesystem).
Anyway, hopefully I've contributed some useful thoughts to this!
-Peter
== Just my opinion(s)
I've mentioned this in a couple of other posts here, but since you've directly addressed the shortcoming of the standard unix mailbox format, I have to chime in that qmail's Maildir format addresses the weakness of having to read the whole mail spool, without incurring the overhead of using a remote database (development overhead, mostly, but also a lot of overhead in terms of program size and complexity).
-Peter
== Just my opinion(s)
I've seen some serious Exchange installations. 25000 users is far too much for one exchange machine to handle. I've seen 2000+ users on a quad processor Alhpaserver 4100 with an exchange database over 300GB and growing quickly. You can do 25000 users but you will need many servers each with their own RAID array and some serious, dedicated, professional management. POP3 might be a better option since the server requirements per user are lower.
I browse with my threshold at 2 so I can't read my own comments :-)
The Cyrus server at CMU is probably your best bet. You'll find it at at this link.
It's worth noting that this project is currently supporting all of CMU's e-mail needs. It's also my understanding that it forms the basis for Netscape's Message Server and Post.Office. This should satisfy any concerns about it's scalability. It has lots of handy features like kerberos authentication, a database style message repository, support for ACAP, etc.
Alternatively try QMail. Personally, while I think it provides better SMTP performance than Sendmail, I'd rather use the Cyrus IMAP server than the UW one (the only one supported by QMail). You could go with using a combo of sendmail|postfix + Cyrus for incomming mail (i.e. what your MX records point to) and QMail for outgoing mail. It depends on your performance needs
Exchange Server is NOTORIOUS for being both difficult and expensive when you need it to scale to a large number of users, although I understand it's improved substancially since the 4.x days when it was just impossible.
sigs are a waste of space
I've had good luck using the Cyrus IMAP server and the Exim mail transfer agent (MTA). The Cyrus server handles POP3 and IMAP, and stores the mail in an internal file per message format, and is designed for hosting mailboxes for those without accounts on the system. I've found both Exim and Cyrus to be fast, secure, scalable, and stable for thousands of customers, and I don't see any trouble scaling it further.
It's heavily multithreaded, so the performance is excellent. I couldn't say whether it's ever been used with 25,000 users, though.
Timur Tabi
Remove "nospam_" from email address
qmail is actually not Open Source.
:)
From http://pobox.com/~djb/qmail/dist.html
"If you want to distribute modified versions of
qmail (including ports, no matter how minor the
changes are) you'll have to get my approval."
Please reply in email if you feel the need, I'd
rather not start a flamewar here
--
Kevin Doherty
kdoherty+slashdot@jurai.net
Kevin Doherty
kdoherty+slashdot@jurai.net
Acourding to Microsoft, Hotmail _is_ run on top of Solaris. Check out http://www.microsoft.com/ntserver/web/news/msnw/Ho tmail.asp
They also state that along with Solaris being used, Windows NT is also used, but they fail to mention how/where it is used, so my guess would be as devel, and not production.
My favorite quote from the article is "Solaris is Hotmail's legacy production operating system". bwuahahaha.
I say with relative authority: Puh-leez!
MS would like people to believe that Exchange is an enterprise-level communications tool, when it fact it is a buchered and bloated decendant of a mediocre 1992 X.400 email system from Data Connection Limited (check out http://www.datcon.co.uk/press/messserv.h tm) Don't believe the version number; Exchange is in its second major release (4.x really is 1.x, 5.x = 2.x, etc) and still has significant stability problems.
In my experience, Exchange can support 300 users per server happily on commonly acceptable x86 corporate server hardware (say, a 2 processor PII with 512mb ram). It seems that (in my limited experience, lest MS lawyers take this to be a declaration of fact, which it is not) once you've reached this level, doubling the ram and adding more cpu's has only a minimal effect, which means that you really have to add more servers to add capacity.
Let's do the math. 25,000 users at 500 users per server (to be quite generous) means that you're going to need a Windows NT server farm of about 50 systems just to do email. Again, being generous bargain hunters, let's say you can buy one of these servers for $10kUS. That means you're out $500,000 just for hardware. In my experience, you can support 500 POP users easily on a SPARC 2 or IPX, which can be had these days for about $500 decked out (including a 17" monitor). You could support the same (probably many more) on a $500 x86 box running any of the free *nixes. Assume you blow $500 on disk storage for these boxen just to level the starting line, bringing the total cost to $1000 per. That's still only $50,000.
One less zero usually gets the accountants' attention on an expenditure like this.
But let's talk about administrative support. IMHO you're going to need 1:1 admin per NT server at that usage level, given that remote admin of NT is difficult, and 500 users per server is going to prompt more than the occasional pretty blue interface. (Nevermind the security team you're going to need for a major NT installation.) Say a cheap NT admin costs $50kUS including benefits & overhead. You're looking at an HR budget of $2,500,000us. On the other hand, say you splurge and spend $150kUS per *nix admin. If they couldn't handle 10 little boxen apiece, I'll eat the electrons this was posted with. That's an HR budget of $750,000us.
That's 1/10th the hardware expense and 1/3 the maintenance expense of using Exchange. And that's (a) making some wild assumptions that benefit the Exchange argument, and (b) assumes that you're running *nix on shit hardware. Spend 5 times as much on hardware for new, supported stuff (say $250,000us, which would buy you a couple of well-outfitted Sparc 4500s, or 10 really gorgeous systems from VA Research). Your downtime will become next to nothing, you'll still have spent only half of what you would have for NT and Exchange, and your ongoing yearly administrative cost will be 1/3 of the other option. The *nix administrative savings alone will pay for the *nix hardware in a few months.
Oh yeah. I forgot the expense of 50 copies of Windows NT, 50 copies of Exchange Server, and 25,000 client licenses... (*erk*!!)
I think not...(*poof*)
Cyrus is probably the way to go. CMU uses it, and it seems to handle their needs. I've used it at three different companies, and particularly liked the fact that it doesn't rely at all on system users, except for password authentication. Its trivial to patch the system to authenticate passwords against a DBM file or MySQL.
Its also trivial to write scripts to automate the management of the server, so you can create a new user quickly and easily.
Two years ago I installed Cyrus at a company that was using NT domain servers for their logins on all the client machines. Quick patch to Cyrus to work with PAM, and a SMB PAM module, and people were able to check their mail using their NT passwords without having any security issues of having all those users on the mail server.
I also hacked something together that automagically created the mailbox when an IMAP connection was attempted with a username/password on the NT domain that a mailbox didn't already exist for, so the NT-centric admins didn't need to ever touch the mailserver.
The number of users are much smaller, but other installations have shown that Cyrus will scale, so the ability to extend it like this is also important.
At the UKUUG Linux '99 conference there was a presentation explaining how they implemented a large scalable mail server using open source software. It was also explained how the total cost of ownership would be much higher if they used an NT solution (even using the figures supplied by MS they'd need more machines and more administrators to keep the servers runnig).
The open source solution was much more cost effective and has proved fairly stable.
Unfortunately the proceedings from the event are not yet online, however I'll try and forward you a copy (or post a link to this thread) as it may prove useful to you.
--
I'm short on time, but I wish to submit what may be the ultimate Exchange story:
A sysadmin at, ahem, a "large jeans manufacturer" was put in charge of Exchange on hundreds of NT servers. He dutifully logged and reported dozens of bugs, system outages, etc., to MS support, as the thing crashed and burned like the Hindenburg II. After a few months of this, Microsoft decided to act on the problems. The solution was simple: they sent a letter to his boss saying he was a troublemaker.
---- "If we have to go on with these damned quantum jumps, then I'm sorry that I ever got involved" - Erwin Schrodinger
I hear some people near me talking about Exchange [shudder... been through 2 migrations at companies before].
Knowing MS Exchange is a "Bad Thing", and I'd like to save the company money where possible, I decided to search the web for a collection of "horror stories and MS Exchange"... to my surprise I couldn't find ANYTHING!
Now I've seen articles here and there (InfoWorld, news.com etc.) about Exchange bugs, but I would have thought SOMEONE had collected URL's and posted them. Nothing. I'd have to do a lot of research to get this info, and given my workload it would be an unwise distraction.
The second thing I'd like to know, is how much does MS Exchange COST? I know the price varies, and larger companies get breaks if they "cozy" up to MS, but that doesn't help me much. Say a company has 50-150 employees... what does that translate into just for the software licensing?
Listen to this advice, it's obviously born on the hard back of experience, just as much as me reiterating this same line: do not use exchange.
For example:
This is only a start, but I'm sure other people have many of their own reasons as well...
I remember our migration of a mere 750 (users) with extreme horror. We had to manually create each user.
You can create mailboxes in exchange via a config file with the mailbox import tool, although I figured it out by looking at files it created and not via any documentation. With exchange 5.5 I'm pretty sure you can create mailboxes with ldap (although this is far from documented last I looked).
As to solutions, I haven't used any open source email solutions with more than ~5000 users, for which sendmail and the UW pop3d and imapd worked well for the users that I had (many were very light on email). I'd be really neat to integrate an MTA and an IMAP server with ldap to support IMAP referrals and smart mail redirection. I know some of this is done as sendmail has LDAP patches and example rules for this, but I'm not so sure about IMAP side.
Rule number 1 in Linux -- If it's worth doing, someone probably already has. Earthlink has published a really great paper on this subject. Definately worth the read http://www.earthlink.com/about/ papers/mailarch.html
We have a cluster of Exchange servers for our ~4000 workers here, and even Microsoft can't get it to work properly. There's a crew of Microsoft folks here, almost all the time, and our staff of full time admins. Just for Exchange. This is an "upgrade" from the single sendmail/imap server that handled the load of the entire company with very little problems. The main problem with Exchange is it's terrible performace, and the way it makes Outlook (or Outrage as we like to call it) hang while doing such complex operations as reading the next message in your inbox. On the bad days, it takes me as much as 5 minutes to switch messages. On the best days it's just irratic. This doesn't even bring up the problems with the feature set. For example, you can't check your mail from 'nix any more, sorry. Frankly, I can't see reason one to use Exchange. Even if you have to run on NT, you do have other options.
-- "it's not enough to be a great programmer; you have to find a great problem" - Charles Simonyi
Check out Cyrus, from Carnegie Mellon, which is gratis (but not free).
Or maybe you'd like to spend some money. Then there are lots of companies, like Mirapoint, who I work for.
I designed and currently maintain a FreeBSD system with ~40,000 users, with probably 20-30,000 "active". I only support POP3 access presently, but I'm looking into IMAP. I'm using 2 PentiumII 300s running FreeBSD. One acts as the SMTP deliver-er for all off-site activity, the other the primary POP3 server. Both have 512M of RAM. The POP3 server has 14 4.5G cheetah drives, 2 internal for the OS, etc. and the other 12 inside CMD ultra-daytona external cacheing SCSI-SCSI RAID arrays (RAID-5), each on their own SCSI channel (2 3940UWs in each machine).
I use the qpopper with a *lot* of local modifications for security and performance. A custom perl+mysql system manages the userids locally and talks to a campus-wide "meta-directory" which allows people to manage the users from their Winblows machines...User management is probably a bigger problem than performance.
IO will be your biggest concern, followed closely by getpw* calls, network bandwith, then RAM and/or CPU. There are lots of other issues such as expiring mail, preventing/detecting mailbox corruption.
Cyrus IMAPd will solve a lot of problems with IO bandwidth, quotas, expiring mail, etc...but it will require more RAM to sustain more simultaneous connections, and more disk space as users can/will/should leave more mail on the server. I have not tested Cyrus in a large scale environment...yet...
Sendmail works well, other mailers such a qmail, etc. may work as well, many claim to be more efficient, but a properly configured sendmail environment is hard to beat...but any reasonable mailer should be adequate, the actual MTA load shouldn't be that great, no delivering to the mailbox, that's another story.
Feel free to contact me directly if you desire any more details or statistics.
John Soward...University of Kentucky
That's amazing... we've got 18,000 users using qpopper on a P2-233 ... and no sweat.
Amazing how little hardware we had to use.
---
I run a large (>>25000) free email service. We started out with qpopper, but quickly had to switch because qpopper does (or at least did at the time) very poorly with large mailboxes. If the mailbox was very large, qpopper would crash leaving the mailbox twice as big as it was before.
We were using sendmail at the time, so we started using qmail as the local delivery agent. And pop agent of course. Eventually we switched entirely to qmail.
One thing to watch out for regardless which solution you use is that (last time I looked) linux (or is it ext2?) is limited to 16-bit uids. There's ways to get around that; I just wish we'd considered it when we started.
When you have that many users you have to have a nice structure for the usernames, which isn't the /etc/passwd file. And, you need a mailbox format that isn't linear, like the normal mbox. The rest of the problems can usually be solved with hardware (think about using a raid).
I know of three potential semi-free solutions.
Carnegie Mellon Cyrus (go to the FTP site and download the latest version. Don't rely on the way out of date web page to link to it.) IMAP server.
University of Washington's imapd. This seems to be under more active development, and supports a nice range of features, mailbox formats, and security mechanisms. However, it uses the passwd file (although you might be able to get around this using PAM) and it doesn't natively support quotas. (although you can do this at the OS level.
Darthmouth's Blitzmail Server: This has been ported to linux, and is *wonderfully* scalable across multiple machines. It inlcudes its own directory services too. The only problem is that it doesn't support Imap (although some work has started on that front), and the only database it supports as a backend is oracle. I would love it if someone hacked it to use mysql of postgresql with IMAP support, but that's a tall order. The client is also under-featured.
All of these have their drawbacks though. You might wish to go with a commercial IMAP/POP server on linux. There are a few good ones that exist. You definitely don't want to go with exchange. A lot of people go that route because they are forced to. My experience with exchange 5.5 was so bad that I would not recommend it to anyone.
-OT
I believe Hotmail use Qmail, which is open-source. When MS bought them out they tried to switch to NT/Exchange, but couldn't get it to work, so they're back on Solaris/Qmail now. I believe they support a large userbase which not only have attachements, but a whole lotta spam, to...
In the land of the blind, the one-eyed man is kinky.
Horms is the mail dude for ZipWorld, one of the
e rs/horms/
larger Asutralian ISPs. They run Linux internally
and he presented a paper at CALU on building a
large and scalable mail system.
See:
http://www.linux.org.au/projects/calu/cdrom/pap
for the conference paper.
Cameron Simpson, DoD#743 cs@cskk.id.au http://www.cskk.ezoshosting.com/cs/
Sendmail's the answer for us. The only thing that hasn't scaled well is plaintext aliases files: we've got some 20K mail lists, and it's beginning to get somewhat messy, so we're having to go to the non-plaintext solution. But for all the rest, it's stock sendmail with various GUI backends for end-user ease-of-use (and security). Note that we don't have 25K users, but 17K isn't that far off, and we do a *lot* of e-mail.
"[Vendor] experts in Redmond..."
Hmmmm....who could this be?
---
Put Hemos through English 101!
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
...except an anti-recommendation.
If you are going to setup 25,000 users, do not, repeat NOT, use Exchange. I remember our migration of a mere 750 with extreme horror. We had to manually create each user.
Of course I was simply a lowly programmer working under the direction of our totally incompetent network admin--maybe there was an easier way and she missed that topic in the training the week before.
What you really need is a requirements analysis. Exchange is a totally different thing than, say, Sendmail. Analyzing what you need will tell you which to go with. For instance, do you need public folders, scheduling, etc? If so, maybe use Exchange. Do you need configurability, speed and Internet email? Then you want not-Exchange.
---
Put Hemos through English 101!
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Upwards of 30K shell users, four Sparc 2s, SunOS 4.x, sendmail. And a lot of email. Our only big issue was the number of files in /var/spool/mail, which we coped with by going to Network Appliance NACs.
We noted that directory lookups got worse in a distinct knee -- i.e., we had no problems for a long time and then we hit a magic number and things went all to hell. I do not know offhand how well linux or Solaris deals with directory lookups, but you could test easily enough.
The thing you didn't tell us was what the volume would be like; the number of users matters for the mail spool but the number of email messages matters for the CPU usage... I suspect that you won't need a very heavy box, though. Email is cheaper than you might think.
Oh. Run a DNS server on the mail hub, to avoid a lot of lengthy DNS queries on some other poor machine. Flush the cache daily.
Compaq / Digital professional services deploy and manage virtually all of the largest Exchange deployments- once you get above a certain deployment size, the requirements for maintaining the system require extremely specialized and arcane knowledge that (apparently) only they have. MS actually has structured the MS Exchange support contracts so that above a certain deployment size, you pretty much have to use Digital, or you won't get any support from Microsoft.
So, unless you're willing to fork over the $$$$ for consultants from Digital to come and build the whole thing for you, I'd avoid Exchange like the plague for a project this size.
- The mbox format it uses is very nice for moving things around, but don't forget, if one acct gets bogged down, it gets blocked due to file locking problems. (if you use the default local mailer). Multiple mail files are nicer. May run out of inodes quicker if you aren't careful.
- Its ruleset bogs things down just a bit. Significant if you want to get more than 10k emails out in an hour. (I haven't really benchmarked it yet.) Power versus speed, configurability versus mail handled per sec.
- Sendmail is a big program. Qmail is smaller and more modular in the sense. The operating system's process scheduler can handle things better than sendmail's internals. I've had server loads of 10 with higher traffic on a dual 300mhz machine.
Why not to use qpopper- To download your mail, a second copy of your mail spool file gets made so that qpopper can sort out which mail you wish to keep. Effectively you get only 50% of the diskspace.
- All users must exist on the system. Bad system administrators (like at my old job) would make home dirs for all users, leaving ftp open at times. Virtual local users kick butt.
There is your reliable software. For network stuff, I would highly recomend FreeBSD, but Linux would do fine. I would recomend qmail as it is both a popper, mta and local mailer. It doesn't give the problems above.-
ping -f 255.255.255.255 # if only
The talk was "The Design and Implementation of a Large Scalable Mail Server". It's about the mail cluster I set up here for Oxford University and it sounds similar to what the original request was for. We have about 30000 users. I successfully argued against MS Exchange and designed and built a completely Open Source solution based on a Linux cluster with 250GB of disk, UW imapd/ipopd, Exim (MTA) and an Apache/mod_perl-based web to mail gateway that I wrote called WING. The two Solaris nodes in the cluster were there for political reasons and are being replaced by Linux boxes within the next few weeks. There is a WING web page and mailing list which includes a link to the PostScript slides of the talk. The slides will also be shortly be available from UKUUG.
I'm currently working on the feasability of a mail cluster for 100 000 accounts. It's not done yet, but our various experiments gave us various answers.
:-)
We use exim, Qpopper with mysql patche, mon, fake and rsync. Each base box hold 88 GB of data and are fully duplicated (double delivery with exim, and further syncronization with rsync). The switch between a main base box and his double are handled by mon and fake. A hot spare then reconstruct a new double, delivery and popper deletions are blocked during the reconstruction.
Two problems aren't solved yet:
- raid 1 between boxes
- imap
I hope that imap will work when nfs locking will be reliable. For raid 1 over boxes I have a very tiny hope that nbd could be a solution.
Anyway, we made some tests, and it somewhat works already. We are now tuning various parts and writing procedure to handle the beast and react to failures (our current estimate is one major but handable failure every month).
If you have ideas of working solutions for my 2 problems don't hesitate to share
Nicolas
Handling a 25,000 active user base on one machine might be a problem for intel machines, and (dare I say it), most Open Source OS's.
Solaris will do this, but you will probably need to run it on a _big_ box, like a Sun Ex500 class machine with about 8 or more processors. And get their SIMS product, too, it's pretty well optimised for the high end. Other high end commercial unixes like AIX and IRIX will no doubt scale this far as well.
If you are able to go distributed (ie, the organisation is easily divisible geographically), then something like Linux or FreeBSD with qmail or smail will probably cut it.
Beware that exchange servers offer a fairly high level of integration with Outlook, which a product based on open standards will not be able to deliver.
To be perfectly blunt, to implement Exchange like that would be *INSANE*.
6 8525655b0050f2f2/35BEC3BF6D717A3F852567120 07A435A?OpenDocument
First of all, suggesting to implement an NT solution for an organization of that size is already tempting your job security, but to actually do it?
Assuming standard users and needs for this system, I can only recommend using a Lotus Notes/Domino system. If you've got the cash, there is simply no better solution out there, or even close.
Run Domino (the server end of Notes) on several UNIX servers. Solaris (SPARC and x86), AIX, and HP/UX are all supported, with a Linux port (Caldera 2.3 (currently in beta) and Red Hat 6.0 will be supported, as well as SuSE 6.1 and Pacific HiTech) on it's way Q4 99 per DevCon.
Notes has got all you'll ever need, and R5 simply blows away anything M$ has out there. You've got to pay for seats with Notes, but to tell you the truth, Exchange is free, and you get less than you pay for.
Plus, your users can run the Notes client on any Win32 they think is prettiest (please tell me you'll use NT and not 9x on the client end).
Look at this for a guy in your situation who had to deal with Exchange.
Some other really good links are here, here, and here:
http://www.notes.net/50beta.nsf/7d6a87824e2f097
(problem with the last one, copy it and cut out the space that is stuck between the zero's, the href tag keeps putting it in! It is a great article though : )
(TIP: Show the guys with the money those links so they know why you should use a Domino/Notes solution.)
Check out www.apexmail.com . they have over 50 000 active clients, running on a totally linux based system. (They use totally custom software). Their setup includes a dual PII 450 file server with a 150 GB RAID (on a Mylex RAID controller). The entire public site runs on 4 or 5 single PII 450s, all on a switched 100BaseT network.
...si hoc legere nimium eruditionis habes...
FWIW, FreeBSD takes of the web frontend and Solaris handles the database which stores all the mail. To my knowledge it was never anything _but_ FreeBSD & Solaris. M$ tried, and quickly backed out NT because it just couldn't scale to something this size.
Having to still support an old network installation run by a bunch of idiots (they are attempting to implement a mostly micro~1.oft shop), I can tell you that outlook doesn't scale to more than 2000 users before the maintenance starts to become a headache.
This is a large client trying to implement a server farm of 20+ NT machines, each server supporting 600-800 users, and combining the whole lot into a coherent whole. Fortunately I only have to fix their poor network designs. The team of administrators now numbers more than 50, most are MCSEs, none less than 5 years experience with Micro~1.oft products. They are tearing their hair out on a daily basis. Complaints number in the hundreds every day, and thats just the users who haven't given up completely.
My advice is to start looking at the larger commercial products, possibly Netscape's server. Get a reputable vendor to support it.
If you look at open source systems, start with OpenBSD and NetBSD.
Divide your system up between the MTA doing delivery/reception of the messages, and the MTA serving the users. Its ok if email to the outside world goes down for short periods of time, its almost expected. But if users cant get to their mailbox 100% of the time, you will look bad.
You also need to look at managing more than 32000 or 65000 users in the future, remember that various *nixes have either 15 or 16 bit UID fields. You should make sure user accounts/authentication/logins are separate from any UID system on any machine type. This means getting some kind of medium sized DB, and tying it into your auth and login schemes. Others have done it, its not that hard (look at AOL with 10million+ user accounts)
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
No matter what solution you pick (or gets chosen for you) would you please report back to us with how it went? It would make a great case study either whether or not MS can handle a true enterprise level application or how well FreeBSD/Linux/whatever handles it.
Thanks...
putting the 'B' in LGBTQ+
Currently the ISP I work has 50,000+ mail accounts on a multiserver solution... NetApp 720 for the file store on a private segment, A DellPowerEdge 2300 for the incoming mail server, and a 450 Pentium II with 256M ram for the POP box, MySQL POP Authentication tables, and a standalone MX20/Relay on another Micron Pentium II (I believe, haven't had to mess with it much)...
;-)
;-)
If you need IMAP, it gets tough... Except for IMAP I'd recommend Qmail.. it's the most robust thing, besides the NetApp that we have.. With NetApp and a RAID0 backed queue drive it screams...
You could all of the above with Penguins or VARs for pretty cheap... at guess 10,000 excluding the NetApp... If you use a beefy linux box with a fast raid 5 for the NFS server back end you'll also allow your servers to "load gracefully"
If you need IMAP but on a single domain, use the UW IMAP server... It even comes as an RPM and looks great... If you need multiple virtual domains like we do... IMAP looks pretty grim...
As it is when we go to web based email it looks like we're going to have to do a WebBased POP client...
Careful though, IMAP can get *WAY* more abused though... With IMAP you have the tendency for people to park and use more space... With POP it's just grab and go..
Resource wise POP is a better bargain and most clients can deal with it just fine... IMAP isn't worth the server load IMHO... Use qmail anyway ya can...
"Don't fear death... fear not living..." -me
The paper is available here (in postscript).
At the talk I had the impression that the softwware was free. I cannot find it on their (skimpy) web site though.
From their description, 25,000 users wouldn't begin to make it sweat.
i work for an isp in the uk and we run our mail on mcis (microsoft's commercial internet server) in conjunction with LDAP and SQL. it works fine with around 300,000 users (adminttedly with clustering, but allunder nt using compaq h/w).
i would recommend against using ms exchange as i know many people who have had loads of trouble with it.