Slashdot Mirror
Ask Slashdot: Building a Large Email Service
Rewd asks:
"I'm looking at implementing a large scale email server (cluster) to handle POP3 and IMAP4 for about 25000 people, including a lot of attachments. I'd like to go for an Open Source solution, but a lot of people around here want to go for Microsoft Exchange on NT.
Has anyone here successfully built anything like this? Can you recommend any combinations and components which are particularly
efficent, capable, secure and reliable?"
Use Netscape Messaging Server. It's not
open source, but it's the best IMAP server
on the planet. Plus, it runs on Linux.
You are obviously not totally familiar with
Netscape's messaging systems. It uses an
LDAP directory for all the addresses and
aliases, which is the absolute ideal scenario
in this day and age.
In supporting 25000 users you want to be sure
you can address whatever problems may present
themselves.
The NT solution involves pulling out your
credit card and maybe MS can fix it for you.
Open source means a fix might already be
findable on dejanews.
Forget it, not even worth buying that many licenses on exchange. It'll end up crashing just like every other piece of microsoft software.
FreeBSD can handle that with no problem, regardless of the size of messages.
Just make sure you use SCSI! =)
I'm not talking about how it stores the aliases, but for example, if you want to change your aliases in sendmail you just run new aliases, it processes everything and it goes pretty fast. In netscape you run the equivalent and it will process each entry one at a time. It can make a process that might take a few minutes take hours and hours.
General impressions of Netscape's stuff is that it tries to make itself easy to administer, etc, but sacrifices some functionality in the process.
Yeah... simplicity says you want the big box approach... Personal preference lies with getting a Killer Alpha (the new Compaq XP1000 with a 600 MHz 21264 Alpha). 90-100 Gigs of RAID 5. A far inferior Alpha to that runs e-mail for 20000 on this college campus, and everyone here runs pine on the server (more overhead).
Not Open Source, but the Tru64 License is going to be a lot cheaper than the Microsoft fiasco.
I was the Exchange admin of a company with 10,000 Exchange users on NT boxes. No real problem. Handled the load just fine. 10,000 users on 4 Pentium 266's. No sweat.
Actually, I'd suggest going with a Sun. Alphas are very good at heavy duty processing but they don't have as much strength in the realm of I/O. Sun's are weaker processors, but they have great I/O. Since it is mass processing of mail that we're talking about here, the I/O is the important part.
First, sendmail handles only SMTP, not POP or
IMAP. Plus, unless you completely customize it,
sendmail will store mail in one giant file per user,
making random acess via IMAP painfully slow.
Second, there are many things Netscape Messaging
Server can do with aliases that sendmail cannot.
For example, dynamic LDAP aliases. So you could,
for example, have an alias Hawaiian users that
automatically matches anyone with an 808 area
code...
Plus, Netscape Messaging Server 4.1 comes with
a wicked cool HTML mail interface a la Hotmail,
but much better.
I would check out Cyrus IMAP. It has been fairly well tested in large environments (Colleges and such). It also has an active mail list that is responsive to questions such as this.
http://andrew2.andrew.cmu.edu/cyrus
current beta is 1.6.1, not sure about the current stable release.
It has many options and allows you to have mail accounts that are not mapped to a real account on the box.
Sorry this is under Anonymous Coward, but when I went to sign up for an account netscape hung...
We had a similar discussion in my workplace (specifically Exchange vs. QMail). We're a small company, so volume wasn't an issue. What finally tipped the scales in the favor of Exchange was that it isn't just a POP3 server - when used with Outlook and Exchange on the desktop, it lets you do nifty things like share address books, schedule meetings that are automatically entered into people's calendars, and other things that management types like.
If the people arguing for Exchange want these features, you're going to have exchange, 'cause there just isn't an open-source solution (at least, not one that I'm aware of) that integrates email with scheduling and addressbook functions into one tidy package.
p266?
4?
No one used the mail system did they?
No attachments?
huh...
Often, they run a MS Ex or Notes next to our linux isdn server, which transfers incoming mail to their neighbour.
We get many calls from those who shipped the NT,
cause
actually, a sad thing.
If Exchange is so great for serving a large number of email clients, why did Microsoft themselves refuse to use it for MSN?
Brief historical note: the original plan was to use Exchange for MSN, but after a quick evaluation, the MSN engineers decided that Exchange would require a server for each 100 users or so, so they wrote their own email server within a couple months (probably based on open source) and use that instead.
Conclusion: Exchange is such a piece of crap that even Microsoft refuses to use it, and you want us to base our system on it?
There is also tons of ancedotal evidence of people having trouble with exchange, and basically it requires a full-time admin just to keep it running.
The maildir format should solve the problems of large mailspools. qmail supports it natively, there are procmail patches for it (perhaps by now it ships with support for it) if you're willing to use procmail as your local delivery agent. There are pop servers that support it natively, and patches for others.
25,000 isn't a huge number of users. I would expect a high-end Intel box running Linux (or *BSD) to handle it with no problems. The main concern, based on my experience on somewhat smaller mail servers, will be disk speed. Lots of RAM will help somewhat, but a RAID would probably also be a good idea.
If they were willing to pay for the software and hardware required for a MS solution, however, you may want to spend it all (depending on the politics of the particular company). Personally, I'd investigate a Sun Enterprise 250, 450, or 3000, and any external hardware RAID. Nice hardware.
I did something very similar using Sendmail on a FreeBSD box. I used the Cubic Circle Pop3 daemon during the beging phases of the operation then changed to Popper due to a pushy client. I had no problems what so ever.
Speaking also from an administrative point of view, avoid GUIs at all costs. At my previous postion we used PostOffice 3.5 for NT, and the GUI was unusable whenever we got 300 users in a catagory. We also had HORRIBLE problems with file creation. It would split one email into 3 or 4 files and before we knew it our server would be hanging since we had 20000 files in a directory.
I used to work at a company where we used Exchange for about 5000-8000 people. Everyone had TONS of attachments. The typical user's mailbox was 20+ meg.
The system worked. But the sys-admins had to reboot the servers every week or so. They would lock up otherwise. The worse part about rebooting was all that disk space made the reboot take HOURS... The servers would occasionally lock up for no reason even with the rebooting.
They also needed about 25 dual processor Compaq servers (and they were adding more every month or so) to handle all of this.
But we could use the MS/Outlook/Exchange scheduling and stuff, which is nice. If you don't need that, I'd go UNIX if at all possible.
shure? Another reason why Slashdot needs to
spell-check all of the incoming comments.
Anyways, Hotmail runs a patched version of qmail.
Another one that has come up recently in
discussions on debian-devel is postfix
(http://www.postfix.org). Check it out.
A large ISP I know of :-) once used FreeBSD to support ~ 25K users' email accounts, that was pushing it's capacity, on pentium pro 180MHz box, but it did handle the load quite admirably but slightly more than 25k started to sink the ship so to speak. However the rate of new accounts being added pretty much made having only 32-bit UIDs a very soon forseeable serious problem. If 25K users is you absolute max, then FreeBSD will do great for least money spent. Just be sure to use wide ultra scsi disk and most RAM and fastest CPU you can afford. Single CPU is ample, SMP is wasted money for such machine. If you forsee many 10's of K more users in future growth, you need to get a unix that support 64-bit UIDs in filesystem, like Compaq True64 (DEC Unix .... an Alpha box will do the job quite very nicely.), Solaris 7 on UltraSPARC box, or similar competing product.
--Lorky
Netscape Messaging Server is wonderful...unless you're using version 4.0x (Solaris)
/tmp taking down your ability to use telnetd to fix it...a nightmare...but 3.6 doesn't seem to exhibit these problems...
There are known memory leaks in 4.03...smtpd will drop unexpectedly, admin will fill up
However, supposedly Netscape is working on these problems diligently, and they should be solved by the end of this month! So either wait a little while, or put 3.6 in there now...you won't be dissapointed
If Netscape people are reading this (We all know you are), let's get those fixes out ASAP!!!!!!!!!
i agree, one of the companies I work at has a SBS with 23 users, its a nightmare. Not to mention trying to back it up.... I have had nothing but problems with the ArcservIT Exchange Agent.... what does tech support at CAI say "oh yeah we broke that, ya gotta do this....
ARRRRGHHHH
But I guess thats incuded in the "Lower Operating Cost".
Exchange has many features, but it is not the most reliable system around. I would rather use another system such as qmail. It may not have all of the bells and whistle that exchange has but it's a lot more stable than exchange. I don't know who in your organization is pushing for exchange, but they're probably non technical people. Good luck trying to get 25,000 user on exchange
Microsoft want to release a solution called MCIS that works using IMAP and allows web based email access. Anyone know anything about it? Apparently they want to swap Hotmail over from Unix to NT (yet again!) using this new software.
Another IMAP/Web based email front-end is IMP from horde.org - this looks the business. Open Source and PHP-driven - yummy!
(dare I say it), most Open Source OS's.
Solaris will do this, but you will probably need to run it on a _big_ box, like a Sun Ex500 class
machine with about 8 or more processors. And get their SIMS product, too, it's pretty well
optimised for the high end. Other high end commercial unixes like AIX and IRIX will no doubt
scale this far as well.
If I were going to set up such a mail server, I would try to get something along the lines of a Sun Enterprise 3500, 4 CPUs, and 2-4GB RAM. I like the hardware, and of the commercial OSes, Solaris is my favorite. I would feel confident with that machine that I would never have a performance problem (because it is vastly overkill). But disk speed is by far the most important. I would buy the fastest RAID array I could.
On a budget, I think that a single high-end Intel system with a fast RAID (even mirrored Seagate Cheetahs) would handle this many users with no problem. But I can't say that for sure. I've seen a P133, 64MB RAM, slow IDE drives, handle mail for 2500 users, with negligible load.
While I do not advocate any Microsoft product, I do work at a company that has around 5000 users on about 3000 computers running NT. We have Exchange running on an Intel box with two P3-500's, a gig of RAM, and 18 gigs of RAID controlled hard drives. However, we recently upgraded it. About 2 months ago, it was two Pentium Pro 200's on 384 megs of RAM, with 12 gigs of SCSI. I have never had a problem with Exchange in any way.
Not IMAP you haven't. POP, sure. You only handle
the mail twice - once to get it and once to send it off to the POP client. IMAP, you gotta keep it around and maintain persistent connections to each client all day long that eat up 1MB+ of RAM each (more depending on server software).
Hotmail uses a combination of FreeBSD/Solaris. they have sparcs on the back-end and FreeBSd on the front-end.
Linux alone is bad enough, but a Linux hack like TurboCluster is just asking for trouble.
The problem with database driven mail, like Exchange, is betting your whole org on the integrity of that one file. And please don't argue "restore from backup." Exchange is a beast to even restore backups to. Fsck groupware!!! Fsck Exchange and Fsck Notes!!!! Sendmail, baby!!!!!
Oh, wait, I forgot, sendmail's mbox file format never gets corrupted. Hahahahahahahaha.
(OTOH, maildir/mh use the file system as a database, so you get the benefit of a database with no additional risk of corruption beyond what you would have anyway with any other solution)
GroupWise can scale to that number of users. NDS would make administration less painful.
For those with doubts, answer this: Who did the email system for Comdex the last few years. Novell!
Also note that they have an email system targeted for ISPs in beta. Looks pretty good.
Okay so you have a number of options for an OS:
Windows NT:
The largest concerns with NT are stability and security. The level of intergration excahnge has with office is both a blessing and a curse. Look at the recent melissa virus. NT is fast, and it is easier to setup.
FreeBSD:
Most of the internet uses FreeBSD for their servers, and i've worked with a few ISP's that use it. It's stable, fast and relatively secure out of the box. FreeBSD was built to be a server. The biggest cons are that it is not as easy to use as NT.
Solaris(Sparc);
Solaris on Sparc (IMO the only way to use Solaris) is another wining combo. Solaris is fast and stable on a Sparc. but you lock yourself into a proprietary hardware scheme. This is the other big unix player on the internet.
Linux:
Linux IMO is not a server OS, there are some concerns about security, speed and stability. Also the speed at which changes are made can be a problem in a production environment.
MTA:
The mail agent is a big issue, Sendmail or Qmail will handle the SMTP aspect fine. Read the docs on both and decide which best fits your needs.
POP3 :
I'd use either Qpopper with APOP for security reasons. Newest version of Qpopper is rock solid.
IMAP:
I honeslty am no IMAP expert, there are some nasty problems with older versions of IMAp so make sure you read the docs and get the newest version.
Hardware wise: You should go with a multiprocessor machine and two seperate drives, one for the OS the other for data, this way you data is isolated from the rest of the system.
Qmail is excellent when it comes to both performance and security. There is some concerns about IMAP support, though (but that's not an issue for us).
We handle scalability by adding front machines that know where to route mail to specific users in a domain, so that we can just add another server, copy in the standard setup, reconfigure the front servers, sync, and we've added capacity. It's been a breeze to set up.
We also have web interfaces, and do extensive session based caching to reduce load on the backend servers.
All Linux based.
You need to define more carefully what 25,000 users mean, and then do some "email server math". Basicly, you've stated you want 25,000 mailboxes and POP3 and IMAP. You need to answer questions like:
:-)
1. How many messages per mailbox per day, so you may derive the message injection rate (msg/sec.).
2. What percentages are POP3 and IMAP. How many connections are concurrent?
3. How often do the POP clients check their mail, and what is their authentication rate? (25K users checking for new mail every 10 minutes is ~41.7 authentications per second. This is not trivial!)
3. What's the average number of messages per IMAP mailbox, and based on #2, how much RAM do you need. Most Open Source IMAP servers have a few hundered Kbyte VM footprint per connection, with an additional few dozen bytes per message in the mailbox. Using this formula, you're looking at quite a chunk of RAM, regardless of the number of servers you settle on.
You should probably investigate Sun's SIMS, or Netscape's mail server. Both are engineered to handle email problems of this size. Sun's product can be configured in a HA cluster to minimize downtime. But I'm biased, as I work on this project.
Also... Email problems on this scale are often more of an I/O problem than a CPU problem. Be prepared to invest in some serious disk I/O. Think disk array's with write cache, and be prepared to waste disk space to build up I/O speed thru striping. Use VxFS for your filesystem.
Good Luck!
and A.C. email engineer
From the BLURB file included with qmail-1.03 (http://www.qmail.org/):
"On a Pentium under BSD/OS, qmail can easily sustain 200000 local messages per day---that's separate messages injected and delivered to mailboxes in a real test!"
While my site is no where near that large, we easily deliver 600 messages a day on a Pentium 200 running Linux. The box is also serves logins, web, and dns, and it doesn't even break a sweat.
Just because something isn't GNU doesn't mean it isn't open source. You should try crawling out of your hole sometime.
And if you think qmail is harder to configure than sendmail, you obviously haven't played with either. I had qmail up and running, virtualhosting and everything in a matter of minutes.
They make exactly the stuff you are looking for. You can still use Sendmail/Qmail or whatever you are usingm; but now running it on clusters of cheap PC's for fault tolerance and scalability.
Hardware/Software upgrade is done without stopping the system.
Just read all of the postings above this one and I was suprised not to hear about Lotus Domino. One of the best enterprise mail servers out there IMHO. Now obviously these 25000 users aren't in the same building. At the most what, 250 per building? 100 sites? With bandwith what it is and some companies with little 56k links, a solution with more than 10 boxes is nowhere near overkill.
Also to slacken a little of the Exchange flames. When melissa hit big time I heard of all these machines that were going down so I loaded exchange on my sandbox. Wrote a little e-mail client to flood it (in linux). 10,000 mails in 2 minutes for over an hour. And it never bit the big one.
*
-Open mindness is an incredible force. Its what created Linux and will also kill it.
Why would we believe anyone that can not even spell?
If you're looking for an open source email system, that can handle a huge amount of traffic and lots of attachments, check out Blitzmail which can be downloaded from ftp.dartmouth.edu. Blitzmail was designed at Dartmouth College and uses the Blitzmail protocol, but it can also speak POP. I believe there is also a version that will speak IMAP.
m l
At Dartmouth's site, there are over 15,000 Blitzmail users who on average send over 250,000 messages a day. The average mailbox size at Dartmouth is 2 to 4 megs (people transfer all their files using it). The system is also incredibly reliable.
There are also some cool email clients that work with it besides the standard POP and IMAP clients. There's a Macintosh, PC, and Java Blitz client, and a web based client called WebBlitz. And the servers can run on Linux.
http://www.dartmouth.edu/pages/softdev/blitz.ht
https://basement.dartmouth.edu/blitz
Sun's SIMS (Sun Internet Mail Server, http://www.sun.com/sims/), now a product sold by iPlanet (the Sun-Netscape Alliance) is perhaps the most scalable POP/IMAP mail server around. It's available on Solaris/SPARC and Solaris/Intel. It includes a web-access feature, and includes an integrated LDAP directory for user info. Many big ISPs use it and love it. Sun is also having success selling it to emerging mail services like cellular phone service providers that also offer internet messaging. It also supports high-availability clustering on the SPARC platform.
As for storage, many ISPs use Sun's A1000 arrays for e-mail. They have 18 GB, 10,000 RPM drives and cached RAID controllers on board for high-density, high-performance storage.
Best of all, given SIMS' ISP success, iPlanet should have many reference accounts with the user base size you are looking to support.
While I am not plugging exchange either. Charles Schwab runs Exchange, and has about 25K users. I think a fairly well equiped FreeBSD box could do the same job using IMAP, or POP. You can and or could use LDAP to manage the users. Netscape Mail server could handle this load, I use to run it on Sun Solaris. I guess you could probably do it on Linux too, in fact I am almost positive. I think you would get better performance out of FreeBSD with all the file I/O going on.
For what it's worth, FreeServe who are the largest ISP in the UK with 1.5 million users do their mail on four linux boxes.
They have 2 Linux boxes running qMail to recieve and send the mail, and they have two Linux boxes running Cyrus IMAPd as customer facing servers.
These boxes are PII-300s or thereabouts.
This is backended by a few big NetApp filers purely for the disk capacity (1,500,000 users remember) but this job could be done by a Linux NFS server for 25,000 users.
The user database is kept on an LDAP server.
FWIW Cable and Wireless Internet in the UK used to run 50,000 users on a single Sun Ultra 2 with 256MB RAM, 130GB of fiber channel storage and Sendmail 8.x.x. The POP3 server was Qpopper.
They at some point got it into their stupid heads to use NT as their mail server. So they threw out the Ultra 2 which was idling and replaced it with 8 dual processor PPro 200s running NT and MCIS (Microsoft Commercial Internet Services) and instead of failing once a month it started failing once every four hours.
The 30,000 machine cost them 1.5million to replace with an NT system which was substantially worse service wise than the original.
NT is not an option for an organisation this large. Not if you want to maintain your sanity.
Never had a problem with Exchange??? You are probably the only one. I think it is typical to have Exchange go down at least once every couple months.
Keep in mind that if you use the POP protocol, like most ISPs, you have an significantly easier life because you don't have to store mail on the server. You can actually get away with linear mailbox formats. However, if you're using an IMAP-like protocol, which is superior in many other respects, your backend scalability is very important.
I've servers attach this problem by using one or more of the following strategies: indexed or sorted mailbox files, saving each message in a seperate file, storing each message in a database.
The important thing to keep in mind though, is that for IMAP you won't be able to get away with just sendmail, the password file, and flat mailbox files like many ISP's do.
-OT (on a different computer)
I love the site. But Rob, this whole site is turning into an anti-microsoft flame war. Microsoft makes some good products, and exchange works excellent.
As anyone can point out an exchange box going down, I can point out one of my FreeBSD or Linux mail servers going down too.
This should not be a pissing contest portal.
could you have pulled these figures out of your ass any more?
800 million divide by 25,000 users = 32,000 dollars per person for an email account using exchange.
you = dumb+fuck = dumbfuck
800 million / 25,000 users = $32,000 dollars per user
Could you have pulled these figures out of your a$$ anymore??
You = dumb + f-u-c-k = dumbF-U-C-K
I've been to their colocation facility and seen the servers. They are running zillions of Intel architecture boxes, packed so tightly they removed the skins to fit more per rack. An astounding sight, really. Hundreds of bare boards mounted in racks.
They probably have Solaris boxes too, but from what I saw, the ratio of Intel to Solaris would be on the order or 50+ to 1.
I thought it telling that the Hotmail site appeared to have at least two people manning it fulltime. Across the aisle, the much more compact EBay.com was purring along unattended on a couple of Sun 10000 machines.
I would suggest taking a look at Communigate Pro
http://www.stalker.com
It will be worth your while in features and scalability...not to mention the admin features. I now work in a LARGE exchange environment and it takes everything we have to keep the exchange servers up...with Communigate and Linux this was never a problem at my last job.
The UW did just this, for over 40,000 students and twice that many faculty and staff. They used a pretty cool solution.. a cluster of cpu workstations, a cluster of filesystem workstations, and a cluster of mail workstations. Checking email via pine was easy, because it was just logging in and using IMAP to bring the stuff down. POP, on the other hand, connected directly to the mail cluster. It's worth a look at what they did, especially considering that they did it over 5 years ago.
Ahh, there's nothing like an Exchange-bashing fest to bring out the Notes people.
Actually, Windows NT is the least scalable platform that Domino runs on. For that kind of user base, you should be looking at Solaris, AIX, AS/400s, or (even) S/390 mainframes to run Domino.
Agreed that Domino would be a waste of energy if you want vanilla mail only. Also, be prepared to get IBM or another consultancy to do the deployment.
Hmmm, if that's the same "large jeans manufacturer" I heard about, the place was crawling with Microsoft Consulting people a few years back (on MS's dime!)
Does this company start with L end with an S.. and have a evi in middle.. I couldnt stop laughing.. After working in a few technical shops I was wondering where MS was selling all the server software.. My small mail server runs sendmail/procmail works great.. just dont use the solaris pop/imap kills mbox.. and sure its faster.. but I feel safer when I can access the box via more than one facitilty. How much cheaper could my next pair of 501's be if they went to a freebsd / sendmail solution.. ahh open source
Dont think it really matters which box you choose.. both have strengths.. I prefer the alpha.. but I admin a large group of solaris machines that are awsome. Linux on Alpha is just great.. I have been using it for a while.. We had 3000 students on a multia (alpha 166) for a year.. had a few stabilty problems.. but I think it was cause I watched it catch fire as it procssed all the mail.. and had 30 people reading in pine.
Amazingly scalable, free, open source.
10% utilization at a time??
at 4mb a pop?
I think you're off by an order of magnitude or so.
I'm running a 150 person mailserver, with people mailing autocad drawings about. On a mac. On a 68040 (this week at least, it's on the backup machine).
I rarely see more than one pop connection at a time, and a couple incoming or outgoing smtp transactions.
So call it 1% utilization on pop, and 3-5% on smtp.
All of this is running in ~100 mb of drive for logs and userspace ++ 4 megs of memory for the server process.
man, you're an idiot.
hotmail uses a front end to a mail server.
this is the funniest post in a while. a 25k user mail server..written in PHP or perl?
and umm..i doubt that hotmail is open sourced, they're owned by microsoft.
So rotate your logs more often and don't let them get so big. Duh. Logrotate can be configured to check by period or size or both. Do it as often as nessisary not just some arbitrary schedule.
AdamT (at work)
With 22,000+ users, a big dual 533 alpha w/ a raid 5 driveset isn't to shabby. Sendmail and qpopper (compiled for server mode, of course) run ok.
Openmail (HP now seems to have a beta program
for the linux version) is reputedly the most
popular large-scale unix messaging store in the
Fortune 1000
It has MAPI interoperability with Outlook, as
well as IMAP4, POP, and Web-interfaces.
Download it from http://www.hp.com/go/openmail
and try it out.
On the qmail homepage there is a link to a website which describes how to setup many, many mail accounts using just one uid.
I have not tried it as I run qmail for just 4 users, but maybe you should check it out.
We have 30k+ email boxes running on sendmail on
DEC Alpha Linux and RAID 5. the only problem we had
was users sending BIG attachment. one time, someone send
a 1GB attachment. Now, everyobdy's quota is 20MB.
that's how to reduce cpu overhead. We use CUCI-pop for
small memory foot print. we also forces email off the server
every six weeks. there is a seperate Que and smtp machine.
I think a good quota, flush policy should be your first
concern. why make the mail server an anonymous ftp server?
There are two performance gotchas in Solaris you have to watch out for, because they will affect any simple configuration using Inetd-spawned POP and/or Sendmail.
Both of these performance problems will sucker you into thinking you've run out of CPU/disk horsepower, but it ain't so!
There's a filesystem implementation issue that bites you badly if you have multiple processes (like POP and Sendmail daemons) trying to create files in the mail spool directory. There an RFE for this one; in the meantime, you should absolutely use a Veritas filesystem for your mail spool.
There's also an Inetd-related performance problem. Hitting the server rapidly with requests that use Inetd to spawn off POP daemons will cause some of the processes to take a really long time to start up. My co-worker and I are working on this, but we have yet to characterize this fully or find a workaround.
The best thing I'd suggest is to use a daemon that avoids spawning new processes to handle incoming requests. Perhaps something that pre-spawns processes? Multiple processors probably do help the situation quite a bit.
I believe SIMS 3.5 (and later?) will avoid these problems. It's pricey (around $10 per user, non-discounted), but not *nearly* as expensive as even basic Exchange, which will eat you alive with hardware costs for multiple servers (figure 4000 per server), NT server licenses, Exchange server licenses, and client licenses.
Sun tells me that a loaded-up 450 will handle half a million mailboxes, and up to 5000 *concurrent* POP sessions. Even if that's inflated, it's still pretty impressive... And yes, it's probably overkill if all you're trying to do is offer basic POP/SMTP services.
Other gotchas: Users will want Web-based mail to overcome the stupidity of PC/NT mail client design, where all of a user's mail is kept on the local PC (not exactly convenient when the user is in a different office, across town, or across the ocean). Or they will want everything kept on the server, which radically increases the amount of disk space per user. All those damn attachments will make you start thinking in terabyte terms...
Oh. Find some way of enforcing a minimum automatic-check-for-new-mail time. Users will set their PCs to check every single minute of the day, driving up the average number of POP processes per second (which will make you run into the filesystem bug). Sigh...
ramct@iglou.com-remove.this.part
Oh yeah, right. 25K+ users on an NT box.
Been there. Went back to Solaris...
For fsck's sake man! Have you ever stress tested NT? IT SUCKS!!!!!
A *BSD (or linux) box can handle the same load with no problems whatsoever. Save yerself a headache. Go for a *nix solution. Really, I mean it.
I'm not biased. Experience has just led me to hate windows products. If you don't believe me, try them yourself. (hehehehehheheheheheh!!!)
*barf* windows sucks. But it's not as bad as AIX (which is pretty awful).
If you really must.. run solaris on SPARCs.
Well, my "Exchange nightmare" came about shortly
after installing the software on a plain-vanilla
Gateway NT4 machine -- which was replacing a
Slackware box.
Booted up for the first time, and Exchange
immediately pegged the CPU at 99%. It took fifteen
minutes for the Task Manager dialog to appear.
Took five more minutes to highlight the process,
and attempt to kill it. Five more minutes, and
NT4 said it couldn't kill the process, because it
had "run away".
After SEVENTEEN hours of Microsoft "tech support"
and having been transferred to various support
centers across the U.S., including (eventually)
being transferred to a PROGRAMMER on the Exchange
team, MS finally told me my only option was to
completely nuke the harddisk, and reinstall
everything.
I am NOT kidding. THEN MS had the umitigated gall
to charge my company FOR TWO support "incidents"
even though it was only ONE! The cost was added
to our phone bill -- we ended up telling AT&T that
they'd have to sue us in court to get the money,
and MS backed down.
. . . and all this for an office of ten people.
Needless to say, I punched a hole in the wall,
and then vowed to punch BillG if I ever meet him.
Yes, please write back with the final solution!
MIT has a network roughly that size. 5000 undergrads, 5000 grads, faculty, staff, a few "associates" and select alumni, etc. It has an e-mail system that hasn't had any problems in the 3 years I've been here. Everything is free software, and runs on Unix. You should contact MIT IS, and find out how they set it up. The system is really quite incredible.
In my experience, Exchange is not bad if you only have to serve several hundred, maybe a couple of thousand, mailboxes. For what this individual wants to do, I suspect a commercial UNIX system would be desirable.
If he must go with Microsoft, he may not want to go with Exchange to handle this many mailboxes. Microsoft has a product called MCIS Mail Server which is aimed at ISPs.
Have you considered tallying up the cost per user you're going to spend on hardware and support, and instead outsourcing the whole mess to someone like critical path?
( www.cp.net)
This isn't an ad for them, it's a serious question. At what number of users does outsourcing beat out Exchange? Linux?
postfix + cyrus + ldap = fast, flexible, scalable
>I wish there was something in the open-source community
>that had all the features of Outlook
You mean an email client that does ten-thousand things, but
none of them well? (not even the stated purpose of reading
and sending email?)
Did you heard of a new tool called the mail robustifier ?
I think it might interest you...
Go to http://www.bluetail.com/
I did not try it but I would like to...
Mickaël
Back in '93 I supported 87,000 users on 12 Pyramid servers (up to 4 R3000 cpus each) using mostly standard sendmail. It can be done quite nicely without paying M$ but check out sendmail.com...
-tim
web.abnormal.com
With MS exchange, a lot of the migration/management is hidden - only if you pay $$$ for the premium support database do you get to find out the limitations/bugs/features.
Its pretty obvious why it is not a soda going from lotus to MS or anything else.
MS=Cushy job, stacks of high $$ overtime, and slower than a snail when load balancing, and ongoing tweaking - if that helps
The clustering is done by proprietary code, but you can use any MTA you like (even mix them) and upgrading to new versions (of MTAs) should be painless.
There's a whitepaper for download at http://www.bluetail.com/products/wp.ps.gz
Felix
You need clients.
When you provide an e-mail solution you have to consider what the users (your customers) need to ge their job done.
MS-Outlook gives you an awful lot of groupware functions, but you are limited to POP3 or EXCHANGE servers. If you turn on IMAP, bang goes the groupware capability (honest, and its a bastard trick). You do have to option of turning off the mail function entirely and just running groupware. That means you need to roll out a separate mail client.
I mention this because Outlook is the Client that you will be up against, alongside Notes (Lotus does actually ship UNIX servers), but that is another big infrastructure - and one you could suggest evaluating).
If you are to remove Outlook from your list of products, then you are going to have to offer up a reasonable alternative for the desktop.
My suggestion would actually to look hard at a web-based mail & groupware solutions. They give you central admin, low operations costs and you can have calendaring etc. too. Some of these look to be open platforms (perl & php are popular start points, others involve java). Others are commercial. The main benefit is you - as an admin - only have to deploy a browser. You do not have to roll out a load of clients. It is an *enormous* saving.
In any case, start with a proper requirements excercise and properly understand the feature set the user community(ies) need.
Good luck.
I work for a company that runs around 30,000 Exchange seats. It works just fine (and has done for the last 18 months). It works fine because it was implemented and is managed in a professional manner by people who see it as a critical communication tool and not something that they can screw with as the pleasure takes them. Open Source messaging - yeah right.
Use a simple workstation as mail server? crazy! :-) I would go for the DS20 server with 2 or 4 Alpha 21268 processors and PCI64 gigabit ethernet NIC insteed.
I'm not recommending you use it yet, but ReiserFS is intended to deal with poor dir-lookup times.
Remember, however, that the version of Exchange the Senate was running was at least two major revisions out of date, running on (I think) NT 3.51.
That still doesn't mean it should be allowed to crash.
Um, close, but no cigar. Even Microsoft wouldn't be dumb enough to try and run Hotmail on Exchange, it's a completely different model. What they tried to run it on was the MCIS mail server which they're currently hawking around desperate to find people who haven't heard how unreliable and unscaleable it is yet.
their low-end product, post.office might suck, but intermail, which software.com claimed was designed for distributed, for millions of users, with accounts from GTE ..
(we too are a major isp and would be interested in your views. pls email yunib8@cyberdude.com for further discussion).
If you had problems with 50 people, maybe you should work at McDonalds. I've NT/EXCHANGE for 30,000+ users, with no problems/
(Australia's) Telstra big pond was using exchange for 20 + thousand users of their big pond internet service... until it crashed and they lost a weeks worth of messages.
And regarding Hotmail, Microsoft are still too shit scared to transition it from unix to NT.
!
No it's not. Exchange is great. We have >2000 users here. It -=NEVER=- goes down.
OK, it went down once, but that was because a Cabletron switch went down. The NT server running it was still fine.
That's the 'w' delivery agent flag (check in /etc/passwd). So, I suppose you could turn it off and handle the stuff in your delivery agent yourself -- I think if you exit with a certain error code, you can make sendmail return the right error message (maybe described in sysexits.h?)
Check out the Delivery Agent section in the Sendmail Bible.
To properly answer the question, a few more details are required as to what specifically you want to do. If all you care about is POP and IMAP , go the Solaris/Sun/Sendmail route. It's a very stable, robust, frequently used solution (I use it, and it's been rock-solid).
If, however, you want to possibly do more (calendar scheduling, internal apps, workflow tracking, full-text searching of mail, etc.), then I would suggest Lotus Domino running clustered across whatever OS/hardware you want (RS/6000, AS/400, NT, Solaris, Linux - later this year- whatever). Something very few people know about is the fact that Domino has its own clustering technology built in, platform independent. I set up about a dozen boxes for 122,000 users back in 1997, and they *love* the setup.
You're talking $2000 a server for the Domino license, and then the e-mail can either be accessed through the Notes client ($80 each, probably $25 each for 25,000 users) or through a browser (free). Both methods are encrypted and highly secure. The Notes client will look a little nicer, run a little chunkier (it's made to do a WHOLE lot more besides mail), but a browser is a very capable (and portable) means of reading your e-mail.
Nortel (who i did a couple of co-op terms with in the distant past) has switched to exchange.
and the number of problems (i do not work there, so i am not violating any NDA type things) that i have heard from friends who work there is truly phenomenal.
microsoft mail does NOT work well over about 25 users. it's crap-o-la.
More than exchange being a pain, it's a resource hog. We have an exchange server for ~1500 people that is a Quad capable, dual Xeon machine (1 meg cache) with 3 gig's of ram, and the ram/swap usage is fairly high (processor usage is pretty good though). Add to that the fact that we had to have a ~100 gig raid array for the message store!
Then we have a Linux box as mailserver/caching proxy/firewall that is a pII 350 with 256 megs of ram and a couple of 4.5 gig drives. It handles the same number of users but does so with not even a 1/4 of the hardware, and with better performance IMO.
For that reason alone Linux proves itself to be a much better solution.
Anonymouse
Another problem is that if Exchange is running and you have a client with Lotus Notes and this client "encrypts" his/her e-mail the Exchange "single file" is in trouble and will have a hard time restoring from tape!!
that's because you're incompetent.
I'm biased because I'm a developer working on OpenMail, but 25,000 users is within the scope of an OpenMail server. Of course, you need to answer questions about disk space, number of active users users, availability, etc etc.
OpenMail isn't Open Source and isn't likely to be, but it does work well on large systems. It also has the advantage that if people are baying for Exchange that it also supports Outlook '97.
Check out http://www.hp.com/go/OpenMail
I work for a large telecom company and I can assure you that if you are having memory problems with 3 gigs of ram and only 1.5k subscribers you have something in your configuration majorly screwed up. We have over 100 servers with approx. 1k-2.5k subscribers per machine. Our most powerful machines are three year old NCR worldmark 4300s with dual P.Pro200s and 512megs of ram. These machines usually run at about 50-60% processor load with insignificant usage of swap files. The amount of space that you need on the Exchange server has nothing to do with exchange "being a resource hog", but rather the storage limits that you wish to support. Exchanges single instance storage of email is perhaps one of the most efficient mail databases out there.
Your linux box is probably strictly pop3/imap, so it doesn't need the harddrive storage as users do not keep mail on the server and it has nowhere neer the functionality of exchange, Public Folders, collaboration, etc..
I suggest splitting the mail folder of your users onto a number of servers and crossmounting them so that /var/spool/mail/user is a file on its home server and a symlink to /mnt/server2/mail/user on other machines. This way each of many machines has access to every mailbox. Crossmount them completely with NFS so every machine sees every mailbox. Then, have distinct machines in the MX records to deposit mail. Encourage each user to use the machine that actually hosts their mailbox as the POP/IMAP/Webmail server.
This way, no single machine can bottleneck your entire mail system. No single failure can effect all users. (though each user could be effected by one specific failure)
I like the uw imap server for both POP and IMAP.
Until recently we had Exchange server on a beefy
intel box. The dumbass (menager in operations)
was comming every sunday to reboot the box 'cause
by the end of the week it was crawling like a bug.
Now we happily run sendmail on linux, all of the sudden all the problems magically disappeared.
if you think you said something smart, think again dodo.
Uh huh... And what company might that be with 30,000 exchange addresses? I don't know what your game is or why you're making up this little imaginary 30k-address exchange server, but you haven't been the first one to do so in this thread.
I run this and it can do that and that other thing sucks since I just proved that this can do that because I SAID what I run can do this, so it must be true.
Blah.
I run across people like you on both sides of the fence in nearly every OS vs OS or App vs App debate and/or argument. What in the hell do you have to gain, or protect... is it your own choice and hence your pride?
It kills me when I see blanket statements like this made. We have an Exchange system here with 700 local users on it, and 176 x.400 connectors supporting about 650,000 more users. Maybe you guys need to get some better admins?
AC
Well, hold on a sec, sparky -- have you ever worked with AIX in a large environment? We have about 20k users and we have two mail servers. Two. And they are old dual Power boxes with only 1GB RAM. We use SSA disk on a 7133 and mirror the internal, and we make sure that they stay healthy. They ge a good workout, and they serve about 45GB of mail a day. I think that that qualifies. Average uptime has been limited to 100 days largely because IBM keeps coming up with no-really-we-mean-it-this-time-final Y2K microcode updates and patches, but last year we only cycled them to move them, after 302 days. We can afford "better" -- we do have those 12 s70As fully loaded over there by the RAMAC arrays and the SP thicket, but we don't need to because the hardware and the software work just fine (we are using sendmail).
AIX has a lot of quirks and stuff that takes "too long." That is because AIX is tuned for mindbending loads and the whole focus is not "how can I make the user pleased with snappy skins" but rather "how do I maintain subsecond console response times with a load of 180." Don't know AIX until you have seen the truly amazing things that you can do with it.
My only experience with Exchange was at the UT Business School in Austin, where they brought it in last year, when they made everyone coming into the program buy crappy Dell laptops (right after I had bought a nice PII400, thank you very much). Outlook took 5-10 minutes to come up, and the culprit was apparently Exchange. This is such a joke, as the MBA program is apparently a "Microsoft University" and gets all of this stuff and support for free AND THEY STILL CAN'T GET IT WORKING.
... and it all begins again in a few short months.
The first years are all getting Thinkpads. One thing about the program (MBA, University of Texas at Austin) -- they get you used to being screwed by a faceless corporate MIS machine with a megalomaniac CIO at the helm. I never thought much about computers before UT, but now I want to get out of this place with my sanity intact and an MBA and go into MIS and fire everyone who resembles the jerks who "run" things in the basement.
It has been an education, oh yes
That sounds like Dell. They moved from one SS10 to four quad PPro boxes with radically more RAM and the system, which had previously delivered mail with boring regularity, became wildly unstable, sometimes not delivering mail for three days and then dropping 800 messages at once. It was an eye-opener. I am told the the problems have continued -- I am not longer there. But sendmail on a single CPU, slow SPARC beat 8x the CPUs and about 20x the disk like a gong. I don't need a lot more proof.
Can you post some hardware specifics please? As with a lot of posts here, I suspect that the glowing reports of Exchange reliability aren't coming from the systems people themselves...
Could you please post hardware and config specifics? I think that this, like a majority if not all of the glowing accounts of NT and Exchange performance, are coming from end users who see a pretty good job coming out of an ungodly expenditure of blood, sweat, and tears from the MIS people entirely out of proportion to what the job (delivering mail services) requires. Can you be specific, please?
Could you please post hardware and config specifics? I think that this, like a majority if not all of the glowing accounts of NT and Exchange performance, are coming from end users who see a pretty good job coming out of an ungodly expenditure of blood, sweat, and tears from the MIS people entirely out of proportion to what the job (delivering mail services) requires. Can you be specific, please? We would appreciate it.
Could you please post hardware and config specifics? I think that this, like a majority if not all of the glowing accounts of NT and Exchange performance, are coming from end users who see a pretty good job coming out of an ungodly expenditure of blood, sweat, and tears from the MIS people entirely out of proportion to what the job (delivering mail services) requires. Can you be specific, please? A 22k user enviroment would be impressive if this was done with an MIS budget comperable to that spent on a UNIX solution.
Why is it when someone comments that NT just doesn't work the way it's supposed to, the NT-advocate's argument is *ALWAYS* "you just have clueless admins" or "your admins should be fired" or some other crap about the admins not being competent. While there are incompetent admins, there are just WAY too many cases of shit not working on NT to blame it on the admin everytime. Yet in each case where the supposedly "clueluess" NT admin switches a function over to a Unix box, it just magically starts working for some reason. Here's why:
Properly configured Unix box = working product.
Properly configured NT box (is this even possible given the number of voodoo reg hacks and secret configurations needed?) = working product for the time being..
This is the one advantage I see in the Unix environment. If something is configured to work, it *stays* working. In the Windows domain, there's some bizarre, constantly changing variable that randomly screws with applications and the OS in an effort to undermine all of your hard work. If it's this freakin' difficult to admin NT properly, just give me a Linux box any day.
Jet is so well named -- sucks from one end and blows from the other.
I complained for years about sendmail. Then I was given Exchange. Now I think that sendmail is simple and reliable. Just a matter of perspective, I guess. I could never get more than 20 days of uptime (never broke three weeks -- never) and I was using 10x the hardware (replaced a dual PPro unit with five quad PPros with 9GB as opposed to 2GB disks, 512MB RAM as opposed to 128MB, etc., etc.).
"I believe it is possible to set up exchange a lot better than that, but apparently not by default."
Arghhh!! This is my main complaint with NT and it's server products. Since "everybody and his dog knows that product X should be configured as follows.." it would make sense that Microsoft would know this as well, correct?
So then why isn't it configured this way BY DEFAULT??!
Oh, I know why... So MS can charge thousands of dollars per person for it's half-assed MCSE courses.
16 gig limet is gone in latest version. Although letting it grow (40gig at one account) makes recovery a even more of a joke.
Exchange standard edition is limited to 16GB per server. Enterprise is 16TB per server.
But 3.51 is more stable.
Not all IMAP unix servers use one huge flat file for all mail (new/old/saved). I am not sure what configuration we have at work (although I am 100% positive it would be on a Sun machine... guess why...) and if you save your mail in a different folder, it saves it in your mail folder in a separate file than the inbox. They get on your case if you keep everything in your inbox but they don't care if you have 500 mb of crap in other mailboxes. If you tell people to move things to other folders or delete them from the inbox then IMAP should be OK even with 25000 users...
Os X Server is great, running Communigate Pro, its better than any Linux solution.
I worked at a National HMO for almost two years before I left. We used MSMail when I started working there, and moved to Exchange. We had vast resources for technical issues, yet still we had several issues with exchange. It seemed buggy, loss of data and the server crashing where the most common. We had techs from MS come in at times to help trouble shoot. Heres a fun example. NT Workstations lost user profiles when the user logged in. The Microsoft solution was to reinstall the project, all 1000 or so. We where lucky that we only upgraded a few out of the 6000 or so workstations. FYI, only 1200 or so where on exchange. Anyone else with e-mail would have still been on the old MSMail, or just didn't have it.
What the heck are you talking about? Outlook clients don't poll the server for mail. The Exchange server uses RPC's to inform the Outlook client of new mail which the Outlook client then retrieves. The only time that Outlook clients talk to the Exchange server is when the clients sends mail out or new mail arrives for the client.
Maybe you are talking about MS Mail. It did polling.
Do you need a videocard along with the Voodoo3 3000D AGR Graphics card. Also, where can I buy computer parts at low prices (internal zip, processors, motherboard, etc.)?
how this is not maked -1
Nobody mentioned Postfix as the mail server.
It's orders of magnitude faster than sendmail.
It's not a rumor. I jokingly asked a relative about this, seeing as how he sells sun machines for a living. Microsoft still buys them from his company. They tried to switch, and failed.
We use InterMail at my ISP and it works fairly well with about 3000 accounts. Although it as a nice problem with corupting users messages which then need to be deleted by hand. Other than that, it works fairly well on NT.
I'd love to see the $$$ figures to get the licensing and buy the clustering configuration that might be able to handle that.
I'm sure Exchange will fold up like a little girl with that sort of load on any setup.
my friend, i think i speak for most readers when i say that i believe you...
and that it's possible to have the experience you report with Exchange, et al.
i think i also speak for most readers when i say that we'd rather not have to work at Microsoft
in order to read our mail. if YOU couldn't use your own software, how could anyone else?
Hmm, Why not use a linux box (just use some good hardware, like UW2-SCSI to start with), and use Q-popper, its used by many ISP's.. works OK for me
Although they ahve not released the source they are mostly opensouce based. I am shure, given enough horsepower a Linux / FreeBSD solution with perl or php (or C if you are brave) could handle this.
Will 25,000 user licenses for Exchange not cost the euquivalent of the debt of a small African country - Why not send it on developers to make a (or another as IMP or TWIG may scale that big) free solution in C, Perl or PHP.
Greg
How in the fuck does this piece of repetition get a 3 point rating? It says nothing.
This will seem somewhat unorthodox, but I would highly recommend taking a look at BlitzMail from Dartmouth College. This solution uses an IMAP-like protocol (but supports POP too) and was originally designed for Mac clients, but its track-record alone makes worth taking a look at.
Consider this:
* Very, very efficient. A cluster of six NeXT boxes used to serve 15,000 accounts, and handled more than 200,000 e-mails a day. AFAIK, these machines have now been replaced with 6 Alphas.
* Unbeatable durability. It was first developed in 1987, and has been in continuous used for more than 10 years! Being a non-commercial product it has not suffered from featuritis or bloat. In other words, it has a very solid code-base.
* Excellent scaleability. It is designed so that you can simply slot in another machine in the cluster to improve performance.
* A great feature-set, in particular it has an excellent real-name based user-database as its core, with secure random number authentication.
And yes, it is free. Source code available etc.
Take a look at: http://www.dartmouth.edu/pages/softdev/blitz.html
Skip through the client bits and you will find links to the server software.
First of all, I would like you to understand a few things that I have ran into when looking at a solution of this size.
The company I work for has a user base of 19 thousand mail accounts. These run off of Microsoft Exchange on i386 NT boxes. Hold your breath, because getting this `solution' to work demands a cluster of no less than 60 NT machines. All running Exchange in a cluster. At least 10 of them crash each day due to heavy mail load. No, all you NT pundits out there, this has nothing to do with `bad drivers' or `poor configuration'. The amount of mail and their attachments flying around, craves this amount of machines. We have even checked in with Microsoft on this, and they have only told us that getting more redundancy will help this problem. Exchange, from my point of view is _not_ a good solution.
Then we called Sun. They brought 3 boxes, Sun Ultra Enterprise 450, with qmail and I do believe it was cyrus-imapd. This was because they did not have an `evaluation' of the mail software that was going to power these machines. They did let us borrow the boxes though... Funny i think. Anyway, they set it up during a day, and we let the machines run instead of the Exchange cluster. None of the boxes crashed. They kept going and delivering mail, without _one_ mail going astray for a whole 2 months. Nobody was complaining about mailservers being down or any other kind of cheesy messages in their mail clients.
Now here is the funny part. The chief of investments called up one day and told us that the sales department from Sun had contacted him to ask him if we wanted to keep the boxes or not. He had told them no, and they were going to pick them up two days after... That meant booting the NT cluster again... After about 30 hours of syncronizing Exchange machines, they were online. Sun came and picked up the machines. My boss said that the chief of investments had the firm belief that Microsoft was The Right Way to Go(TM). Now this decision was _not_ based on our impressions of stability, scalability and reliability. But rather a term of Microsoft buying executives too many lunches...
So my advise... If you don't want excessive work, stay away from Exchange. Call Sun. They will let you evaluate their boxes. Trust me, they are eagre to sell you their stuff. =)
I found this while going through the sendmail newsgroup on http://www.deja.com It's about the mail system that earthlink uses. It discusses how they use open source to build a scalable mail system.
A Highly Scalable Electronic Mail Service Using Open Systems
"...In September of 1997, EarthLink provided email service for over 350,000 subscribers with a 99.9+% service uptime record. In fact, we expect the current system to scale to well over 1,000,000 users without significant alteration of the architecture as presented....."
My company has over the last 4 months, migrated from a working email system, hosted on (Solaris/Sendmail) & clients (mailtool, Netscape, IE4, generic popmail programs), to Exchange/Outlook.
The Solaris system is a sparc 10, with a couple of gigabytes. I worked well for years, with little complaint.
The NT server, with just as much diskspace, has had backup problems, spaces problem (no quota system), is slow at accessing it's 'PST', the single file it stores mail in.
Additionally, our network calendar system, years in use, was dumped for The Exchange's calendar system. Exchange/Calendar is very limited in memory/diskspace usage, compared to Solaris.
And of course the NT manager did this on a shoestring, so no backup servers (Solaris does this easily) were bought.
Now for the best. The company start with Exchange servers at several sites. But has now consolidated to a single server. The other sites must put up with very slow mail access over TI or ISDN connections. Late afternoons many NT systems seem to hang, as the wait on the complex network file access to the Exchange server.
For my money then nt/exchange system is a waste. It's slow when you need it most, hard to backup, file structure is not easily adapteable to other mail program (like M$ would make a program interoperable).
Sendmail (on any good U*IX platform) is a bitch to setup, but reliable year after year. Any good pop3 or imap client can be used.
Go with the Gold, Sendmail!
Linux IMO is not a server OS, there are some concerns about security, speed and stability. Also the speed at which changes are made can be a problem in a production environment.
These concerns are, for the most part, unsubstantiated. Recent tests in c't magazine showed that Linux and FreeBSD perform nearly identically well for web serving; FreeBSD is slightly faster serving static content and Linux is slightly faster serving dynamic content. Linux' slow userland NFS is being supplanted by a kernel NFS daemon comparable in performance to FreeBSD's, and Linux is significantly farther along in SMP support than any of the BSDs.
Arguing about Linux' stability is absurd. Properly administered Linux boxes, like properly administered FreeBSD boxes, can stay up for a year or more at a time. In fact, IIRC, a Linux machine currently holds the uptime record between these two OSes.
WRT security: well, if one is prepared to admit NT to consideration as a server OS, one certainly cannot write off Linux as insecure. I am not aware of any outstanding security holes in the Linux kernel. Both Linux and FreeBSD are widely deployed commercially, have withstood considerable scrutiny, and are certainly suitable for anything short of mission-critical banking and military use.
The key theme here is that, in the hands of a competent sysadmin, either Linux or FreeBSD is an excellent choice for a server OS. If the recent influx of Linux newbies has managed to tweak the BSD camp by their exuberance, let me assure you that the recent rise of self-righteous BSD advocacy has been at least equally vexatious to experienced users of Linux and *BSD alike.
Let me apologize if this has come off as a flame, but it really is time for the Linux and *BSD circles to play nicely and get along with each other. Unsupported statements like "Linux IMO is not a server OS" are flamebait, pure and simple.
AC
I've done such a thing already with open source
and it works fine. All users are stored in
a QI/PH Database which is designed to handle a large amount of mail users and is integrated with sendmail.
For fetching the mails I mofified popper in a way,
that it searches for password and mailbox in the
Database as well.
All in all it was done within a few days easily.
Send a mail to 007@freemail.at for further infos if you are interessted in the system.
When I was working at a large state agency in Texas (rather well run -- Texas goverment is a mixed bag, from third world corruption to world class on the same state office building), I was trying to get a handle on WAN transfer times (Austin, Lubbock, SA, Houston, and so on). I discovered that pulling up Word files with embedded bitmaps was ... interesting. It seems that Word would load the bitmaps (these were signed forms, so they were large and high resolution) in the background as it loaded the document (from Lubbock or wherever, across a T1 to Austin), and, as the user scrolled down the document, LOAD IT AGAIN when the user got to it. Why, I don't know. But that was what was killing the WAN. Word. OK, I called M$ -- I had no real feelings towards them at the time. I did AIX, never worked with DOS and Word (6.0 at the time). M$ was not only completely uncooperative, they looked at the same sniffer logs that I had and claimed that they didn't see the same things. Finally, they called my boss and told him that I was not competent to be working on software as sophisticated as Word. I heard my boss screaming at them from across the floor. I later learned that he had done a lot with OS/2 before Windows95 and disliked M$.
So no, that isn't a unique story.
I'm familiar with one setup that used Netscape's messaging systems and they can become a total nightmare. There are lots of wierd quirks about the way it handles aliases, etc, that can be a real giant pain.
I've never tried any of the open source products on any large scale so I don't know how they perform. I wouldn't go with Exchange if you are looking to do good scalability. It would be nice to run the service on a big beefy box, and Sun boxes are far better than Intels.
Sendmail isn't the part that doesn't scale, it's POP3 when dealing with Sendmail files.
----
----
Open mind, insert foot.
That has got to be the single most idiotic error message that has ever come out of redmond. That, by far, beats error messages such as "Error 13, your system will reboot now" or anything like that.
:)
Exchange does suck. Thankfully I've only had to use it once in my life. I'll stick with sendmail and qmail.
Not server related, but I worked at one site of a very large manufacturer. For MS support they had to sign a support contract and pay for the pleasure, call a 1-900 number and pay again, and only one person was allowed to call it, or pay even more. The combination of having to pay for support, that one guy was usually sick/at "training"/etc, and the typical MS answer, "What, you installed non-MS software on this computer?? We can't help you," made for some interesting conversations with the boss. It was just accepted among the support staff that you delete/reinstall programs or the whole system anytime a problem got too complicated. I just can't understand how that is acceptable behavior for something so basic to your company's computer systems. Yet, every year they get themselves more and more entrenched in MS products and headaches.
Are there any web sites with business case studies of implementing open source solutions, be it good or bad? It's important to know when to hold 'em and when to fold 'em with regard to open source things. I've often wondered just how much say Samba, various MTAs, etc can handle. One central resource for this sort of stuff would be quite useful for me a least.
You don't need to license NT for 25000 users if all they are using is Exchange. However the licensing costs for Exchange alone will probably bankrupt you.
While I am not (thank god) the Exchange Admin, over the 2-3 years we've had it, we've had 2-3 complete mailbox corruption issues and lost the entire database. We've had 3 weeks or so total where it was unavailable (3 1-week blocks). :) I am frequently asked by management, why it's so slow.
It doesn't scale well at all (NT doesn't...so Exchange doesn't) and those boxes look in pain as they try to do what people ask them to. However, response-wise, it frequently "pauses" to check new mail and your composing is also paused...what a great product!
Unix solutions will be far less headache if anyone understands Unix in your environment and the support is gonna be worth training someone, versus paying Microsoft to vainly (in our case) strive to repair your dead Exchange databases/servers. Our company is STILL headed into a MS only environment and it's sickening. Ok...enough rant and rave.
On the Web server side, sadly, it's Windows NT and IIS/ASP, with some ASP components including a custom-written client-side mail store. We use the commercial AspHTTP component from ServerObjects to send requests to the mini CGI server on the Unix box when we need to create accounts. We also use AspMail and AspPOP3 to handle sending and receiving messages. (The mail server is firewalled, so you can't connect to it from the outside with POP3.)
Qmail is definitely industrial strength and free, two qualities that we appreciated. It's also easy to configure and fairly easy to customize. Recommended. Oh, and you can find the end result at www.webb.net.
Eric
--
Be who you are...and be it in style!
This sounds like an excellent project for FreeBSD to tackle. I've deployed large mail stores and mail servers on FreeBSD and BSD in general with no problems.
I've recently investigated into switching users from POP to IMAP and have realized that IMAP simply doesn't scale well. Client CPU for doing the type of operations IMAP does is cheap, while server CPU is expensive.
Cyrus and UW IMAP are not light solutions. They are very large binaries which run from inetd and are anything but clean implementations.
Really, I'd say stick to POP3 using something like Qpopper or go with a commercial vendor. Qpopper is probably the best POP3 server you are going to find in the OSS world, though it does still run from inetd.
Open source POP3/IMAP4 servers really are lacking in the Unix world once you get past a certain number of users.
--
The world is neither black nor white nor good nor evil, only many shades of CowboyNeal.
The senate uses it for, what, maybe 1000 users (Senators and support)... When they were flooded with email last year, the servers locked up totaly. Many messages for days on end were lost.
Remember, however, that the version of Exchange the Senate was running was at least two major revisions out of date, running on (I think) NT 3.51.
...phil
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
Groupwise? If your company is looking to have the stability of a 'name' behind it, then Groupwise might fit. It works nicely, scales well, is pretty stable, and a whole lot less expensive than Exchange / NT client licenses. ($55 per seat for Exchange solution, less than $20 per seat for Groupwise.) And, you can run it on free server software - Novell run time servers work great for this, since you don't use a Novell file server connection connection for mail. NDS for security and user management, and pretty decent performance on mid-range Intel software, too.
...phil
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
I know for a fact that Lockheed Martin uses Exchange for it's LM-Xpress email program that handles 175,000+ users. How's that for capable? The two important characteristics that they were looking at when selecting an email system were security and reliability (i.e. no lost emails), and Exchange has been a real success.
Hope you don't mind some questions. How many servers? How much maintenance staff for those servers? How stable is the system? Can you provide more details? We've received lots of details about Exchange systems that have failed, how about providing some more info on a successful system?
On a related note, anyone who says, "Don't use Exchange because Micro$soft sucks," really needs to be a little more open-minded.
Agreed. How do you respond to those that say 'Don't use exchange because it crashes all the time, and here's another example'?
...phil
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
How about stupidity?
/cgi-bin directory to IIS, Hotmail is still running FreeBSD and Apache.
n
He was talking about the user interface, not the mail server. The mail server is a hacked Qmail running on Solaris. The web servers executing the mod_perl code are FreeBSD.
And what does hotmail being owned by Microsoft have anything to do with it? Microsoft bought Hotmail after it had already been established, after all. Unless Microsoft has suddenly added a
bash-2.03$ telnet www.hotmail.com 80
Trying 216.33.151.7...
Connected to www.hotmail.com.
Escape character is '^]'.
HEAD / HTTP/1.0
HTTP/1.1 302 Found
Date: Fri, 30 Jul 1999 02:26:01 GMT
Server: Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b
Location: http://lc3.law5.hotmail.passport.com/cgi-bin/logi
Connection: close
Content-Type: text/html
-E
Send mail here if you want to reach me.
So: Microsoft Sales says that you should use NT because someone who doesn't know what they're doing can successfully configure and maintain it, thus saving on expensive Unix system administrators.
Then it crashes, and the Microsoft apologists say that it's because it takes an expert to install, configure, and maintain an NT installation, and of course it's going to crash if you have the janitor maintaining it.
Typical. Just typical.
My question: If you need a skilled system administrator in the first place, regardless of the operating system, where's the TCO benefit for the Microsoft software?
-E
Send mail here if you want to reach me.
I think it depends upon what you're doing and how you're doing it. POP is straight file slingin', little CPU involved. IMAP is, as you mention, a bit of a hog.
IMAP for 30,000 users on Linux or *BSD would require a cluster of machines for the front end, but you'd still only need one machine feeding the data into the Netapps on the back end. But the big iron Solaris solution definitely has the cojones to handle this situation without clustering, and will be tremendously easier to configure and maintain than the cluster. And also quite a bit more expensive hardware and software-wise. So it is a tradeoff, and it depends upon how much in-house talent you have. If you have a lot of inhouse talent, the Linux cluster will save quite a bit of money. If you don't, the Solaris solution will require less expensive consultant time to set up and configure, meaning it will be the more cost effective solution. Wanted a simple answer? The only simple answer I can give is "Don't use NT, at least not with Exchange" (grin).
-E
Send mail here if you want to reach me.
OK, here's the scoop:
Hotmail runs many email servers behind an ip switch (ala F5's BigIP, Ipivot's Broker, Alteon's switch, Cisco's LocalDirector, LinuxDirector, etc). The responses gained from queso or nmap are the answers from the load balancing box, not the mail servers. This has been experimentaly proven and confirmed.
According to all of their press releases they run sun hardware and solaris.
Hotmail runs a (heavily) modified Zmailer for inbound connections, and local delivery, probably in some failsafe and distributed manner.
Hotmail uses qmail for delivery, outbound. This version of qmail is also modified, at least somewhat, and probably a *lot* by now.
-Peter
== Just my opinion(s)
Here are some ideas that have been used to good effect at other sites, based on anecdotal evidence:
:)
Qmail - Fast mta, reliable NFS delivery in the Maildir format. Others may vote for PostFix. I don't know it, but it may do as well.
Netapp - Fast nfs server. Multiple mail server can write to it. There are other NFS solutions from much higher-end storage vendors like MTI and EMC, but that may be more then you want
F5 makes BigIP, a load-balancing ip switch. This lets you multiplex things like pop3, imap, and smtp to a farm of servers, and elimintas the possibility that anyone won't be able to check their mail or won't be able to receive email.
Cyrus is an imap server. It's pretty good from all I hear. I still have yet to implement it. The other 3 parts I've used a lot, and love 'em.
-Peter
== Just my opinion(s)
The original poster was an open source advocate. They wanted to be sure that people couldn't say that this was a notes installation or a groupwise installation that the anecdote refers to (though I've heard that these systems have their own issues).
-Peter
== Just my opinion(s)
SIMS sucks, and is unnecessary and high overhead for this application. I'd chage the recommendation to using veritas filesystem and volume manager w/ a clariion (or other) array, so you get the faster filesystem (essential for mail-queue types of applications - lots of small writes and creates) and allows for live resizing of the filesystem.
Use cyrus or something else for imap. The above system, according to sun sales literature from about 1.5 years ago regarding SIMS, should support about 250 users. Plus there's no way to recover the SIMS database that I've heard of if there's corruption.
Good Luck!
-Peter
== Just my opinion(s)
I highly doubt that every employee at every rent-a-car location has email access or even email access.
-Peter
== Just my opinion(s)
Yeah, this is a real filesystem issue you're running into. The time to traverse the linked list that is a directory structures is quite long when you have more then about 1024 files in a directory. The answer in the future is using a different directory structure, and in the short run is to hash things like active mail spools across multiple directories (see qmail's bigtodo patch from Russ Nelson).
FreeBSD probably would have served you better then linux in this case as well.
Question: anyone know if future filesystems will address this issue, or is this outside of the filesystem and in the kernel?
-Peter
== Just my opinion(s)
Performance problems at the outset:
1) syslog - for this much volume, syslog will slow your system a lot.
2) Qpopper requires a read through the entire mail file for that user each time mail is checked. For a user with a couple of megabytes of crap (think attachments) this can be a few seconds worth of activity just to get the first 5 lines of each message. Solution? Use the maildir format, which gives each message a file. Don't use MH Mail file format. Why? Because mh will do ungodly amounts of rename() calls each time the user deletes a message from the middle of their mailbox. Maildir is much more efficient.
3) Sendmail takes a lot of tuning to meet this sort of demand. Sendmail also has a large footprint. Using a mail server like qmail (my pref) or postfix (others' pref) will buy you a lot of performance for a one time learning curve of about a week's time, without having to guess at how to get high-capacity out of the system.
4) Linux is good, but unfortunately if you're going to do this on a local file system for a system with 25,000 users you need to have a lot of space. I think a Journaled (sp?) filesystem is called for here. Currently for supported tools that really means a commercial unix. I've used solaris and veritas' filesystems a lot, and I know that for a mail queue and for mail delivery veritas does amazing this. In addition, it makes recovery in the case of a system crash amazingly fast, and its snapshot facility allows you options to backup that are better then that is usually available on a mail system (i.e. minimal to no downtime to perform a backup of a stable image of the filesystem).
Anyway, hopefully I've contributed some useful thoughts to this!
-Peter
== Just my opinion(s)
I've mentioned this in a couple of other posts here, but since you've directly addressed the shortcoming of the standard unix mailbox format, I have to chime in that qmail's Maildir format addresses the weakness of having to read the whole mail spool, without incurring the overhead of using a remote database (development overhead, mostly, but also a lot of overhead in terms of program size and complexity).
-Peter
== Just my opinion(s)
Sendmail 8.9, too!
It's worth noting that qmail's maildir format is very efficient at handling large numbers of email messages. It also stores the messages in the users home directory which makes the system a bit more secure and easier to track down disk usage. Of course it supports the "standard" unix mailbox format if you want to run with unpatched versions of IMAP/POP tools(why?).
One other nice thing about InetMail is that incoming mail is stored in multiple files, so the POP3 server doesn't have to churn through large mailspools when fetching mail.
I believe.. But that's still no excuse for the substandard spam-handling of Exchange versions prior to 5.5SP1, you could control which IPs clients can connect from (tcp-wrappers, basically), but that's a layer 3 solution (and a bad one, at that) to a layer 4 problem.
-o
an MCSE who hasn't used MS products in 15+ months..
I've seen some serious Exchange installations. 25000 users is far too much for one exchange machine to handle. I've seen 2000+ users on a quad processor Alhpaserver 4100 with an exchange database over 300GB and growing quickly. You can do 25000 users but you will need many servers each with their own RAID array and some serious, dedicated, professional management. POP3 might be a better option since the server requirements per user are lower.
I browse with my threshold at 2 so I can't read my own comments :-)
Why obscure the vendors name, when its obvious that the vendor here is Microsoft? Unless there is some other OS and MTA vendor in Redmond Washington...
--
Python
Python
As for the MTA, well I've seen plenty of votes for Qmail and Exim. I'm still pretty partial to Sendmail though. I think they'll all work (though I've been told Sendmail on a single server probably would have a tough time keeping up with the load on something like this).
With all of these solutions, if the users are getting much mail, you'll probably have to do something kind of exotic to break up the mail requests across multiple servers. The more transparent you can make this, the better. Either the users will have to know WHICH server thier mail goes on, or you will have to make the multiple servers ALL have access to ALL the mail.
One possible solution would be to use something like CODA (also from CMU http://www.coda.cs.cmu.edu/). This is a cacheing network file system that you could set up on a backend server (running over something like a multi-ported 100Mbps Ethernet Switch with the multiple client servers on the front-end exposed to the network). When client server "x" gets a request from "joe", "x" accesses the file system and gets all the files in joe's mail box (a series of directories) (the ones requested first, then pre-caching all the others). When joe stops using his files, they are allowed to expire on "x" (releasing the cache for use by mary, or adam). Once downloaded, the files can be manipulated on the client and changes are sent to the server when there is time/bandwidth (I'm not sure how the locking and similar mechanisms work on this ... read the coda docs for details).
This way, you can dedicate one or more MTA servers to stuffing mail into the backend CODA server, then have one or more client servers pulling the data out and handing it to the clients. You spend most of your money getting a BSB (Big Stinking Box) for the backend, and use cheap, easily-replacable-if-it-crashes machines for the front end.
Another nice thing about Cyrus: It allows you to set per-user space limitations and access restrictions, and mail sent to multiple users is put into a special cache directory meaning it doesn't take up space for each copy.
One warning: Cyrus suffers from the same problem as INN's traditional storage system - it eats the hell out of inodes because each message is a file. Most email messages are in the 1-2 K range, so when you create the filesystem for Cyrus, make sure to create the maximum Inodes.
I know from the docs that CMU uses this on a 10000+ user mail network, and they apparently are quite happy withit. I've heard similar things from other large sites.
Basically, Cyrus is what Exchange hoped to be :^).
Don
"I don't think software should necessarily be free
Just in case anyone didn't know this, the Cyrus IMAPd fully supports public folders. Scheduling is another matter. For that you need an open source calendaring project. ;-)
sigs are a waste of space
The Cyrus server at CMU is probably your best bet. You'll find it at at this link.
It's worth noting that this project is currently supporting all of CMU's e-mail needs. It's also my understanding that it forms the basis for Netscape's Message Server and Post.Office. This should satisfy any concerns about it's scalability. It has lots of handy features like kerberos authentication, a database style message repository, support for ACAP, etc.
Alternatively try QMail. Personally, while I think it provides better SMTP performance than Sendmail, I'd rather use the Cyrus IMAP server than the UW one (the only one supported by QMail). You could go with using a combo of sendmail|postfix + Cyrus for incomming mail (i.e. what your MX records point to) and QMail for outgoing mail. It depends on your performance needs
Exchange Server is NOTORIOUS for being both difficult and expensive when you need it to scale to a large number of users, although I understand it's improved substancially since the 4.x days when it was just impossible.
sigs are a waste of space
I concur....
for 25k users, I'd go with a higher-class system to support the load that they'd generate.
I'd recommend running Sun SPARC-based systems running Solaris 2.7 and InterMail, with a fiber-channel storage solution like a A5000. The multi-threaded design of both Solaris and InterMail will do you much better than any Linux/*BSD solution.
To put greg's comment into perspective, we have 1200 users on site here, and we have 3 Exchange servers. 2 servers couldn't handle it. Each server is a Compaq Presario (2xP6 @ 350MHz, I think), with 512Mb RAM and several GB HD space.
I'd seriously look at all other solutions, wether commercial or OpenSource.
Sorry man, throwing more CPUs at the problem of mail is not the solution. What you need to do is get a very fast SCSI/raid controller and some high-RPM low-latency/seek time LVD drives or something. Large CPU won't help, since mail is mostly a matter of throwing files from disk to disk over the network. I mean, this isn't heavy rendering or anything.
So get a good, fast disk subsystem and attach it to whatever Unixoid OS you like. Run Qmail if you like (it's a bitch to configure, and not as versatile as sendmail, and it's not open source but it's fast), or get a bunch of good sendmail admins.
You can configure any mailer to handle large load on the SMTP side of things by using multiple MX records and mail relays. POP and IMAP are a little tougher.
--
I noticed
--
I noticed
It's getting about time to leave everywhere
Are you serious? Exchange at this level of usage would be absolutely nuts, but Notes? Puh-leeze.
Nobody buys Domino for the mail capability, they buy it for the groupware and application system, and they *accept* the mail function that comes with it. The client UI is so amazingly bad that a UI design site did an in-depth critique, attacking pretty much the entire app. (http://www.iarchitect.com/lotus.htm)
The server is difficult to administer; the user directory is flat, slow and scales badly; and a given user's mail store only *looks* hierarchical, but is actually just one big lump of data which must be read in its entirety, first-in-first-out, for many applications, especially connectivity with third-party apps attempting to access the VIM subsystem (which is itself pretty awful) for messaging. This means that anything attempting to read a Notes message will read every single thing in the mailfile in order of its creation until it finds what it is looking for, including any archived, sent, and deleted items.
The client is difficult to navigate, crash-prone, and really beats the hell out of a network.
Hopefully the newer versions are not so bad as 4.1, but I was involved in a rollout of Notes to a mere 600 users at a chemical plant two years ago. Notes was hosted on a single high end NT/Intel box, which should be sufficient for the intended usage. Users were sent to training classes on Notes in small groups twice a day, and their machines were converted to the new mail system while they were in the class. So we knew how many users would be on the system on any given day throughout the rollout. The guy onsite from Bay Networks and I made a bet on when the network would crash from the load - I was within an *hour*. We didn't even get halfway through the deployment before the net died. Not the server, the network. Admittedly, the network was sufficiently stressed and just plain odd at the best of times, which is why the Bay guy had a desk, but 300 casual mail users should not kill a production network.
(One of the plant managers was upset by the burning fuse I drew up on a whiteboard in the MIS room and updated daily. After I was proven to be exactly correct in my forecast, he bought us lunch.)
Domino is not a mail system. If you don't need all of the other whiz-bang features of Notes/Domino, don't do it. For groupware it kicks some serious ass, really doesn't have any serious competition, but for straight mail you are better off with nearly anything else. Except Exchange.
-reemul
who dumped Notes 5 testing on someone else just this morning, and is still smiling
You're just jealous 'cuz the voices talk to *me*
Certainly with Qmail, probably with other MTAs, you need to think about filesystems. mbox mailboxes should NEVER be on an NFS mount, they're bonud to get corrupted (dodgy locking). Qmail's maildir format works fine over NFS, but not all clients like it (no matter if you use a pop server).
ext2fs returns from write calls while the metadata may still be in RAM cache (making it fast) -- if you want complete end-to-end reliability, that won't do. XFS will fix this for Linux, (as does a tiny patch Linus sent to the qmail mailing list) otherwise FreeBSD may be a better free UNIX to use.
--
I've had good luck using the Cyrus IMAP server and the Exim mail transfer agent (MTA). The Cyrus server handles POP3 and IMAP, and stores the mail in an internal file per message format, and is designed for hosting mailboxes for those without accounts on the system. I've found both Exim and Cyrus to be fast, secure, scalable, and stable for thousands of customers, and I don't see any trouble scaling it further.
the solution we use (for +150k users) is to have users divided up and we have a mapping server that does nothing but map user to servers. we have eight sun E5k's that handle the users and two of them run (cough...) Notes. The others run sendmail.
A couple of them feed into the mainframe systems.
The only ones that ever give us problems are the notes servers.
overall it works very well.
get your dirty sig off me, you filthy APE!
Have any of you ever administered the pgpkeyserver that works on some servers on the internet?
Ok, me neither, but the server I work at has a pgpkeyserver, run by our admins.
We HAD to change to qmail, because sendmail wasn't able to cooperate with the thousands of emails we have to handle in about half an hour (about 7200, which is about 4 incoming mails per second, this being a low number) when the pgpservers synchronized.
Little side note: we run a 2.0.36 linux kernel, on a single 233mhz intel pentium 2 with 128Mb of ram, and sometime we reach loads of 24 or more when the pgpservers synchronize.
It is one public key per mail.
Bulk email, open source, qmail rulez.
Regards
Ya, cyrus works wonderfully @ CMU...now if someone would only get sieve working .
-Harry
Setup at the old ISP was this
3 ultra 2's
512MB RAM each
30GB array by artecon that was NFS mounted.
This was slightly overkill for us. A few this to keep in mind.
1. Have more then on machine running this. I would say use 4 PII's. Use dns round robin for load balancing. If you have the money get a real load balancer. With an NFS disk array and sendmail file locking this isn't hard to administer.
2. Use as much RAM as you can afford. 512MB min. 1GB to 2GB is better.
3. Fast local disks. QPop servers files locally. Have at least 4 gig for mail to be queued. We had at least one user trying to cycle 2 gig attachments through our machines every month, Bastards.
4. Set up qpop (or whatever) in server mode. This will decrease the traffic from you to your raid array. Server mode tels it that it is transfering data across the network, if nothing in the data changes just revert and don't move the extrea traffic.
5. Disk. We were fine with 30, but upgraded to 50GB with the last upgrade. Artecon NFS mounted to the 3 machines. Look at Netapps though. You can cluster them in a failover config. I have heard of some hardware problems with them though. We pushed 16MB/sec across a peak so make sure you are AT LEAST ultra wide if not ultra2. You could set up a server with the disk attached and let it do your NFS instead of a full network disk thingy.
6. I'd suggest sendmail, but qmail is nice, too. Sendmail seems easier to set up to use with pine if you want to hook a shell machine up to it with Pine or use a webmail package, but honestly I haven't played with Qmail much.
Kashani
- Why is the ninja... so deadly?
we user
/etc/passwd for mail delivery only which immediately shot load up to 12.
/var/mail/u/s/user for ours with an Oracle backend for authentication. We tried switching back to
Kashani
- Why is the ninja... so deadly?
It's heavily multithreaded, so the performance is excellent. I couldn't say whether it's ever been used with 25,000 users, though.
Timur Tabi
Remove "nospam_" from email address
Problems: If you don't buy the enterprise version, you have a limit of (I think it was) 17Gig TOTAL of mail... Which came out to about 40 users on our system until we started bitching at people to delete old mail. It really bites because exchange shuts down with no warning when you hit the limit. If you must use exchange, get the enterprise edition, which has no database size limit.
Exchange's database under 5.0 is guaranteed to corrupt eventually. You need to shut down regularly (at least once a month) and do an ISINTEG, or you get fun things like the President's email being delivered to the Janitor with a header from the Lawyer. 5.5 is a lot more stable, but I still don't trust it enough to not want to run ISINTEG once in a while. To date though, 5.5 has yet to corrupt.
Make backups!
Consider a massive farm of smaller Exchanges rather than one large box, and reserve one day a month of downtime for maintenance BEFORE deploying users.
If you go for a unix box, I'll give you my metric. While working at an ISP, we had a limited budget due to accounting problems (ie: no accountant, so we didn't know how much cash we had. *SIGH*), and we had a unix box in desparate need of upgrading, but it still ran:
486DX2 66MHz
narrow SCSI-1 drives
FULL news feed
400 domains
2000 mailboxes locally
routing mail for another 2000
web server
shell accounts
anonymous FTP
96 modems doing SLIP, and PPP at 28.8
Accounting
logging
and to add insult to injury, it was SCO!
I wouldn't ask that much out of ANY single machine if I had the choice, and it was SUCH a relief when we got 4 Alphas running OSF/1 (and some telebits) to replace the tired SCO, but you know it can be done with a solid OS. So a 486 properly configured with fast disk and lots of RAM should be able to handle 2 to 4 thousand users. Use of Cyrus or similar types of software will drag that number up, otherwise large mailboxes drag that number down (LARGE POP3 mailboxes on IO bound machines can actually take too long to scan. If POP3 doesn't report the number of messages fast enough, POP3 clients will time out).
Today, if I were you, I'd get a Sun Netra T1, and add a FibreChannel RAID card, with as much ram as you can cram in (on the motherboard AND the RAID card). That should handle your 25k users on one box in a pinch, but use two at least. Spreading the people out makes everyone breathe easier.
I setup and managed a mail system with over 50k users for the state of Oklahoma.
We used Control Data's MailHub. A system built on a portable x500 database send/pop mail and Perl scripts. It ran on a SUN spark box.
These days you could replace the hold thing with an LDAP server and Linux.
I'd use large Intel box (dual PentumIII) with 128m of memory and 19gig of hard disk, mirrored. This would be a good start. Strip down the Linux system, only run mail on it. Put the LDAP on another box same size and power.
I'd use Open LDAP for the user server. Then write some Perl scripts (Web forms) to update and manage the LDAP data. (The PHP module for Apache would work good too.)
If needed, use a round robin DNS to balance the load between several POP servers. These servers will all need to connect up to one large file server. I'm using IBM's AFS file system. It's better then NFS.
Mark@grennan.com
There are 10 type of people in the world, those who understand binary and those who don't.
I setup and managed a mail system with over 50k users for the state of Oklahoma.
We used Control Data's MailHub. A system built on a portable x500 database send/pop mail and Perl scripts. It ran on a SUN spark box.
These days you could replace the hold thing with an LDAP server and Linux.
I'd use large Intel box (dual PentumIII) with 128m of memory and 19gig of hard disk, mirrored. This would be a good start. Strip down the Linux system, only run mail on it. Put the LDAP on another box same size and power.
I'd use Open LDAP for the user server. Then write some Perl scripts (Web forms) to update and manage the LDAP data. (The PHP module for Apache would work good too.)
If needed, use a round robin DNS to balance the load between several POP servers. These servers will all need to connect up to one large file server. I'm using IBM's AFS file system. It's better then NFS.
Mark@grennan.com
There are 10 type of people in the world, those who understand binary and those who don't.
qmail is actually not Open Source.
:)
From http://pobox.com/~djb/qmail/dist.html
"If you want to distribute modified versions of
qmail (including ports, no matter how minor the
changes are) you'll have to get my approval."
Please reply in email if you feel the need, I'd
rather not start a flamewar here
--
Kevin Doherty
kdoherty+slashdot@jurai.net
Kevin Doherty
kdoherty+slashdot@jurai.net
Acourding to Microsoft, Hotmail _is_ run on top of Solaris. Check out http://www.microsoft.com/ntserver/web/news/msnw/Ho tmail.asp
They also state that along with Solaris being used, Windows NT is also used, but they fail to mention how/where it is used, so my guess would be as devel, and not production.
My favorite quote from the article is "Solaris is Hotmail's legacy production operating system". bwuahahaha.
On what kind of hardware? Are you actually suggesting 25,000 users on a single box? That would be almost two orders of magnitude beyond anything that I've seen in real production. More detail, please.
I think not...(*poof*)
As other posters have noted, large installations of Exchange usually mean very high maintenance, and as a result, very high cost. Running 1200 users on a single server is an entirely different animal than running 1200 users per server on a 25-server installation. The fact that your installation works well should be taken as a tribute to your planning, implementation, and maintenance skills. The average shows that it doesn't usually work that well.
My statements and projections are based on my experience, but I don't pretend to know everything. I worked on the Exchange team at Microsoft (a chapter in my life that I sometimes like to forget). I worked very closely with the system and component test teams, and with several customers trying to implement multi-thousand-seat installations. I wrote most of the Exchange Administrator's Guide, and a good chunk of the Concepts & Planning Guide. BUT my opinions are just that: opinions. You obviously planned your implementation well, and it works for you. In my experience/opinion, you are doing much better than average, and you should stick with Exchange.
I think not...(*poof*)
I say with relative authority: Puh-leez!
MS would like people to believe that Exchange is an enterprise-level communications tool, when it fact it is a buchered and bloated decendant of a mediocre 1992 X.400 email system from Data Connection Limited (check out http://www.datcon.co.uk/press/messserv.h tm) Don't believe the version number; Exchange is in its second major release (4.x really is 1.x, 5.x = 2.x, etc) and still has significant stability problems.
In my experience, Exchange can support 300 users per server happily on commonly acceptable x86 corporate server hardware (say, a 2 processor PII with 512mb ram). It seems that (in my limited experience, lest MS lawyers take this to be a declaration of fact, which it is not) once you've reached this level, doubling the ram and adding more cpu's has only a minimal effect, which means that you really have to add more servers to add capacity.
Let's do the math. 25,000 users at 500 users per server (to be quite generous) means that you're going to need a Windows NT server farm of about 50 systems just to do email. Again, being generous bargain hunters, let's say you can buy one of these servers for $10kUS. That means you're out $500,000 just for hardware. In my experience, you can support 500 POP users easily on a SPARC 2 or IPX, which can be had these days for about $500 decked out (including a 17" monitor). You could support the same (probably many more) on a $500 x86 box running any of the free *nixes. Assume you blow $500 on disk storage for these boxen just to level the starting line, bringing the total cost to $1000 per. That's still only $50,000.
One less zero usually gets the accountants' attention on an expenditure like this.
But let's talk about administrative support. IMHO you're going to need 1:1 admin per NT server at that usage level, given that remote admin of NT is difficult, and 500 users per server is going to prompt more than the occasional pretty blue interface. (Nevermind the security team you're going to need for a major NT installation.) Say a cheap NT admin costs $50kUS including benefits & overhead. You're looking at an HR budget of $2,500,000us. On the other hand, say you splurge and spend $150kUS per *nix admin. If they couldn't handle 10 little boxen apiece, I'll eat the electrons this was posted with. That's an HR budget of $750,000us.
That's 1/10th the hardware expense and 1/3 the maintenance expense of using Exchange. And that's (a) making some wild assumptions that benefit the Exchange argument, and (b) assumes that you're running *nix on shit hardware. Spend 5 times as much on hardware for new, supported stuff (say $250,000us, which would buy you a couple of well-outfitted Sparc 4500s, or 10 really gorgeous systems from VA Research). Your downtime will become next to nothing, you'll still have spent only half of what you would have for NT and Exchange, and your ongoing yearly administrative cost will be 1/3 of the other option. The *nix administrative savings alone will pay for the *nix hardware in a few months.
Oh yeah. I forgot the expense of 50 copies of Windows NT, 50 copies of Exchange Server, and 25,000 client licenses... (*erk*!!)
I think not...(*poof*)
They use Linux boxes with a exim as their SMTP system and a proprietary POP daemon, 9 pop hosts.
1.5 million users (so far), I don't know the number concurrent. They have very recently been quoted on the stock market.
Deleted
I love these guys...
/
http://www.freeyellow.com/members7/geraldholmes
Deleted
With 25000 users, scaling is your largest issue. For this reason, don't go with WinNT.
First off, storage for your pop-server. Assuming an average 1mb per mbox, you'll need about 25gig of disk. You could stripe (raid0) across 4 disks, but this multiples risk. I'd recommend 4 20gb Ultra/33(or 66) ide drives and a "new" motherboard that supports them. This will give you ~60gb of raid5 storage (and you'll still bottleneck on 100-BT before you bottle on disk-IO). I'd hate to have 25000 users asking me where all of their mail went. (make sure you have a reliable backup solution!)
Next, build a few "outgoing" smtp-boxes (call them {smtp1,smtp2,...,smtpn.yourdomain) Configure sendmail to deliver local-mail to go to pop.yourdomain (the above machine). Single 10-15gb disks _should_ be plenty to hold your mqueue's (you probably don't want them growing that large anyway. When your queues start to back up, there's no way out.)
Lastly, get a Cisco Local Director (or some other form of load-balancing). Configure a virtual called smtp.yourdomain with smtp1,smtp2,...,smtpn as its component real-servers. Advantages here are that you can quickly add new (clone) machines to the pool, create new pools, etc.
You could additionally try getting other pop machines, mounting /var/spool/mail via NFS, creating a virtual in the LD, and putting them behind that.
/* MAGIC THEATRE
ENTRANCE NOT FOR EVERYBODY
MADMEN ONLY */
Many of these situations prefer to work in an e-mail only situation -- no shell accounts, no home directories, etc. Sendmail, I know, looks for accounts by checking passwd to determine if delivery to a given mailbox is valid. How can you do the same if people don't actually have an account on the box?
Cyrus is probably the way to go. CMU uses it, and it seems to handle their needs. I've used it at three different companies, and particularly liked the fact that it doesn't rely at all on system users, except for password authentication. Its trivial to patch the system to authenticate passwords against a DBM file or MySQL.
Its also trivial to write scripts to automate the management of the server, so you can create a new user quickly and easily.
Two years ago I installed Cyrus at a company that was using NT domain servers for their logins on all the client machines. Quick patch to Cyrus to work with PAM, and a SMB PAM module, and people were able to check their mail using their NT passwords without having any security issues of having all those users on the mail server.
I also hacked something together that automagically created the mailbox when an IMAP connection was attempted with a username/password on the NT domain that a mailbox didn't already exist for, so the NT-centric admins didn't need to ever touch the mailserver.
The number of users are much smaller, but other installations have shown that Cyrus will scale, so the ability to extend it like this is also important.
It's about time that you checked out the qmail website http://www.qmail.org thoroughly. Qmail matches sendmail feature to feature. It has long had support for Real Time Blackhole list.
The thing about qmail is that it is modular in design, so it is necessary to search for the required component. And each component can be used independent of qmail. If you want Real Time Blackhole list then just grab the rblsmtp daemon.
As well there are some very useful independent packages at the qmail such as tcpserver which is a complete replacement for inetd and cyclog which is suggested as a replacement for syslog under heavy loads.
Qmail was remarkably easy to set up and in fact I've done the deed for my ISP who have been running it happily for quite a whil.
M$ SexChange is definitely _not_ the way to go. Other M$ products to avoid. M$ Expel. (popular spread sheet) M$ Abcess. (weak data base) M$ Turd. (popular turd processor)
At the UKUUG Linux '99 conference there was a presentation explaining how they implemented a large scalable mail server using open source software. It was also explained how the total cost of ownership would be much higher if they used an NT solution (even using the figures supplied by MS they'd need more machines and more administrators to keep the servers runnig).
The open source solution was much more cost effective and has proved fairly stable.
Unfortunately the proceedings from the event are not yet online, however I'll try and forward you a copy (or post a link to this thread) as it may prove useful to you.
--
I can't really offer any suggestions beyond what the gurus around here have already given, but as someone who's attempted large-scale Exchange set-ups, I can definitely say:
For God's sake, dont. If you'd like me to sign something saying I've seen havily-used Exchange servers take over an hour to shut down in a power outage, and then wipe out a few thousand users' mail when the UPS finally gave out, I'm more than willing to.
It's a bad, bad idea.
You want to make sure that your server can efficiently retrieve data from large individual mailspools if you're going to be dealing with large attachments. More traditional mail servers do a pretty poor job of handling mailboxes > 10Mb. You may want to look at modifying an existing MTA to utilize a database or purchasing one that is database-driven to best suit your needs. Hope this helps!
David E. Weekly (dew, Think)
David E. Weekly
Code / Think / Teach / Learn
h4x0r for
Without endorsing any specific approach, when you do your architecture design, look for opportunities to make it distributable. In general it's always cheaper to buy ten average PCs than it is to buy a single quad-xeon or Sun E10k or such -- though the job may not require that much if you do your engineering right. In the scenario you've described, look for use of multiple POP/IMAP servers sharing a common data repository (or, if it's not essential to have everyone share the same disk, divide that duty up also). Check out the high-availability Linux project for useful tools. Even if you wind up building things on one big server rather than several littler ones, leave room in your software setup so that you can distribute the load later on if need be without major reengineering.
And stay the hell away from Exchange. The degree of pain to be had from that pile of dung defies description, and the pain seems to grow exponentially rather than linearly as demand rises.
I'm short on time, but I wish to submit what may be the ultimate Exchange story:
A sysadmin at, ahem, a "large jeans manufacturer" was put in charge of Exchange on hundreds of NT servers. He dutifully logged and reported dozens of bugs, system outages, etc., to MS support, as the thing crashed and burned like the Hindenburg II. After a few months of this, Microsoft decided to act on the problems. The solution was simple: they sent a letter to his boss saying he was a troublemaker.
---- "If we have to go on with these damned quantum jumps, then I'm sorry that I ever got involved" - Erwin Schrodinger
Maybe I'm missing something here, but what is it about QMail that allows you to get around 16bit UIDs? Do you just use vpasswd and users/assign for everyone or is there some less, err, clunky way in which it helps?
(We use Qmail and love it, BTW - It saved out old mail server from death when we finally dumped qpopper)
"don't fall into the fallacy of believing that Perl can solve social problems. Maybe Perl 6 can, but that's a ways off"
qmail can do what you want. It's open source, free and very well done.
Depending on the hardware and OS, you should have no trouble running >25k users. There are POP and IMAP daemons available (some by the author, some not) and it is fast, secure and customizable
It natively supports Maildir and mbox delivery. Maildir stores messages as separate files in a directory instead of the mbox structure which makes it the ideal solution for NFS mounted mailboxes, which is crucial for that many users.
I can't say enough good things about qmail. You should at the very least evaluate it.
Don't make the mistake of trying to use Exchange! It is a real PITA!
"You can twist perceptions, reality won't budge." --Rush
I hear some people near me talking about Exchange [shudder... been through 2 migrations at companies before].
Knowing MS Exchange is a "Bad Thing", and I'd like to save the company money where possible, I decided to search the web for a collection of "horror stories and MS Exchange"... to my surprise I couldn't find ANYTHING!
Now I've seen articles here and there (InfoWorld, news.com etc.) about Exchange bugs, but I would have thought SOMEONE had collected URL's and posted them. Nothing. I'd have to do a lot of research to get this info, and given my workload it would be an unwise distraction.
The second thing I'd like to know, is how much does MS Exchange COST? I know the price varies, and larger companies get breaks if they "cozy" up to MS, but that doesn't help me much. Say a company has 50-150 employees... what does that translate into just for the software licensing?
Well, for us (University of Innsbruck, Austria), the UWash IMAP server works well. We do serve nearly 20.000 users using a SGI Challange dm workstation (a powerful Intel or Alpha dual processor would do the same job I suppose - those SGI machine is 4+ years old ...). There are no problems so far (8 month).
However, we did some minor modifications - we do use dmail from imap-utils for mail delivery (does integrate well with procmail, too), changed to MBX style mailbox formats and changed the default mailbox lookup from ~/mail to ~/. Everything else is quite standard. We do not have any problems at all. The users do access their mail using Netscape Messenger (IMAP), Outlook Express (IMAP), pine (local or IMAP) or IMP.
The only problem is that there are many processes (especially due to mail checkers). Normallly, this isn't a problem. But we do also offer a modified version of IMP for our users to access their mail worldwide. IMP is very slow if the server has many processes as it happens to create a process for every single mail when it reads a mailbox. We did modify IMP to overcome the major flaws, but everything else is pretty stand (or at least there are some publically available patches).
Outlook works fine as a mail client against non-MSMail/Exchange mail servers (i.e., POP3, IMAP).
Just beware the differences between Corporate (Exchange, but can use IMAP/POP as well) and Internet installs (IMAP/POP only)... check up at Woody's Office Watch for a synopsis on the differences at http://www.wopr.com/wow/
Listen to this advice, it's obviously born on the hard back of experience, just as much as me reiterating this same line: do not use exchange.
For example:
This is only a start, but I'm sure other people have many of their own reasons as well...
I remember our migration of a mere 750 (users) with extreme horror. We had to manually create each user.
You can create mailboxes in exchange via a config file with the mailbox import tool, although I figured it out by looking at files it created and not via any documentation. With exchange 5.5 I'm pretty sure you can create mailboxes with ldap (although this is far from documented last I looked).
As to solutions, I haven't used any open source email solutions with more than ~5000 users, for which sendmail and the UW pop3d and imapd worked well for the users that I had (many were very light on email). I'd be really neat to integrate an MTA and an IMAP server with ldap to support IMAP referrals and smart mail redirection. I know some of this is done as sendmail has LDAP patches and example rules for this, but I'm not so sure about IMAP side.
I currently work for a company with only a few hundred users at our location. We have been using Exchange servers for about a year and it has been a constant nightmare.
Scalability and stability of this solution has finally even convinced pro-Microsoft management that Exchange is not the product for large scale deployment.
We are now moving to an ISOCOR IMAP solution running on big Sun Solaris boxes.
* As is generally the case, my opinions do not reflect those of my employer.
Rule number 1 in Linux -- If it's worth doing, someone probably already has. Earthlink has published a really great paper on this subject. Definately worth the read http://www.earthlink.com/about/ papers/mailarch.html
In defense of exchange, at least where I work the exchange server can be accesses via BOTH outlook and IMAP4. Have you tried it?
Andy
We have a cluster of Exchange servers for our ~4000 workers here, and even Microsoft can't get it to work properly. There's a crew of Microsoft folks here, almost all the time, and our staff of full time admins. Just for Exchange. This is an "upgrade" from the single sendmail/imap server that handled the load of the entire company with very little problems. The main problem with Exchange is it's terrible performace, and the way it makes Outlook (or Outrage as we like to call it) hang while doing such complex operations as reading the next message in your inbox. On the bad days, it takes me as much as 5 minutes to switch messages. On the best days it's just irratic. This doesn't even bring up the problems with the feature set. For example, you can't check your mail from 'nix any more, sorry. Frankly, I can't see reason one to use Exchange. Even if you have to run on NT, you do have other options.
-- "it's not enough to be a great programmer; you have to find a great problem" - Charles Simonyi
I have had nothing but problems with the ArcservIT Exchange Agent
I'm in the same boat here. I get the agent to run for a couple of days, then it simply stops dead.
what does tech support at CAI say "oh yeah we broke that, ya gotta do this....
So far when I contact CAI, all I get is the same response emailed back to me.
Java gaming nut - http://www.retep.org/ or for the rail http://uktra.in/
I recently migrated 450 users from One exchange server to another. Because the two servers were not in the same domain, I couldn't simply move the users from one to the other.
The solution? I created a list of the users full names and user ID's using emacs, and a short Perl script that converted this into a CSV file that Exchange admin could import. Then on the Exchange server I'd import this, which then created not just the Mailboxes, but the NT Users aswell.
If it wasn't for the number of Linux boxes I have here, I don't know what we would do.
Java gaming nut - http://www.retep.org/ or for the rail http://uktra.in/
Intermail from Software.Com has POP3, IMAP, and
is designed for the 5000+ client market.
It is pretty easy to configure, and if it is like
their other product, Post.Office, a Unix version is available.
I guess I have never run a system with that many users, but wouldn't an appropriately beefy
set of boxen with a clever DNS MX scheme, running
Sendmail, and the freely available POP3 and IMAP4 daemons of your choice do the trick?
Anyway, that should be Cyrus, or more specifically, the Cyrus imapd.
Check out Cyrus, from Carnegie Mellon, which is gratis (but not free).
Or maybe you'd like to spend some money. Then there are lots of companies, like Mirapoint, who I work for.
This must be recent, because I recall reading about the attempted switch to NT long ago, and when I checked Netcraft at that time, it revealed Solaris. My guess is that Microsoft found it embarrassing to leave such a high-profile project in the hands of a competitor, so moved it to FreeBSD, which they probably consider less of a threat than Sun.
It could also be that the web-server front end is FreeBSD, while all the backend mail-handling routines still run on Solaris.
That, or Netcraft's detection routines aren't entirely accurate.
I hope this doesn't tick anyone off. I really do believe that it is a solution to consider.
-jwb
I say FreeBSD because I know that it's reputation of stability and fast networking are true from experience, but I guess Linux or another open source operating system (OS OS, heh) would do the job fine. I've seen a system at a commercial ISP running with about 15K users on FreeBSD & Qmail, which is why I'm recommending it. They switched from sendmail when it started becoming too slow because of massive mailboxes; every time a user with a 50Mb mailbox logged on the mail server would chug until it had gone through the whole file. If someone gets sent one 50Mb attachment, that means that the pop3d gone through 50Mb of data to say "1 New Message" - it's not an issue with the maildir format.
I guess you could achieve the same effect with a clustering solution, but I think that's probably unnecessary.
You might also want to check out postfix.
URLS:
- http://www.qmail.org/ - Qmail
- http://www.postfix.org/ - Postfix (by Wietse Venema)
- http://www.freebsd.org/
Good luck!--
Everything I know in life I learnt from
$1M for a 120K user SIMS, hahaha, mulitply that by 6 or 7 and you're talking...
sims looked/looks pretty nice though, if you can afford it!
John Soward...University of Kentucky
I designed and currently maintain a FreeBSD system with ~40,000 users, with probably 20-30,000 "active". I only support POP3 access presently, but I'm looking into IMAP. I'm using 2 PentiumII 300s running FreeBSD. One acts as the SMTP deliver-er for all off-site activity, the other the primary POP3 server. Both have 512M of RAM. The POP3 server has 14 4.5G cheetah drives, 2 internal for the OS, etc. and the other 12 inside CMD ultra-daytona external cacheing SCSI-SCSI RAID arrays (RAID-5), each on their own SCSI channel (2 3940UWs in each machine).
I use the qpopper with a *lot* of local modifications for security and performance. A custom perl+mysql system manages the userids locally and talks to a campus-wide "meta-directory" which allows people to manage the users from their Winblows machines...User management is probably a bigger problem than performance.
IO will be your biggest concern, followed closely by getpw* calls, network bandwith, then RAM and/or CPU. There are lots of other issues such as expiring mail, preventing/detecting mailbox corruption.
Cyrus IMAPd will solve a lot of problems with IO bandwidth, quotas, expiring mail, etc...but it will require more RAM to sustain more simultaneous connections, and more disk space as users can/will/should leave more mail on the server. I have not tested Cyrus in a large scale environment...yet...
Sendmail works well, other mailers such a qmail, etc. may work as well, many claim to be more efficient, but a properly configured sendmail environment is hard to beat...but any reasonable mailer should be adequate, the actual MTA load shouldn't be that great, no delivering to the mailbox, that's another story.
Feel free to contact me directly if you desire any more details or statistics.
John Soward...University of Kentucky
There are also some other issues to think about when working with Exchange. One is the way users are going to receive their mail. You can use any generic POP3 client with an Exchange server, which makes life easier if you're worried about Outlook and the VB holes in the Office API. If your users are going to be leaving mail (which can be limited) on the server, then you have an issue with provisioning the hard disk on the machine. The size of the store file is ridiculous. Plan for double the HD space you think you'll need.
Exchange also has an HTTP back door. I don't recommend using it, as it depletes the resources on the machine at amazing rates. IIS is another poorly designed system.
Other issues to be addressed if you are even considering Exchange are the stupid things users like to do. If your users are going to be e-mail huge documents to each other instead of putting things on file servers, then you're going to run into problems. Administrative users tend to like to mail each other Word docs that are upwards of 10MB, with pretty pictures, letterhead, and stuff like that. Then you have a network provisioning issue, because you really don't want to be sending huge messages over e-mail anyway. That's what file servers are for.
The biggest pain I had with Exchange (5.0 - dangerous upgrades aren't my thing) is that you can't do anything to filter on the smtp port. People can spin spam through your server and it'll just smile and nod. There was a rumour that this was going to be mended in 5.5, but it's Microsoft.
Microsoft basically is living in a dreamworld with this server. It's not SA-friendly at all. It crashes, it's slow, and it doesn't create managable logs. Log files are anywhere from 5 to 25 MB, and are written over continuously. So if you're going looking for a spammer, you better be aware of the actions within an hour of anything happening, otherwise you'll never get to see it in the logs. And to read the logs, you'll have to turn all the services off for Exchange server, or it will crash. (I crashed an Exchange server reading a file of SMTP conversations once. What a pain)
In all, Exchange is poorly designed for a mission-critical application. It's convenient in that if you have only a few users, you can run all the services and the server will be moderately content, and extra drive space can be devoted to file sharing. (But don't make it a P/BDC. Too much work there.) You can add users from flat file, there are tools you can buy for that.
Unfortunately, everything about running Exchange server is expensive, especially if you're going to run Exchange client. (FYI - Exchange client doesn't use the Office API, so is not affected by viruses like Melissa) You need a big machine, big disks, and lots of patience. Oh, and Pepto. Don't forget to order a case of Pepto.
--Mandi Walls
walls@nospam.juniata.edu
Perhaps they tried an early version of win 2000 or some different version of NT. When M$ switched their website to w2k, netcraft identified it as solaris. But it was fixed shortly thereafter.
Don't know about imapd then, maybe that Cyrus product.
# amo, ergo sum
IMAP seems to scale like complete crap until you learn one thing... convert everyone's mailbox to MBX. MBX is still basically a flat file but with references inside to how big the current message is. It makes it much easier on clients in large mailboxes and everything supports it. Where I work we were about to abandon IMAP because it was so slow with our typical mail box (~42megs) until switching over to MBX. Now it just flies.
/var/spool/mail/$user /var/spool/mail/$user.unix"; /var/spool/mail/$user.unix mbx /home/$user/INBOX"; /home/$user/INBOX";
F /...
Here a quick dirty perl script to quickly move users over (one @ a time) that will work with all IMAP clients that support mbx... mbxcvt is braindead in that it requires complete paths.
#!/usr/bin/perl
$user = @ARGV[0];
$mbx = "/usr/bin/mbxcvt";
$command = "mv
system $command;
$command = "$mbx
system $command;
$command = "chown $user.mail
system $command;
---
Openstep/NeXTSTEP/Solaris/FreeBSD/Linux/ultrix/OS
--- I do not moderate.
For the most part, if you're buying x500s, you're probably looking at Ax000 disk arrays, too.
I forget what 8 was for.
Not to mention the bug/hotfix routine and a
proprietary database scheme which is tough to
transport.
One bug I ran into recently dealt with the
Internet Mail Connector. If you set global
limits on the size of user mailboxes, IMC
will bomb when it hits that limit - Keep in
mind it handles internet mail for all users
on that server. Yes, there is a fix. I
have not tried it yet, as I have to go through
the phone call screen at MS first.
Miles Lott
On a free email system with >>25000, we've run into several bottlenecks. The first was on a 32MB P90, in which the RAM size was the bottleneck. We upgraded that to 128MB. Then sendmail was the bottleneck, so we got a K6 225, again with 128MB RAM. Then qpopper wouldn't handle large mailboxes, so we started going to qmail. Then 16-bit UIDs was the bottleneck, so we finished going to qmail. Then we didn't have big enough disks, so we got another server (k6-300,256MB; now single p][450, dual when we figure out why nfs doesn't like dual) and put several multi-gig UW disks in and did software raid-5, and delivered email over nfs. Then network (10BaseT ethernet) was the bottleneck so we put the nfs traffic on a separate 100BaseT network. Then CPU was a problem because of the high traffic on the raid-5. Oh, and fsck-ing takes forever. Qmail's queue is on ide, and showing signs of bottleneck so we'll have to deal with that soon too.
I run a large (>>25000) free email service. We started out with qpopper, but quickly had to switch because qpopper does (or at least did at the time) very poorly with large mailboxes. If the mailbox was very large, qpopper would crash leaving the mailbox twice as big as it was before.
We were using sendmail at the time, so we started using qmail as the local delivery agent. And pop agent of course. Eventually we switched entirely to qmail.
One thing to watch out for regardless which solution you use is that (last time I looked) linux (or is it ext2?) is limited to 16-bit uids. There's ways to get around that; I just wish we'd considered it when we started.
This has probably said before but - check out Qmail, it's use of binary hashed files and Maildir format makes its a formidable solution for HUGE mail sites. It i also VERY flexible, much better then Sendmail. Wonderful documantation.
;-)
because of the greatness of Maildir format, you an store mailboxes on NFS volumes which is a great thing for HUGE mail sites (out of disk space? get a (good but) cheap NFS file server and slap it in!
Oh, did I mentioned a hefty $$ reward for finding security holes in it that went unclaimed?
I'm REALLY pleaed with it, and it currently drives a site of 300,00 users (a webmail application) and just works.
Gilad.
I used to work for USA.NET - the second largest email provider in the world (behind Hotmail).
:) So those are custom too.
:)
We used Ultra Sparc 2's for all the web servers, Enterprise 4000's for Oracle, and EMC storage solutions.
The software is custom written. All C. The incoming mail is put into Oracle and on the EMC file system, and the outbound is done with qmail.
Convential pop3/imapd daemons wouldnt work for high load
Maybe a bunch of us should start an open-source solution for email.
Especially if it's in a corporate environment. And do NOT try to get 25000 email users on one box with MSEXCH.
This has been said many times already.. but it
deserves extra emphasis, I was involved with a relatively small company (400-500) poeple, the plan was to migrate the mail system from a solaris box to NT and Exchange, The onsite NT engineer (It was a computer company.) was initially tasked with
after 6 months, he still couldn't get it to work
properly, so we got in a contractor who had been involved with Exchange installations before, he
then helped to set everything up, from 1 solaris box we went to 5 NT boxes, everything seemed to be working, they were shipped to the separate locations and the system and turned on, within a day they started failing over.. we could not diagnose the problem and so put the solaris box
back into service, where it has remained since.
Solaris will do this, but you will probably need to run it on a _big_ box, like a Sun Ex500 class machine with about 8 or more processors.
Nonsense. DePaul University runs 25,000 shell/email accounts on a 5-year-old SparcServer 1000 with only 4 CPUs. IMAPD and SENDMAIL are the only pieces of software used. Works great.
Look into postfix, it actually does a very good job of utilizing MP systems. the trick is to have many processes loosely managed which communicate with eachother (rather similar to a multithreaded design, all in where you share the ram, really).
EarthLink uses Unix systems and NetApp filers for mail service. They have a lot of customers.
They've written up a a paper on their mail system architechture: http://www.earthlink.com/about/ papers/mailarch.html. It's definitely worth a read.
... for such dubious reasons as: fallacy #1: "everyone else seems to be doing it", or fallacy #2: "they're a $$100+billion company so it must be good", make darn SURE the decision whizzes read and understand the license agreement, for your own protection, to wit: (paraphrasing) they'll take your money for all the licenses but if the SOFTWARE PRODUCT fails to perform as expected ('substantially' is the weasle word these days) or if there is loss of critical business data, employee productivity, etc, pfft, too bad! Make sure they understand that the SOFTWARE PRODUCT is not under your complete control, you are just handed a binary and patches from Microsoft® and if it contains bugs, there's nothing you can do except live w/ the inconveniences and wait for fixes that may or may not appear. For goodness sake, protect yourself! Don't let them place the 'onus' on you! Office software politics 101: if I make the decision, I'm responsible. If you make the decision, I'm not responsible.
Chuck
try { do() || do_not(); } catch (JediException err) { yoda(err); }
I consider that even sendmail may not fit your requirements. Anyway you have to plan very carefully the topology of your servers to avoid serious hangups. We have a quite active network with 5000+ accounts and even some servers with a few hundred accounts may give some headaches. Make some expectations on how dynamic will be your network to avoid these situations.
Besides don't focus on IMAP _OR_ POP3. Sometimes it is better to have both. There were some situations when we realize that it would be much beneficial to have ppp accounts on POP2 rather than IMAP. Meanwhile the LANs are mostly based on IMAP.
Anyway don't ever dream on Windows stuff. I didn't have the bad luck of dealing with E-Mail on Windows but I had almost my job in ashes due to managing accounts on it. It might look easy with a few tens or even hundreds. But when you have 2000 people on a wide profile of activities and professions, Windows administration is HELL! Thanks God He had already shown me what is there...
At NTNU (a university in Norway) we used to serve mail and homedirs (NFS) for approx. 20000 users with one HP (not sure which model) server, and one Sun Enterprise 2. This actually worked :), although a bit slow at times, and with some crashes.
:) - that's overkill now - but probably not in a year or so:)
We now have 2 Sun E450 servers (2 CPUs each), and are planning to get one more (4 CPUs)
Solaris is a good choice for something like this, Sun makes big, nice servers, and the OS scales well.
AFAIK isn't Exchange limited to a total of 16gb of mail messages? Isn't that only like 670k/mailbox for 25000 users? =)
-- sudo.ca
WRT to ease of use, you're obviously going to run up against some disagreement. I find FreeBSD easier to configure, and to maintain.
:^)). One must ask your self, do I *really* need what IMAP offers?
I think hands down, maintence is easier on FreeBSD; this is very hard to deny as it's so much easier to automate things and do things remotely on any Unix than it is on NT; setting it up perhaps takes a little getting used to.
WRT to the MTA, there's also Postfix, which aims to be Sendmail compatible, while also being oodles faster.
WRT to IMAP, there are a few other IMAP servers than the UW stuff (which was the one with the security problems). But IMAP tends to be more demanding for the server (and the client coder
The revolution will be mocked
"Rumour has it" that Microsoft attempted to switch
Hotmail to NT, but failed. It is still running the
database end on Solaris and the web end on FreeBSD.
When you have that many users you have to have a nice structure for the usernames, which isn't the /etc/passwd file. And, you need a mailbox format that isn't linear, like the normal mbox. The rest of the problems can usually be solved with hardware (think about using a raid).
I know of three potential semi-free solutions.
Carnegie Mellon Cyrus (go to the FTP site and download the latest version. Don't rely on the way out of date web page to link to it.) IMAP server.
University of Washington's imapd. This seems to be under more active development, and supports a nice range of features, mailbox formats, and security mechanisms. However, it uses the passwd file (although you might be able to get around this using PAM) and it doesn't natively support quotas. (although you can do this at the OS level.
Darthmouth's Blitzmail Server: This has been ported to linux, and is *wonderfully* scalable across multiple machines. It inlcudes its own directory services too. The only problem is that it doesn't support Imap (although some work has started on that front), and the only database it supports as a backend is oracle. I would love it if someone hacked it to use mysql of postgresql with IMAP support, but that's a tall order. The client is also under-featured.
All of these have their drawbacks though. You might wish to go with a commercial IMAP/POP server on linux. There are a few good ones that exist. You definitely don't want to go with exchange. A lot of people go that route because they are forced to. My experience with exchange 5.5 was so bad that I would not recommend it to anyone.
-OT
Umm. I understand they do use Solaris on their SMTP servers. I did a nslookup for their MXs and then tried to use Queso and telnet to different ports but couldn't find a way to prove that has Solaris.
Alejo.
EXCHANGE BACKBONE AND DESKTOP
Were you to choose an Exchange backbone you would probably need to plan for about 1,000 people per server. That would make it about 25 servers.
They would cost about $30,000 each for hardware, software and installation costs.
You may also need bridgehead servers to connect remote domains if you haven't all your people at one location. You'd certainly need one Unix box running SendMail to handle spam, spoof and all the other nasties coming at you from the net.
Your total cost will be running to around $800M and you will need support staff of about nine people to run directory, administration, servers and help desk...that's about $1MM a year.
OUTLOOK DESKTOP and LINUX BACKBONE
Using IMAP4 you can take a leaf from Unilever and run Linux and SendMail as the backbone only using Outlook for the desktop. Unilever in fact uses Solaris...but there.
Installation costs would be somewhat less, say for eighteen servers (hardware, one day installation and loading clients - say $270,000) and support costs maybe for seven staff a year, you're talking $700,000.
FULL SMTP SYSTEM:
You're installation costs would probably be the same for the backbone...but you'd save big time for the end-user desktops.
CAVEATS:
Whatever system you chose, you need good backup/recovery and need to account for that and you need to be able to manage recovery at the desktop too, so that all your expensive people don't spend all their time helping each other recover from I've-shot-myself-in-the-foot crashes.
You need to set up a system of measuring message delivery and warning of problems (SNMP-like) and publish these goodies to your management and user groups.
ADVANTAGES OF OPEN SOURCE:
It costs less...that's obvious...it's more stable and, if you get a support contract, then your risks are much minimized...
Twenty-five thousand is not small in eMail terms...you are beginning to get into stretching some of the limits...but you haven't given too much detail...
...are you intending to use X.400 for example..?
...are you intending to use LDAP for directory?
...where are your main groups located?
...what kind of calendaring will you be using?
Good Luck...I would go with Open Source, but whatever you choose, it will be an adventure!
Buzz Lightyear
IOW, It was the users fault that he had those problems. That's typical of Microsoft. When all goes well, they are plenty ready to take credit. However, when there are problems, you are able to automatically create large number of addresses, The servers don't stay up, it doesn't scale, and so on, then it becomes the users fault. The user doesn't have enough training, no experience, they aren't certified, or some or "reason".
It would be interesting to compile a repository of problems that Microsoft "alledges" are the users fault. I know that a few examples have been posted on /. in the past
That's the benefit of Open Source. If we give Microsoft the benefit of the doubt and say that it is the users fault that "bugs" and missing features exist, how does that help the user? Does that gets the e-mail accounts created? Does it get the bugs fixed? Does it keep the e-mail servers running? Open source means that even if it is the users fault, they have the resources to correct the problem. The can fix it in-house, the can hire a consultant, they can buy a support contract from one of the many open source support companies, they can even check the resources available on the internet, if support from the company that "developed" the product isn't able to help them. With Microsoft what have you got? If it's more complicated then "rebuilding" or replacing "corrupt" DLL's, I'm afraid that the "support" you get from Microsoft is that "nasty letter to your boss" ie, it's your own fault
It is this very reason why I wouldn't use a proprietory solution over an open-source solution. Especially with 25,000 users. You have to ask the question "Who is going to support me?" You know quite well that it isn't Microsoft. And an environment that large is going to require support no matter how rock-solid the platform is.
-Brent.I assume you mean Exchange server, outlook is merely the client. :-)
From my experience, the most problems with badly performing client connectivity to exchange servers can be found in the priority of the protocol bindings on the client. More often than not, I found clients using some protocol that was not used on the server, and having that protocol as #1 in the bindings. That's where problems start.
But back to the original question. Should you use exchange for a large amount of users connection with POP3? Simple answer would be No. Exchange is a powerfull tool, but if you only use it for the e-mail functionality you are wasting resources and money. When you start thinking about sharing calenders, assigning tasks (to-do's) to other users, workflow management and stuff like that, then I'd say to go for exchange.
In my experience, an exchange server is one of MickeySofts least trouble-causing applications. In the 2 1/2 years that I managed a WAN of about 20 locations with some 25 exchange servers in them, I have done migrations, splits, moves etc. with not much of a problem. Only once I had to pull out the backup tape and restore the entire server. Which was due to a corrupt HDD, corrupting the database in such a way that it wasn't repairable anymore.
----------
'We have no choice in what we are. Yet what are we,
but the sum of our choices.' --Rob Grant
----------
'We have no choice in what we are. Yet what are we,
but the sum of our choices.' --Rob Grant
Sendmail isn't the part that doesn't scale, it's POP3 when dealing with Sendmail files.
The Unix mailbox format certainly doesn't lend itself well to POP/IMAP usage, particularly when messages may be deleted singly. However, this isn't really sendmail's fault; it's following historical Unix practice.
The obvious solution here is to select a different mail spool format, with better POP/IMAP support. All you need to do is select (or create) a new Mail Delivery Agent (MDA) for sendmail to replace "mail.local" or "procmail" to deliver to the other format. Apart from that, sendmail should work unchanged, unaware of the differences in the mail spool.
Deven
"Simple things should be simple, and complex things should be possible." - Alan Kay
ITYM "modifying an existing MDA to utilize a database". The MTA's job is to recieve mail, and then hand it off to the MDA (Mail Delivery Agent), e.g. /bin/mail or procmail. Sendmail doesn't care whether your mailboxes are in mbox format or DB format or Martian Swahili.
HTH,
-
-EtJ
--
The dog ate my
IMAP by default leaves your mail on the server, which is good if you move around alot. POP by default "pulls" your mail to the local machine. I prefer IMAP, but it can overload your machine if you do not have good users that clean out their mailboxes often. I recommend quotas and a cron job to emails users when their mailbox get too full.
I usually setup more than one SMTP server for redundancy, (usually the Web server and the firewall) and put multiple MX entries in my zone files. Each department owns a file server that also runs POP/IMAPd and accesses the central mailboxes (/var/mail) by NFS. I don't use NFS for any other purpose because I find it too flaky. Once the mail is accessed, it gets pulled to the departmental file server, and ploped in their home directory.
From that point, they can use any program they want to read/post mail. The advantages to this strategy are that email can always get in and out, and nothing gets lost, even if a machine goes down. (Of course, if the departmental machine goes down, they cannot read their old email, and if the mailbox server goes down, they cannot read their new email.)
Overall, It's better than no email at all.
As for myself, after watching a team at my former company struggle for 5 days trying to rebuild a crashed MSExchange server, I no longer look on it as a reliable, economical solution.
There is no trap so deadly as the trap you set for yourself
There is no trap so deadly as the trap you set for yourself
-Raymond Chandler, The Long Goodbye
grep MTV *
which was doomed to fail since the shell needs to expand the "*" before it can even start call grep, you should have tried
find . -type f -exec grep -l MTV {} \; > /tmp/filelist
If you wanted to delete the files then
find . -type f -exec grep -q MTV {} \; -exec rm -f {} \; -print
in general, I've found that relying on shell wildcards for very large file lists is not the best use of computing resources. "find" is a very useful and powerful tool.
There is no trap so deadly as the trap you set for yourself
There is no trap so deadly as the trap you set for yourself
-Raymond Chandler, The Long Goodbye
There are wizards and things, and there are ways to have Exchange create mail users automagically from an NT Server userID database, but it definitely is a pain. I believe there are third-party products which do simplify the process.
lake effect weblog
{Network engineer in Chicago--looking for work!}
I believe Hotmail use Qmail, which is open-source. When MS bought them out they tried to switch to NT/Exchange, but couldn't get it to work, so they're back on Solaris/Qmail now. I believe they support a large userbase which not only have attachements, but a whole lotta spam, to...
In the land of the blind, the one-eyed man is kinky.
Er, you could go the commercial/OSS route with a Sendmail, Inc sort of thing. I hear great gnashing of teeth from folks with Software.COM, plus you get zero source. Exchange!? Er, not. There was a usenix paper a while ago on Earthlink using sendmail/qpopper to handle 2-3 million. I'm sure that Sendmail can handle a piddling 25k users. IMAP is maturing rapidly and there are commercially supported Source available solutions for that. Most of it depends not on users, but on volume of mail.
I'm using Cyrus's IMAP server and OpenLDAP to hold all the user information and it's worked like a charm. I found an LDAP patch for Cyrus' pwcheck program and this system has worked perfectly. I wrote a few php web pages for managing users in the ldap directory and it has been a breeze to administer. Also since you have an LDAP directory it makes it simple to go back in later and make a nice addressbook for people to use.
Unlike with qpopper or UW pop3, I have experienced zero problems with corruption with large file attachments.
My company has enough trouble trying to have it work with under 50 users.
Exactly. I once worked in a company that used exchange. (And sold lots of microsoft solutions). One day we all got a letter from the mail administrator telling us to change our outlook setup from polling every minute to every 10 minutes instead. Because the mail server machine (with its 500M, several cpu's...) couldn't handle the load generated by every user polling every minute. How any users? 50. The big NT+exchange machine had trouble with one mail poll per second.
I believe it is possible to set up exchange a lot better than that, but apparently not by default.
Consider how a mail server will be used. In the beginning email is new, like the phone once was. Cool, but not something the company depends on. After a while all internal communication goes through email, and a lot of external communication too. Downtime on the phone system is simply not tolerated, the same will become true for your email system after a while. Select something with low regular maintenance overhead. It doesn't matter how hard or easy it is to setup the first time, how many months between hiccups matters a lot with thousands of users.
If you want to do this, check out WING:
http://users.ox.ac.uk/~mbeattie/wing.html
It's a free, Open Source IMAP and POP clustering solution using Exim and, er, other stuff. Written by Malcolm Beattie, Perl guru and all round clever person. It runs Oxford University's 20,000 user system very, very well (no downtime in a year IIRC). It's what you want.
Gerv
Horms is the mail dude for ZipWorld, one of the
e rs/horms/
larger Asutralian ISPs. They run Linux internally
and he presented a paper at CALU on building a
large and scalable mail system.
See:
http://www.linux.org.au/projects/calu/cdrom/pap
for the conference paper.
Cameron Simpson, DoD#743 cs@cskk.id.au http://www.cskk.ezoshosting.com/cs/
NETSCAPE A "GA" LINUX BUILD OF LDAP and MESSAGING. We're looking for production customers.
Most of our clients are easily running 15,000+ IMAP, heavy attachment users on Sun ES 3000s.
We bought the company who was doing the most advanced IMAP work (sorry, forgot their name.
Check out our scalability. Sorry Exchange customers have come crawling back. (Disney, if I recall correctly)
Plus, the customizable webmail is not a CGI front end, but built into the MTA.
Finally, we have a hash indexed mail store that looks just like UN*X. You can hack it and then "rehash" to regain top performace.
And then there's the question if you're going to write any applications with the profile (UN/PW, jobtitle, etc) info from those users. LDAP is the best solution, ask anyone who has a clue. Netscape's LDAP can handle 50 million entries at 5000 queries/sec on one 4 cpu Xeon box. That's pretty hefty. Our Messenger comes with LDAP out of the box.
I think this is a cop-out. Exchange has proven itself to be quite stable, and is one hell of a solution for many, many people. Exchange will handle 25k users, the only limitation is the restore time to restore IMAP data stored on the server. If you use POP3, that problem goes away as the files are retrieved from the server and saved on the user's local machine.
What you should consider is using sendmail for smtp services, with and Exchange frontend. This lets you do all the cool stuff, but keeps it running nice and fast. Email me at service_account@yahoo.com if you want more info.
Jay
Fire your Exchange Admin... wait! Is he the janitor, the LAN admin, and the mailroom clerk too? Try not to be so quick until you have seen some big sites use it.
Maybe this won't be too much help but, the last thing on earth you want to use for this is MicroSoft Exchange. we have 4 exchange servers where I work on World Class hardware, serving about 700 people, and these things crash on a semi-daily basis. Not to mention that they are _SLOW_ as cold molasses. And I've heard other reliable accounts as to how exchange cannot scale. Last I heard, Microsoft was using a huge exchange cluster (700+ machines, I think) and each machine averaged 1300 messages a day at full bore. Man, that's pathetic. For less cost you could hire someone to type all your e-mails up and fedex them.
Talk to the people at Sendmail, Inc. If you are prepared to shell out the dough for exchange (any idea what 25,000 licenses are going to cost?) you might as well pay the sendmail folks to set the system up for you. For enough $ they may even write in custom changes for you.
What would just the client licenses for Exchange cost for 25000 users?
How much would all of the copies of NT Server and Exchange you'd need (you aren't going to run that many people on one box, that is for sure) to support that many users with Exchange cost?
How much is all of the technical support you'll need from Microsoft going to cost?
How many MSCE's will you need to hire to administer all of those NT Servers?
Etc, etc.
My guess is you are talking costs in excess of a million dollars more to implement NT/Exchange than it would cost to implement an open sourced solution.
I develop the Linux port of Netscape Messaging Server. Most of this is not open source, but key parts of the system are open or can be easily customised (web mail screens, account management, spam filters, etc). We also give you a wide choice of OSes. The Netscape Messaging servers scale into the millions and have great performance (IMHO). We even give you great tools to do your own benchmarking (I wrote some of these too :-).
Your request left out a lot of details about usage profile (corporate or ISP). For an ISP profile (at most 10% active at once), you can probably run 25K users on a single Linux box (Dual 400Mhz PII +1Gb, RAID controller) plus a directory server (smaller Linux box). For availability issues, you probably want to distribute this over multiple boxes. If you enable SSL support, you will need more CPU.
The Linux version isn't shipping yet. Let me know if you are interested in the public beta.
Typically, the critical components are memory and disk subsystem. Disk latency really hurts performance. RAID controllers with memory back cache can help SMTP performance a lot. Remember the data is the important part; processors are disposable these days.
I have talked to customers who have millions of mailboxes on sendmail. They had to do a lot of customization to make this work (and I think they are still wrestling with it). They were also doing POP only. IMAP tends to require a much larger message store.
I wouldn't mix mail servers with other functions (shell, web). This tends to make performance variable and create administration headaches.
Test everything before you go public. Every site is different. Until you test it the way you will use it, you won't really know what to expect. Test you backup and restore proceedures to!
-Dan
Exchange is just a plain pain. The senate uses it for, what, maybe 1000 users (Senators and support)... When they were flooded with email last year, the servers locked up totaly. Many messages for days on end were lost. A cluster of Linux boxen would be a bit overkill, but if you wanted to do it, it would be a far better solution than ANYTHING Micro$oft.
If you need some white papers to convince your boss(es) go to: http://www.unix-vs-nt.org/
1.3L, 3 moving parts, 280 HP, no Turbos, wanna Race? RotaryNe
To me, qmail seems like the obvious choice. It's dynamic nature combined with it's speed make it it the best candidate for this job. It's speed when sending mail is simply superior to all others, and it's using a very good pop prog as well. The fact that you use maildirs, not mailboxes provides you with the stability you simply cannot live without.
:)
Also, if you make a setup where you handle all your email from one UID, then build some binary databases, you're at a speed noone else can come even close to.
Check it out at www.email.org, make sure you read the link about multiple email boxes on one UID.
Mail me at delta@xti.org if I can help out somehow. (won't be looking at this thread much, too big for me
Terje Elde
Poke through the Event Log. Note the messages from the IS startup. They plainly state that the Iformation Store is limited to 16 GB in Exchange 5.5. Then go to M$ Technet and read about how the limit no longer exists in Exchange 5.5. Notice how M$ makes no explicit mention of Standard or Enterprise Editions when touting this. We read the BS, believed the ads and PR, and installed Exchange 5.5. We were just starting out, and didn't know much, but M$ solutions are easy, right? A year and a half later, I've had Exchange out and out die for no reason 3 times. The Information Store (800-900 MB)gets corrupted on a daily basis. Setting size limits on a mailbox results in a message stating that the mailbox has been deleted once it is full for God's sake!
--
"Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
I have no doubt that you could eventually build a reliable system from open source tools, but you should give serious consideration to a carrier class software like the stuff from software.com. I beleive their licensing starts in the 25,000 mailbox range and goes up, which should be a hint that they are ready to meet your needs.
Consider this 25,000 mailboxes * 10 MB/mailbox =250 GB. Is this something you are ready to trust to your linux filesystem without a second thought.
If you are going to roll your own you should dig around on netapp.com. They have a document which describes how earthlink built thier own e-mail system on top of a net-app filer.
Of course outlook doesn't scale, it is an e-mail client, not an e-mail server.
Perhaps you meant exchange server. If you did, I would have to agree with you. Exchange server would be wrong for this sort of arangement.
...a very simple one. I am a cablemodem user who is trying to setup something for people behind our firewall as a kind of community effort. I have a small Pentium 150 box with a couple hundred meg of RAM and a 2 Gig HD for about 100 people. I am running the Cyrus/Sendmail/Procmail configuration for the server end. For the clients, since it is IMAP/POP combination which is fully compatible with Netscape's mail client, or any IMAP compliant client for that matter. I then use OpenLDAP for the the directory server. There is a great article at Netscape on how to customize the address book to use a specific server (involves editing the .js files, and I don't have the link in front of me) so that the Linux box is a used as the default LDAP server. Finally, and this is the cool part, I use the IMP/HORDE combination for a web access to the email. If you wanted the cool (yes, I am making a pro comment about Exchange) web interface to IMAP, I would recommend IMP/HORDE which can be found here for the web interface to the mailboxes. It is still in beta, but I am using it and it is very speedy. Only thing is that you will need Apache/PHP with the IMAP module, but it is GNU software. Only problem I have encountered is that the HD is really cranking when I have a lot of users running on it (but I also run the web server, ftp server, and samba for those behind the firewall). Well, that is MY experience anyway. I would imagine if you were to beef up your hardware and get some more distro opinions (one guy I talked to said I should try *BSD since it would handle the load a little better), you would find a happy medium. Just some $.02 from a joe shmo off the streets....
Bryan R.
Bryan R.
The price of freedom is eternal vigilance, or $12.50 as seen on eBay.....
Wow, amazing responses. Thank you to everyone writing such thoughtful, varied and experienced replies and links. It's really helping me think about the issues.
Here's some more requirements that may help explain the situation better.
Requirements:
At the moment the most talked-about local solution has changed from Exchange to a Novell Groupwise system of at least 20 Novell boxes, with a big sendmail system as the main gateway.
I would really like to work out a complete open source system that can do the same job cheaper and better.
--
Here at RIT we use a software package called PMDF from Innosoft. We have a cluster of Digital Unix boxes that support close to the numbers you are talking about. Innosoft supports Solaris, DU, and VMS. It is a great mailer. The support is excellent and they are fast to deliver fixes. It isn't free but it will definately do what you want.
My company has enough trouble trying to have it work with under 50 users.
My current employer has more than 25,000 users on Exchange. It works fine except that exchange crashes every other day! ;-)
My last employer had a similiar load on exchange servers, and it seemed more reliable there. Then again, my last employer had access to more knowledge about exchange and NT than any other company there is. ( Guess who?)
I'd concur with the other posts that suggested something on a proper enterprise scalable real OS.
--iceaxe
WALSTIB!
Although you could go with Exchange - I would not suggest it. It is not terribly well supported even by Microsoft - and it is horrendously expensive.
... they just released a new version today, and it was quite promising about two months ago, at least in theory ;>
... but I would have to say that UWImapd is really quite nice, clean and effecient.
My overall suggestion would have to be qmail. The most secure, fast, effecient, and worthy daemon to run your entire site... check out www.qmail.org. But if you want something else I have listed out some other possibilities.
Pop3:
I would have to say, from personal experience that qpopper still works the best, go to Eudora and grab a copy.
If you want to play around with a purely open sourced one go get the GNU-POP3d (the old IDS-POP3d)
Imap:
Some people think I'm nuts
SMTP:
Sendmail is free (the pro version has support and the money is minimal). Sendmail is the default mail daemon for smtp on almost any Unix.
I work for a LARGE bank with about twice as many email users as you require and we use Lotus Notes with pretty good results. I understand there will be a port to linux, if it doesn't already exist. Regardless of the platform, you will need multiple Notes servers that will replicate with one another. The last company I worked for also used Lotus Notes and ran the servers on OS/2. I am not a Notes admin, so I don't know what platform we use.
Notes supports encryption and has dial-up connectivity for mobile users. You can use IP, SPX, and I believe a few other protocols. The client side is thin enough to run even on fairly weak machines, 16 or 32 bit.
Good luck
i'd definitely recommend to consider notes/domino. it supports pretty much all current internet protocols. R5 will run under linux by year-end and many platforms today.
;) and includes out of the box scheduling and collaboration.
it definitely scales to the number of users that you need (even on one machine if its s/390
go ahead, download it from lotus (its free for 90 days) and check out for yourself, then go to http://www.notes.net/ and ask around (they have a forum) if you have questions.
hth
I strongly suggest you use qmail. It's FAST flexible (i.e. you can write your own authentication and delivery modules). It scales perfecly. Actualy just about ALL of the big freemail services run Qmail.
We are currently using it for about ~2500 email addresses with no problems at all on a PII-233 with the load going between 0.04-0.05.
Stefan.
Hardware:
Software:
This'll last you at least 4 years.
Exchange is absolutely unusable in your situation unless you have unlimited financial resources for 5 times as many machines, corresponding sysadmins, and here's the kicker: licenses.
You'd have to license both NT Server and Exchange server for 25,000 users [the nt server is a little different, but exchange def. the 25k ].
It was prohibitively expensive for a 50 person insurance office, so it would kill your operation. I setup a linux box for them a year ago, and haven't touched it but three times since, all b/c of extended power outages...
I've had my fights with Exchange, I'd go with *NIX solutions anytime, but corporate standards have been set prior to my arrival and these people really need the calendaring.
Let's just say I've managed to keep the box up and running for !! 30 !! consecutive days, and I'm now touching wood that it wont crash and burn under the burden of a mere 250 users.
Sun Tzu must have been running Linux...
- Hold out baits to entice the enemy. Feign disorder, and crush him. (Sun Tzu, The art of war)
Marriage is considered capital punishment for the theft of a goat in some third world countries...
The company I am working for has many more users than 25,000. We have many different Exchange Servers segmented across several company divisions. First hand experience, Exchange goes down all too frequently. The Exchange clients are horrible to support. They just love to keep re-trying the server for as long as 10-15min before erroring out. The licensing is astronomical, even with large enterprise discounts. Required add-ons to allow for Fax Gateways, LDAP, etc. are also very expensive. The only good thing is group scheduling via Outlook. I shudder to think how much we could be saving by running Sendmail on SUN Servers. TCO with Microsoft is a complete crock. The servers were overloaded today because someone emailed a global distribution list and a bunch of people used "reply to all". It brought 8 servers to their knees and will cause headaches for months to come as people keep doing it. Granted, this will stress any email server. Exchange and NT in general doesn't function under heavy load very well.
The University of MN hosts over 75,000 email accounts using sendmail running on Solaris. I'm not sure how big the Solaris box is though, but buying yourself $100,000 worth of Sun equipment and using sendmail is still going to be *WAY* cheaper than going with Exchange.
Qmail is fast and easy to set up, but lacks many options that are available with Sendmail. Sendmail also has built in support for the Realtime Blackhole to filter out spam. Sendmail is definitely the way to go. As for the machine you run it on, you can always get some Sun Ultra 2's, and cluster them together. Maybe an A5000 fibre channel file library too.
You could try running it on a 4 processor Xeon running Linux, but I don't know if x86 processors could handle the load. I seriously doubt it, especially in the morning when everyone is checking their mail. Go with a RAID setup to speed up disk access. And you may want to turn off logging for the mail, syslogd take up an awful lot of CPU when we turn on logging for mail. If I was on a very restricted budget, I would try Linux on a 4 processor box from VA Research with sendmail compiled with pentium optimizations, a really big RAID array, an optimized kernel, and lots of memory and no mail logging. Of course, I'd be ready to plug the old mail system back in very quickly just in case.
Need Free Juniper/NetScreen Support? JuniperForum
Sendmail's the answer for us. The only thing that hasn't scaled well is plaintext aliases files: we've got some 20K mail lists, and it's beginning to get somewhat messy, so we're having to go to the non-plaintext solution. But for all the rest, it's stock sendmail with various GUI backends for end-user ease-of-use (and security). Note that we don't have 25K users, but 17K isn't that far off, and we do a *lot* of e-mail.
To quote an email I just received:
/. is asking about large scale email handling. I was going to
>> someone on
>> mention that you guys do this and use FreeBSD. I thought I'd ask you or let
>> you post rather than getting your site slashdotted.
--me
> Large-scale in our terms means millions. Also, we use FreeBSD in some
> places, but most of our service runs on Solaris. And, we sell service,
> but not our software, so I don't think we'd suit this guy's needs.
>
>25,000 is (assuming typical usage patterns) within the capability of one
>powerful Linux or *BSD box. Use NT, and you'll need a cluster. =)
--he
AFAIK there is only one way to keep Exchange from relaying and that is to go back to version 4.
The downside (well, one of the downsides) is that while Exchange 4 doesn't relay, neither does it tell you that it doesn't relay. So spammers try to relay against your Exchange server, which fails and sends mail to the mail admin, but to the spammer it looks like it worked so they keep using it.
---
Put Hemos through English 101!
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
"[Vendor] experts in Redmond..."
Hmmmm....who could this be?
---
Put Hemos through English 101!
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
...except an anti-recommendation.
If you are going to setup 25,000 users, do not, repeat NOT, use Exchange. I remember our migration of a mere 750 with extreme horror. We had to manually create each user.
Of course I was simply a lowly programmer working under the direction of our totally incompetent network admin--maybe there was an easier way and she missed that topic in the training the week before.
What you really need is a requirements analysis. Exchange is a totally different thing than, say, Sendmail. Analyzing what you need will tell you which to go with. For instance, do you need public folders, scheduling, etc? If so, maybe use Exchange. Do you need configurability, speed and Internet email? Then you want not-Exchange.
---
Put Hemos through English 101!
Linux MAPI Server!
http://www.openone.com/software/MailOne/
(Exchange Migration HOWTO coming soon)
Hello...
/etc/{password,shadow} lookup. Good luck with the PHBs :)
my small team of five has set up 10 exchange servers
My team of ME Runs 10 servers alone.
and it took about three days.
I can set one up in about 30 minutes.
The busiest server handles about one e-mail related (SMTP or POP) transaction every second using sendmail and the default RH pop daemon. An edit here and an edit there and inetd can now handle 160 connections per 60 seconds. ( I sometimes peak at 55 to 65 pop connections per minute). Works here great! A quickie bash script on you can add as many users as you want with passwords in a few minutes.
Did I mention I do this alone? (and run the web/DNS/etc).
For large numbers of users you need a robust OS with a lot of disk space and ram. From my experience the limiting factors are disk I/O ( get fast SCSI in a raid) and RAM, not the processor(s). To make authintication faster use NIS (hashed) or a SQL batabase so lookups are sorted in some way first instead of a flat
Christopher McCrory
Lead Bithead, Netus Inc.
chrismcc@netus.com
admin@netus.com
"Linux: Because rebooting is for adding new hardware"
Christopher McCrory "The guy that keeps the servers running" chrismcc@gmail.com http://www.pricegrabber.com
We use the netscape messaging server here. This was a compromise between the exchange solution that some people wanted, with all of the coporate wightbehind it, and the sendmail/unix open source solution that other might prefer. So far we've had pretty good success with netscape messaging server using Netscape's enterprise LDAP server as a directory on the backend. There's a pretty gui to show the boss and a great Perl interface for LDAP for myself.
On our unix shell server users can connect to the messaging server via IMAP, and they can look each other up in the database as well from within pine. Obviously there's a nice interface to Netscape Communicator as well.
We also have a microsoft exchange mail server here. It's using a number of multiprocessor 300mhz PentiumII boxes, it cost us a fortune, and in my humble opinion doesn't provide the level of service we get from our messaging server.
If a simple sendmail/unix setup won't work for your managers or your environment, you might want to give this a try.
...for actually answering the author's question. I saw a ton of posts from people plugging software for every reason except from personal e xperience with the problem at hand. And it's comforting to know that sendmail can still do the job with higher numbers of users (as I suspected). It helps me sleep better at night. :^)
Yeah, that's an exceedingly fair point. I will admit that when I was running email at Netcom, IMAP was in no way a factor. These days...
Upwards of 30K shell users, four Sparc 2s, SunOS 4.x, sendmail. And a lot of email. Our only big issue was the number of files in /var/spool/mail, which we coped with by going to Network Appliance NACs.
We noted that directory lookups got worse in a distinct knee -- i.e., we had no problems for a long time and then we hit a magic number and things went all to hell. I do not know offhand how well linux or Solaris deals with directory lookups, but you could test easily enough.
The thing you didn't tell us was what the volume would be like; the number of users matters for the mail spool but the number of email messages matters for the CPU usage... I suspect that you won't need a very heavy box, though. Email is cheaper than you might think.
Oh. Run a DNS server on the mail hub, to avoid a lot of lengthy DNS queries on some other poor machine. Flush the cache daily.
Why not Notes? you will get that kind of scalablity and you can let you boss start you out on WinNT but after they see the light you can switch them to Linux. Domino isn't currently out for Linux, but supposively R5 will be out by the end of the year.
Here's a link to get you started
http://www.carumba.com/imap/
Have fun and ask if you have questions.
Compaq / Digital professional services deploy and manage virtually all of the largest Exchange deployments- once you get above a certain deployment size, the requirements for maintaining the system require extremely specialized and arcane knowledge that (apparently) only they have. MS actually has structured the MS Exchange support contracts so that above a certain deployment size, you pretty much have to use Digital, or you won't get any support from Microsoft.
So, unless you're willing to fork over the $$$$ for consultants from Digital to come and build the whole thing for you, I'd avoid Exchange like the plague for a project this size.
I think your best bet would be to split the load amongst several machines. You can use aliases at the main MX machine to split the mail between several servers, the users would only have to use different POP servers, according on where their mail is stored.
You can even go further, setting up a (large) central POP server, that mounts via NFS, SMB or your favorite system the mail spool - For the users, it would be 100% transparent, like a single, large machine. This would be a good candidate for setting up a Storage Area Network: A superfast file server, and the SMTP/POP servers accessing it remotely.
Finally, maybe you could cluster together several machines to achieve some redundancy and better integration than with the SAN scheme...
- The mbox format it uses is very nice for moving things around, but don't forget, if one acct gets bogged down, it gets blocked due to file locking problems. (if you use the default local mailer). Multiple mail files are nicer. May run out of inodes quicker if you aren't careful.
- Its ruleset bogs things down just a bit. Significant if you want to get more than 10k emails out in an hour. (I haven't really benchmarked it yet.) Power versus speed, configurability versus mail handled per sec.
- Sendmail is a big program. Qmail is smaller and more modular in the sense. The operating system's process scheduler can handle things better than sendmail's internals. I've had server loads of 10 with higher traffic on a dual 300mhz machine.
Why not to use qpopper- To download your mail, a second copy of your mail spool file gets made so that qpopper can sort out which mail you wish to keep. Effectively you get only 50% of the diskspace.
- All users must exist on the system. Bad system administrators (like at my old job) would make home dirs for all users, leaving ftp open at times. Virtual local users kick butt.
There is your reliable software. For network stuff, I would highly recomend FreeBSD, but Linux would do fine. I would recomend qmail as it is both a popper, mta and local mailer. It doesn't give the problems above.-
ping -f 255.255.255.255 # if only
HotPOP is a Linux based solution that handles this kind of traffic easily and cleanly. Its probably the most cost effective solution too. Send me an email if you want more details.
This is not the greatest sig in the world, this is just a tribute.
The Cyrus server from Carnegie Mellon does very very well at delivering POP3 and IMAP4--they designed it to be very very scalable. There's one server currently handling the entire campus right now (12000ish people? More? I forget), including an IMAP gatewayed set of usenet. And it still works quite well.
The machine they're using is big iron (a big Ultrasparc with raid and so on), but I don't expect you can handle this sort of load with anything but.
Also of note is that the Cyrus folks are working on ways to do good load-balancing with IMAP (for example, allowing clients to transparently find the right server for a given mailbox).
I doubt very much that any MS solution could scale this well without a lot of effort. Cyrus or some similar piece of IMAP4 software is probably what you should be aiming at.
Finally, if you're only planning on doing mail, and not shared mailboxes or usenet-via-imap sorts of things, and if you have the space not to need to migrate users from one server to another to balance, you should be able to do decent by-hand load balancing right now. (i.e. "your server is mail7.people-with-mail.org")
Exchange DEVOURS hardware. Many mailbox servers PLUS PDCs, BDCs, WINS boxes, mail connectors, SMS to manage the whole thing (if you buy Microsoft's recommendations), and IIS servers (for Web-based e-mail). And there's building a domain structure (if you don't already have one) that you'll at least in large part tear down when MS makes you go Active Directory. It's a hardware and network nightmare if you're geographically distributed because the RPC/COM model they implemented really works best if the machines are close to each other, which means that you may have to duplicate a bunch of the aforementioned devices and underutilize them.
Oh, and the whole thing's a Service Pack/HotFix nightmare. Oh, and did I mention the Service Pack/HotFix thing? SNMP memory leaks alone will eat your RAM in no time. Service Pack. HotFix. Now. Tomorrow. Forever.
P.S. Yes, we are stuck with Exchange. And yes, these are Microsoft recommendations. MS Consulting Services just finished bleeding us dry, thank you very much.
P.P.S. And don't get me started on the Outlook client. My God!
P.P.P.S. No, I didn't have any say in the matter.
It seems that a lot of the services that people are suggesting are vanilla smtp offerings. If thats all you need, I'd definately stay away from exchange. However, if you need groupware type things (shared calenders, address books, etc), you should take a look at IBM's domino server. Just be aware that you will need a pretty beefy cluster of NT machines to support it; but then again, you'd need just as many machines if you went with exchange. Domino running on top of NT4 with nothing else sucks up about 80mb-140mb of ram (thats with about 40 or 50 user accounts in the system). Mail files get rather bulky (they start off at about 3mb when empty). A port of domino to linux is apparently in the works but is not quite there yet.
Let me warn you though, Domino has its own little quirks you get to deal with, and it has several unintuitive aspects about it. On the other hand, there are also some shiney things about it (eg: webpage access to all services, complete integration of all common communication needs, and extensibility through notes designer).
We would like to use an exchange type system on our company network with scheduling/email/etc... is there anything out there like that?
Microsoft will tell you themselves that they still run HotMail on non-MS products. Seems they can't get a 64bit compile of NT Server to handle the the load without crashing. Up to the last thing I heard (about 3 weeks ago) they're still running sendmail on Solaris. You can probably get the stability you need running Linux and save the bucks you would spend on Solaris (Unless Sun is still giving away copies of it) or Windows.
D. Keith Higgs
CWRU. Kelvin Smith Library
My office has been taken over by iPod people.
Oh man, very nice:
http://www.sun.com/sims/performance/index.jhtml
http://www.sun.com/sims/performance/shiloh.jhtml
Probably a $1M box.
Oh well, back to reality. We have found CommuniGate to be a Very robust IMAP/POP3 server. Current message store is about 100MB on a 26GIG MylexRAID on RedHat 5.2. Needs rebooting about every month (not close to our Solaris boxes who run many, many months) because of some strange problem. Check out:
http://www.stalker.com/CommuniGatePro/
I think it even has cluster support these days that can scale up to 1,000,000 boxes...
Prior to CommuniGate we used NT and Netscape Messenger. This was a total nightmare. We regularly lost the IMAP folers and had to bring all mail local via POP3 and re-upload to fix. The server ran reliably enough, but there were terrible and frightful problems with the message store.
Since moving to Linux and Communigate 98% of all our mail server problems are over.
Laters,
Mij
I work at a medium sized ISP, and I have a qmail running on FreeBSD for about 6000 users.
:-), but they were queued up, and all successfully delivered.
I use mysql for storing and validating the usernames & passwords, and the mailboxes are on our main (raid) file server mounted by nfs, over a 100mb ethernet. It would be pretty straightforward to add more boxes to cope with higher load, I would probably just need some kind of round-robin dns to spread the load accross the boxes.
I wrote the mysql hacks myself, by rewriting the checkpassword program (for pop3) and the qmail-local (local delivery agent). This makes it possible to use a web interface to add/remove mail boxes, and to integrate it with our billing software. I've used perl scripts to do it, although it is showing to be a bottleneck, for any larger demands it would be best to do it in c.
The stability and reliability is excellent, for example after having a (long) network outage, we recieved 60,000 messages from our external backup mail servers, in a burst. It took a long time to deliver them (probably because of my perl scripts
Overall, I would strongly recommend qmail.
Over the last couple years there were some in depth threads on the server-linux email list discussing how ISP's were using linux to serve up POP3 and other email services to tens of thousands of users. To boil down what I remember one problem was the way the vanilla pop servers look things up - it takes to long to iterate through one directory with ten thousand mail spools. So better solution was to custom build and break the mail up into sub folders, or use a pop server that already had this more efficient mechanism built in. Other folks even had seperate servers partioned by user name.
On another note I did some work a couple years ago for a place with probably about 10,000 Exchange users running off of the largest Alpha NT box from Digital at the time. Performance was 100% MISERY. It was almost useless. The NT Domain it was in had close to 30,000 users and they had massive problems finally settling on a few giant Alpha PDC/BDC's as opposed to many small servers.
-Scott
-Scott
One thing you oughta point out to the Exchange advocates is that MS wants to sell you a client license for each computer connects to it, even in an Internet setting. In 20-packs, they're about $55 each, so that would be over $1 million for 25K users. Obviously, a volume license would help, but still.... The money saved by using something else could pay for a lot of custom development.
Cyrus IMAPd is definitely gratis for academic use. If I thought it was gratis for commercial use I would have switched to it years ago. Do you have any evidence beyond the license text itself that says it is freely usable for any purpose?
Oh, wait, they've amended it recently:
Like ssh: I can use it to admin my own machines but not to let clients log in, not without buying a commercial license. It still ends with:
So I conclude that I can't use it to serve customers. I wish I could conclude differently; IMAP4 servers are hard to write and neither truly free one is so great.
My company has just over 100 employees, and our admin is begging to change to UNIX and sendmail. He hates sendmail configuration, but is so sick and tired of messing with exchange and our antivirus stuff every day that he wants to change _bad_.
:)
The config of sendmail may appear rather odd and difficult, but once you get used to it, it's not too bad. Compared to the vast list of problems in Exchange. We got hit by a spammer using our Exchange server to relay through and 'pretend' to come from our domain to send out to just over 1mil users. The mail we got back choked our mail system. I don't know much about the workings of exchange, but our admin found no way to stop exchange from relaying mail that wasnt for our domain. This is a one liner in sendmail. Sendmail is _much_ safer.
(P.S. For sendmail mail read sendmail/qmail - many prefer qmails ease of config)
Musicmaker
Everyone is living in a personal delusion, just some are more delusional than others.
haven't met anyone yet who's used this to any capacity http://www.postfix.org
Pork is not a verb
The talk was "The Design and Implementation of a Large Scalable Mail Server". It's about the mail cluster I set up here for Oxford University and it sounds similar to what the original request was for. We have about 30000 users. I successfully argued against MS Exchange and designed and built a completely Open Source solution based on a Linux cluster with 250GB of disk, UW imapd/ipopd, Exim (MTA) and an Apache/mod_perl-based web to mail gateway that I wrote called WING. The two Solaris nodes in the cluster were there for political reasons and are being replaced by Linux boxes within the next few weeks. There is a WING web page and mailing list which includes a link to the PostScript slides of the talk. The slides will also be shortly be available from UKUUG.
You might give this a try. Basically they have a load balancing daemon running to make sure that nothing gets overwhelmed.
At work here we are running about 16 MSexchange servers for about 3500 people. That's about 200/server. Icky....
-- Who is the bigger fool? The fool or the fool who follows him? --
In another post u asked something like "How can i be nostaligiv for a time that i did not live at?".
;-)
Let me tell you.
You lived before and are close to Enlightenment.
Maybe two three more lives or another hundred I do not know. But you learned how to play well - So now you can win.
Sorry to bother you and say hello to your kids
Hi,
I've have a 6809 based email server
running modified sendmail on OS9,
and it serves >>100,000 users fine.
I used to work for a medium sized ISP. At max (when I was there) we supported 10-15k dial-up users on _one_ mail machine (a 266 Alpha running DecUNIX) with a 1/2 gig of Ram)
When the # of users started to reach 20k, a half-refrigerator-sized Sun box was used to pick up the slack. The switchover was flawless. No mail was lost.
This was accomplished with plain 'ol sendmail for an MTA and a hacked-up Qualcomm POP server.
After the switch to the Sun box, the old single-proc Alpha continued to run the POP server for 20k users quite nicely, and is still going. Maybe $100,000 for hardware (big box, RAID, etc.) and if you use an open-source OS (The DecUnix licensing was kinda steep I think) and tools, the software is _free_.
Granted this was all dial-up (read, low-bandwidth.)
However, this same user base, with M$ Exchange, would have not only required dozens of machines, but Thousands of $ in software...
Do the math. Don't use Exchange unless you like wasting money and time.
Not to speak in favour of Exchange, but it is possible to add users from a .csv file fairly easily.
Look for Import/Export in the help file.
Mark
Y'know, just about every post I've seen has been some variant of "Well, properly administered Linux/FreeBSD/Solaris/*nix can have ten-year uptimes, but our plain-vanilla NT system crashed six times out of the box."
Doofus. How many of you would take a preconfigured Linux system and try to run it right out of the box? Would you call that "properly administered"? NT needs to be tuned, just like anything else. Many of these horror stories end with "and then I had to reinstall the OS!!" which should have been step #1, before anything else happened. Get machine, reinstall OS, tune OS, install software, tune OS and software, then let it all run. I've seen NT boxes with one-year uptimes. I've worked on NT boxes with six-month uptimes. I'm only able to say that because I know that NT is NOT the simple point-and-click operating system that the average *nix user seems to think it is.
And I can't get Linux working to save my life. Crashes on me all the time. I keep trying, because I know that it's an inherently stable OS and that my major problem is lack of knowledge and experience, but you don't see me on here screeching about how all *nixes suck and that I wanna go punch Linus, do you?
Sheesh. Bunch of savages in this town.
"Who is more foolish, the fool, or the fool who follows him?" -- Obi-Wan Kenobi
I'm probably going to regret saying this but...
:) I'm not going to dis qmail or any of the other solutions proposed because honestly I don't know enough about them to form an opinion. I've been working to expand my knowledge of other e-mail platforms but I'm a work in progress OK?
I'm a Microsoft Exchange administrator for a company with 8,000 or so users.
In regards to Exchange's scalability we're running Compaq Proliant 6500s with 512 MB of ram and Quad Pro 200 Mhz processors. Currently an average server holds approximately 1,600 users. CPU utilization hovers around 20-25%. For a while one box was running with approximately 2,500 users on it, but we spread out the users for load balancing.
Is Exchange THE enterprise solution? No, there are numerous products which could be used as you've all stated. But, Exchange is scalable enough to support a large number of users.
One drawback to Exchange is they way the information store is currently handled. As a single database it can get pretty large and so most Exchange server loading is restricted by the size of the information store rather than by the number of users. That limit tends to be at around 30GB because larger information stores require too high of a restore time in the event of disaster recovery for most organizations. So, you can take the average size you wish a users mailbox to be and divide that by 30GB to arrive at a rule of thumb number of users. (The next version of Exchange (is that 3.0) will address the IS size issue natively)
Using CommVault, one could probably triple the size of IS and thus increase the number of users per box.
I will admit we use the Outlook Client which connects via the Microsoft Exchange service, rather than POP3 or IMAP4. I know of no tool to load test an Exchange server running as an IMAP4 server so these numbers are not necessarily accurate for that.
Probably the biggest drawback for this group is the lack of an Outlook Unix client (not that I would imagine you all would be clamoring to use it). Microsoft has indicated they are working on one... but I'm not holding my breath.
There are also a variety of helpful resources and links for Exchange. I know of few questions one couldn't work out in an Exchange implementation with a little help from one of the Microsoft Exchange mailing lists.
Whatever you decide to go with, I wish you luck.
I'm currently working on the feasability of a mail cluster for 100 000 accounts. It's not done yet, but our various experiments gave us various answers.
:-)
We use exim, Qpopper with mysql patche, mon, fake and rsync. Each base box hold 88 GB of data and are fully duplicated (double delivery with exim, and further syncronization with rsync). The switch between a main base box and his double are handled by mon and fake. A hot spare then reconstruct a new double, delivery and popper deletions are blocked during the reconstruction.
Two problems aren't solved yet:
- raid 1 between boxes
- imap
I hope that imap will work when nfs locking will be reliable. For raid 1 over boxes I have a very tiny hope that nbd could be a solution.
Anyway, we made some tests, and it somewhat works already. We are now tuning various parts and writing procedure to handle the beast and react to failures (our current estimate is one major but handable failure every month).
If you have ideas of working solutions for my 2 problems don't hesitate to share
Nicolas
Let's do some quick numbers.
:-) Network, possibly. CPU, almost certainly.
25,000 users. Give each one a 10MB mailbox. That's 250GB of disk. With, say, 10,000rpm 9.1GB disks that's about 27 disks (_before_ adding redundancy). The internal disks in the Ex500 series machines are fibre channel.
Assume we're going to have an average of 2,500 users on at once, each using, say 4MB of RAM. Approximate this to 8GB RAM, or about a gigabyte per processor. Note we're now utilising a 64-bit address space. Good-bye, Intel.
Lots of spindles = fast. 10,000rpm = fast. Fibre channel = fast. Most of the user's requests will be coming out of RAM once they are logged in. I don't think the disk subsystem will be much of a bottleneck in this system
You might be able to get away with fewer CPU's if you were to get an external disk controller for the mirroring etc, but I've always been a fan of doing it in software and just buying more CPU (and avoid RAID 5 at all costs).
Linux or *BSD may be able to run on this sort of machine, but will not perform as well as Solaris (yet - we'll see how SGI's efforts help).
Handling a 25,000 active user base on one machine might be a problem for intel machines, and (dare I say it), most Open Source OS's.
Solaris will do this, but you will probably need to run it on a _big_ box, like a Sun Ex500 class machine with about 8 or more processors. And get their SIMS product, too, it's pretty well optimised for the high end. Other high end commercial unixes like AIX and IRIX will no doubt scale this far as well.
If you are able to go distributed (ie, the organisation is easily divisible geographically), then something like Linux or FreeBSD with qmail or smail will probably cut it.
Beware that exchange servers offer a fairly high level of integration with Outlook, which a product based on open standards will not be able to deliver.
There are multiple solutions listed on www.qmail.org for a single-uid mailstore which scales to millions of users. I have a customer who needed a system which could handle a million users on a pair of HA Sun Enterprise boxes. I delivered it.
-russ
Don't piss off The Angry Economist
You call 20,000 users "scaling"? Multiply that by fifty and then you can start to use the "s" word.
-russ
Don't piss off The Angry Economist
We currently have a mail setup handling mail for about 50 000 users. We have a home-brewed solution which is unfortunately cannot handle imap at the moment. We have 3-4 P166 machines to handle the POPping, with the users spread over these machines. The mail spool is mounted from a netapp nfs machine. There are 2 or 3 dns-round-robined PIIs running sendmail.
The pop client and procmail replacement were coded in-house for speed, simplicity and remote authentication.
However, we are now working on a new solution, which would comprise the following:
3 HP NetServer LC3 machines running modified versions of cucipop and procmail, and sendmail. (This is for NFS safe file locking). cucipop is modified to authenticate off a radius server (via PAM). Procmail is modified to allow all mail boxes to be owned by one user. Sendmail user a virtuser table to accept and deliver mail. There are no extra users in the passwd file. These machines will sit behind a layer 4 switch (or failing that, use DNSRR)
We also have an exchange server (LC3) handling mail for about 400 users. It falls over all the time.
Well, here in mexico we had the same trouble for us, a large mail system for like 20k users, we switch from sendmail which was ok, but not that fast, to a patched version of qmail to support MySQL databases, also the qpop (latest version) supports MySQL.
You loose the ability to offer shell acces cause the entire password system is on MySQL, but you gain a reliable mail system, which can grow to x numbers of accounts., also you can have the same login more than once on different domains with this system.
unless you have a need for groupware type features, don't mess with either exchange or notes. exchange does handle larger installations better than has been reported here- go to http://www.swynk.com for details on such things. we run 250 users on a quad p-pro 200 box, and it could easily handle 1500 users without blinking. the key is not trying to do anything else with that box. there's also a version that can have a store bigger than 16gb. notes is just goofy in my opinion- the clients are slow, and if you want to use something other than the clients, be prepared for even more slowness, as well as quirkiness. IMAP doesn't work well with it, for example. The Georgia state govt. went to notes for everything, and it's just as much of a consultants feast as exchange. unless you have a firm commitment to groupware and changing your enterprise, don't mess with notes. the real reason to go with open source is the plain jane messaging formats, plus users have lots of choices for clients. sendmail is a must for relaying in and out, but beyond that i can't specify.
stored on computers from birth to the grave
Indiana University runs the cyrus server on an Sun Enterprise 450 with a couple of Sun A1000 disk arrays, and it works beautifully. We get about 1000 concurrent IMAP connections, and the machine is not breathing hard yet. It worked so well, we are buying another server and more A1000 to continue moving the bulk of the campus to it.
The total number of users is about 14000 on the one box. I imagine we can pretty easily hit 20000, and I don't think that 25000 is out of range. The machine and the software is really outperforming our expectations.
We use Exchange for some faculty and staff that require calandering. There was a movement to use Exchange for the entire campus, and the projected plans were to use 50 (!) Exchange servers to deal with the problems of recovering and repairing corrupted databases.
Exchange is full of features that you expect as standard on an internet mailer that just don't quite work. Forwarding is done either with the "Out of Office" assistant which rewrites headers poorly, rendering the To: line of forwarded mail useless for identifiying all the recipients of a message. The same feature is used for vacation mail, and responds to EVERY message regardless of the To: line.
Beyond that, Exchange does not properly masquerade or resolve top level mail domains. It is not a good mail system in an environment where everything down to the client software is not strictly controlled.
There are a multitude of other things that don't work as you want them to. Microsoft is generally not a company that is responsive to requests and needs of consumers. Their reaction tends to be, "You don't know what your needs are" or "You shouldn't do that. That is misusing our software".
Frankly, Exchange did not have the robustness or flexibility of sendmail as an MTA and cyrus as an IMAP/POP3 server. For a small workgroup solution, I seems to work fine. Any larger, and the support issues become a nightmare.
I was a small part of an Exchange Installation of about 6,000 users. We gradually added users about 100 at a time migrating them from existing MS Mail servers. Most things said about Exchange here are true: It is a resource hog. It is expensive. It takes a lot of administrators. It also is tightly integrated with Outlook and of course NT. You would need a lot of servers to handle 25,000 clients and so a lot of NT and NT infrastructure. Where I worked this was used as an excuse to eliminate Novel as file and print servers and Sun as a web server. We had such a large commitment to NT to do Mail that it didn't make sense to train people in other OS's. We used 2 Linux machine for mail relays to the Internet and they worked flawlessly. That was what got me interested in Linux. They're gone now.
The best thing I could say about Exchange was that it was better than what we had before which was a patchwork of outdated equipment, servers and creative network design.
Regarding creating users: There is a command line program Adduser, if I remember. That creates an NT Account and also an Exchange account simultaneously. You could use this in a script but NT doesn't have a great scripting shell.
To be perfectly blunt, to implement Exchange like that would be *INSANE*.
6 8525655b0050f2f2/35BEC3BF6D717A3F852567120 07A435A?OpenDocument
First of all, suggesting to implement an NT solution for an organization of that size is already tempting your job security, but to actually do it?
Assuming standard users and needs for this system, I can only recommend using a Lotus Notes/Domino system. If you've got the cash, there is simply no better solution out there, or even close.
Run Domino (the server end of Notes) on several UNIX servers. Solaris (SPARC and x86), AIX, and HP/UX are all supported, with a Linux port (Caldera 2.3 (currently in beta) and Red Hat 6.0 will be supported, as well as SuSE 6.1 and Pacific HiTech) on it's way Q4 99 per DevCon.
Notes has got all you'll ever need, and R5 simply blows away anything M$ has out there. You've got to pay for seats with Notes, but to tell you the truth, Exchange is free, and you get less than you pay for.
Plus, your users can run the Notes client on any Win32 they think is prettiest (please tell me you'll use NT and not 9x on the client end).
Look at this for a guy in your situation who had to deal with Exchange.
Some other really good links are here, here, and here:
http://www.notes.net/50beta.nsf/7d6a87824e2f097
(problem with the last one, copy it and cut out the space that is stuck between the zero's, the href tag keeps putting it in! It is a great article though : )
(TIP: Show the guys with the money those links so they know why you should use a Domino/Notes solution.)
I'm reading with wonder about peoples wildly different experiences with Exchange (I have personally observed massive differences in performance between different instances of Exchange)
What I think people need to fear more is being at the whim of a Micros~1 platform agenda. I personally have a gripe with how Micros~1 locks you into their formats on the desktop and database.
Before you consider exchange, ask yourself how you might be able to move from Exchange to another product. IMO its not easy, and each day you use it, it'll get harder.
Also, I have a chip on my shoulder with how the changing feature set of Exchange managed to rid my old company of its NNTP servers (IT guys said: NNTP is coming in Exchange 5, so we'll stop showing people how to use News and get them on Exchange mailing lists, so there's one client) just long enough to make news disappear from people's internal communications habits.
I wonder how many organisations stopped using NNTP when Exchange started to take off?
-pbk
Check out www.apexmail.com . they have over 50 000 active clients, running on a totally linux based system. (They use totally custom software). Their setup includes a dual PII 450 file server with a 150 GB RAID (on a Mylex RAID controller). The entire public site runs on 4 or 5 single PII 450s, all on a switched 100BaseT network.
...si hoc legere nimium eruditionis habes...
Check out www.apexmail.com . they have over 50 000 active clients, running on a totally linux based system. (They use totally custom software). Their setup includes a dual PII 450 file server with a 150 GB RAID (on a Mylex RAID controller). The entire public site runs on 4 or 5 single PII 450s, all on a switched 100BaseT network.
...si hoc legere nimium eruditionis habes...
Where I work they switched to Exchange and can't even get it to scale for 250. Look at it another way: Microsoft, with less than 25K users, won't run Exchange. What does that tell you?
Lacking <sarcasm> tags,
Go to sendmail.org.. they have a document
there about building large mail systems that gives
you a lot of the ins and outs. Sendmail.com has
a nice gui that makes managing everything easy..
and definatly setup a HA solution with 25k users.
I have a friend in Microsoft's Messaging group,
His response was.. "25k on exchange? Heheheheh"
open source is the way to go with this size
system. I have set up several 5 and 10k user mail
systems and open source has never failed me yet:)
IMHO I would go with a sun product for
reliability. Uptime will be your number 1 concern.
Malice95
You might want to check out Critical Path. On there site they have a document titled A complete email solution for Corporations
Maybe I've missed out on latest MS developments, but until recently Exchange was using an underlying Jet database engine, which is limited to 16 Gig. If your users are getting any attachments at all, forget it.
More, once (not if) it crashes, Jet is almost impossible to recover, even if you've had recent backups.
And don't think about restoring a single mail-box/file - it must be restored to a separate server. Exchange has some serious limitations, not least of which is how difficult it can be to change certain configuration items. And of course, you are limited to NT, and the NT domain architecture.
If you just want email, go with a pure email solution. You won't want to put all these on a single server, but there are many, many solutions available on numerous (stable) platforms. Try and go for something that has a good directory. Consider some of the many email 'appliance servers'.
I have actually rolled out an exchange site of that same scale but for my brother-in-law's small business I setup qmail, www.qmail.org, as my first Linux project. Its great and easy to manage, I have rebooted the box maybe once and that wasn't for any problems with qmail. There is a mailing list where you could ask about scalabilty. All Exchange sites of that considerable size do it with a multitude of servers. I think you can make several qmail servers talk to each other. I wish there was something in the open-source community that had all the features of Outlook(except Journal). That's what is really needed to switch the PHB's around.
Greetings,
I'm an Exchange administrator myself. I have heard that the max comfortable users you can host on a particular exchange server is about 2k.
--- I was far from home, and the spell of the Eastern sea was upon me. -Lovecraft-
Exchange and Notes do a lot more than just email and attachments so you should really figure out if you need all the extra functionality a product like that provides.
If you just want email between employees then you are really wasting your time and effort going to something that provides effectively a relational database optimised for groupware applications. Go with something like sendmail/qmail etc on the system of your choice. If you already have a large NT user account database then NT is probably a good platform to use (from a management point of view). If you don't then you'll save a lot of money up front using Linux where you don't have to pay all those licensing fees.
If you want something that does integrated public folders, tasks, shared schedules and so on with all sorts of automation happening in the background then Exchange is a pretty good solution, and Notes seems to work for a lot of people as well. We have Exchange here and have been *very* happy with it, though you should spend the time to learn how to admin, backup and restore the thing because it has some quirks you have to get used to if a server gets its power ripped out for some reason.
Exchange works best with Outlook as the client. It also does a good job with POP3 and IMAP, but you lose a lot of functionality (forms etc). Again, it really depends what your requirements are.
Remember, do a controlled rollout. Take a few of your most technically minded sections first and give them the new system. Work out the bugs on the more forgiving audience who will help you solve them. Do a staged rollout after that and remember to do capacity planning depending on what sort of use you expect.
jw
Fear: When you see B8 00 4C CD 21 and know what it means
FWIW, FreeBSD takes of the web frontend and Solaris handles the database which stores all the mail. To my knowledge it was never anything _but_ FreeBSD & Solaris. M$ tried, and quickly backed out NT because it just couldn't scale to something this size.
Software is an open source mail and pop server.
The machine is normally not heavily loaded.
This however is not because our machine is a stud and open source rules. (does rule but that's a different post) This is due to our usage pattern.
We have a lot of light users some who read mail with pine some who read via pop, but very few who get large attachments and/or leave their mail on the server for long periods of time so that they build up into huge mailboxes. They also have a tendency to spread the load out over the entire day. There is no 8-9 am peak as some places are going to have as their users arrive in the office and read their mail.
My point here is that it is not enough to say I am going to build a mail system for 20K users you have to understand (or guess correctly) what the users are going to be doing and when they are going to do it and plan based on the *peaks* you are going to support. If it runs fine for 23 hours a day and like a dog when everone trys to use it in the morning that will suck.
So Know Your Users, or your users will all know you.
Check out the Lance Armstrong Foundation
kayaking
While not an open-source solution, I used software.com InterMail at a previous job. It worked very well on NT - it handled a few thousand POP3 accounts without breaking a sweat. I believe they also have versions for a couple of other systems - check out www.software.com
Matt
It's robust, especially talking large figures like 25000 users and much, much higher. Notes 5 allows you to configure accounts as either Notes-only, and/or POP3, and/or IMAP. Sometime later this year, they're coming out with a version for Linux, too. The address book/directory is built for 1 million users. (Yaddah, yaddah, I sound like a salesman, whatever, but it hums along nicely at where I work with nary a choke. Unfortunately it's on NT boxes.)
Are you spontaneously enthusiastic about everyone having everything you can have? - Buckminster Fuller
It's not only that Exchange is less secure than other email environments, but the recent Melissa/Explore.zip worms show that MS products draw crackers like shit draws flys (apologies to any crackers out there). You can use the Melissa/Explore.zip fiascos to explain to your PHB why Exchange should not be used in a corporate environment, at least until MS makes the whole Win/Office/Exchange connection less succeptable to what are really pretty straightforwad script hacks.
The closer you are to the code, the happier you are. - Ancient Geek Proverb
Having to still support an old network installation run by a bunch of idiots (they are attempting to implement a mostly micro~1.oft shop), I can tell you that outlook doesn't scale to more than 2000 users before the maintenance starts to become a headache.
This is a large client trying to implement a server farm of 20+ NT machines, each server supporting 600-800 users, and combining the whole lot into a coherent whole. Fortunately I only have to fix their poor network designs. The team of administrators now numbers more than 50, most are MCSEs, none less than 5 years experience with Micro~1.oft products. They are tearing their hair out on a daily basis. Complaints number in the hundreds every day, and thats just the users who haven't given up completely.
My advice is to start looking at the larger commercial products, possibly Netscape's server. Get a reputable vendor to support it.
If you look at open source systems, start with OpenBSD and NetBSD.
Divide your system up between the MTA doing delivery/reception of the messages, and the MTA serving the users. Its ok if email to the outside world goes down for short periods of time, its almost expected. But if users cant get to their mailbox 100% of the time, you will look bad.
You also need to look at managing more than 32000 or 65000 users in the future, remember that various *nixes have either 15 or 16 bit UID fields. You should make sure user accounts/authentication/logins are separate from any UID system on any machine type. This means getting some kind of medium sized DB, and tying it into your auth and login schemes. Others have done it, its not that hard (look at AOL with 10million+ user accounts)
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
Whatever you do, do NOT use Lotus cc:Mail for large scale email! Our company started using cc:Mail in 1989 (Long before I worked here). Apparently it worked like a dream back then. 10 years later, cc:Mail is a nightmare.
:P While I was writing this, the phone rang. It was someone wanting cc:Mail support! God save us all!
I get calls daily from people who can't get their cc:Mail and are desperate for help. These users simply cannot get through to their mail at all for all sorts of stupid reasons. Lately the main reason being that the automatic directory updates somehow change cc:Mail's settings so much that it no longer works.
And forget about cc:Mail over Wide Area Networks. It takes 5, 10 minutes to connect and download messages in "Mobile" mode. If you are trying to use it in LAN mode over a wide area network with 1000's of users, forget it! It simply will not work.
Dialup is miserably slow. Forget about your 28.8 modem downloading at 28,800 bps. Hell, you'll be lucky if you can get dialup cc:Mail to work at all. I get at least 3 calls a day from people who can't get their dialup cc:Mail to work (I'm not even supposed to take support calls). Half the time I don't have an answer for them. It simply does not work.
A few weeks ago some bigshot from the company was completely blocked out of his cc:Mail. He could not get in at all. Couldn't even view his messages (and apparently he had some "urgently important" messages). After trying for hours, and the cc:Mail admin trying for hours, we finally called lotus and paid them $350 for them to fix it. They couldn't fix it either. Miserable!
Please people! I am begging you! Stay the hell away from cc:Mail. The problems which I described do not begin to scrach the surface.
The question on how to build large e-mail systems has produced an interesting thread on the available technology.
But if you have an organization with 25K+ people who need e-mail, are you sure you need to set up an e-mail system? These 25K people probably need lunch every day, and clean underware, but that does not imply that the employer must provide these services. Could you not just tell them to get Hotmail accounts?
Seriously, this should be an option for schools and universities, maybe even for companies. If security is the problem, then security should be a problem for all users of Hotmail and similar systems. How can Hotmail (or similar service vendors) provide enough security? Maybe if all e-mail is required to be PGP encrypted? Think of it. Which fortune 500 company will be first to outsource their e-mail to Hotmail?
No matter what solution you pick (or gets chosen for you) would you please report back to us with how it went? It would make a great case study either whether or not MS can handle a true enterprise level application or how well FreeBSD/Linux/whatever handles it.
Thanks...
putting the 'B' in LGBTQ+
I think that those assumptions are a bit generous. First, the specs do not mention shell access. Most people (that are Joe Average Users, at least) won't want shell access, for that matter; they'll prefer to store and send mail locally.
Secondly, expecting 1/10th of your user base to be on concurrently is, well, pretty heavy use. And unrealistic. At Queen's, we have a single Sun Enterprise 300 serving all student email - about 16 000 accounts. I've yet to see more than 300 simultaneous logins.
Thirdly, 8M RAM a user? You must be kidding. A couple (including shell and mail client), tops.
Fourthly, and most importantly, a single box can make things very unpleasant come upgrade time. Spread the whole thing across several servers in a cluster (IMHO). That way you can bring individual boxes down (you can even have one fail by surprise) without a major break in service.
--
There is no premature anti-fascism. -Ernest Hemingway
If management is looking for a "legitimate" companys product, I would suggest GroupWise 5.5 over Exchange. By picking this you'll need NetWare/NDS at least for the central server (so this might not fly if you're a 'NT shop') and for administration. There is a Solaris server version. Also clients for AIX, HP and Solaris. GroupWise supports LDAP, IMAP4, and POP3. It also includes a Java- and HTML 3.2-enabled version of GroupWise WebAccess.
Howdy, I've worked for an ISP that was growing rapidly. We shot from 20,000 to 30,000 customers in a very short period of time. Mail broke almost over nite. There was no warning sign. One day we had lots of capacity, the next... nothing. And we had a fairly beefy server: dual proc Sun Ultra Enterprise 2/300 with 1 gig of ram and gobs of disk. When it had it, the load would shoot up over 20 for hours (and we had 20 set as our sendmail throttle limit). We tuned and screwed around with the pop server and fixing sendmail, but it usually only bought us a week or so. In the end we rearcitected. The solution we came to is highly scalable, very reliable, and fairly easy to do. It was inspired by what Earthlink did a few years back (if you're a USENIX member see her e.)
.locking or anything like that. Contention is very low for user files. Each box runs cucipop for POP and sendmail for mail, each using very stock configs. We played with IMAP for a bit, and UW-IMAP is about the best around (in the free world, anyway).
Basically, we nfs mounted our users mailfiles so they were accessable across multiple machines. The biggest problem you run into here is file locking, but NFS will handle most of it for you... no need to implement
We added 2 more similarly equipped mailservers. This gave us N+1 redundancy. So, if a server fails, we still have enough capacity that we don't have any interuptions... we just take the bad server out of the round robin in DNS. (BTW: _all_ servers in this must have an A record for mailhost.domain.com... no aliases.. read the RFC). The NFS server was running on a RAID, so the only real single point of failure was the NFS server itself, not disks or mailserver. To get NFS redundancy, you pay a LOT of money. But, a good NFS server should run for years with a decent load.
So the end result is a care free mail architecture that should run forever and we can scale by just dropping in a new machine when we run out of capacity. Use something like scp and CVS to keep all your conf files the same and you're off to bigger and better projects.
Questions? pls feel free to contact me. Later
I'm down with that, as it were
Hey, I look at your offers but nothing yet impressive. Here is one fact I got while trying to solve one problem we had here.
/var/spool/mqueue. An administrator got a shot at his head and switched out his connection to repair his crashed linux box. Guess, what happened then? All the mail passed and stored at our side so we begin to expirience the same problem.
:)
My expirience began when one of our clients got shut down because his setup doesn't allow to pass some kind of spam: it is a local MTV office here running vote for favourite music and a funny fan just mailed them one message with Cc: full of duplicates of the same address! (Heh, how about 15k Cc: lines?)
So, those good linux box got swaped out by 15k of request at
And now the salt of my story:
#grep MTV *
no luck, no response, Control-C doesn't work (Control-Z;kill %1 works fine)
(the last attempt and a lot of time of waiting gives us 'Arguments list is too long!'
#ls -la
same the thing
#tar zcvf shittymail.tar *
same the thing
#tar zcvf shittymail.tar .
Did what we asked for! Take about 15 minutes to perform.
(some of stuff done here - shutting down sendmail and firing it up at another port just to deliver the mail, and so on... standard sysadmin mess)
So, we just moved this chunk of tarred files into Sun Ultra 2 workstation and tryed to examine the problem. Here is the result:
1. untarring works _fast_, reverse works without a flaw (~5 mins)
2. grep doesn't said there is a problem
3. 'ls -la' begin to work immediately
My conclusion: try your system to simulate the problem like a such stress test. I can't say about *BSD systems but Solaris UltraSPARC-box impressed us despite the fact that this Solaris box can crunch d.net code not enough well vs the same MHz of Pentim-processor system.
Ah, yes: this linux mailserver run latest sendmail and qpopper. Just think is it too hard to write a standard basic application for MTA, POP3, IMAP and so on and then give it a run for a community? May to say that after some facts in my life I'll sooner migrate from Sun to FreeBSD then to Linux box.
I don't know much about MacOS X server but i've heard that it is fine. Dunno about the mail server.
At Microsoft, we all use Exchange, many of us POP3 for laptops and such, but most MAPI (I personally prefer IMAP). Our organization is roughly 20,000 individuals and while adminstrating it may be a headache in the background for all I know, our servers almost never go down. When they do, it's typically because of scheduled maintenance.
The prevalent opinion on Slashdot seems to be that this is impossible. Obviously it isn't.
While I personally like Outlook2000 a lot, I can understand how some people might dislike it. The positive side is that the Exchange servers are naturals with the POP and IMAP protocols, so any compliant client will do. Eudora and OE are the two other clients I use commonly against the server.
So sure, it's scalable. As to ease of implementation, my small team of five has set up 10 exchange servers (releases varying from 5.0 to 5.5sp2) for team use and it took about three days. Your order of magnitude would be considerably greater but after you've done it a dozen times, I'd guess the method will be clear.
I can't say it's the best solution because I don't know. But it is a solution.
-konstant
-konstant
Yes! We are all individuals! I'm not!
Did I just not read this thoroughly enough? This was a hardware problem.
-konstant
-konstant
Yes! We are all individuals! I'm not!
Which is why there may be good odds that ext3fs will be ReiserFS, a filesystem designed for small files, though it is faster across the board..
One test (which it IHMO flunks) is a directory with 50,000,000 (yes, 50 million) files. It isn't creating that directory fast, though you can pull it off if you create it in chunks of 50,000 files or so. (Create a directoy with 50,000 files, thenk copy it over. Repeat)
Overall, its at most 10% slower (in some operations) than ext2fs on benchmarks for files around 10-20k, but faster in every other benchmark. 10%-60 faster on large files, 2-4x the speed on small files (200 bytes) Not sure if the benchmarks have been fully repeated with the newer version, or how old they are. But if you have either very large directories, or a large number of files, ReiserFS will be the way to go.
http://www.idiom.com/~beverly/reiserfs.html
Cyrus is "sort of" free. The only restriction I can see is that you can't sell a product based on it without permission from CMU.
That might mean the license doesn't meet the OSI requirements, but I don't think it places any practical limits on use, modification and redistribution of the source.
Lachlan.
For what it's worth, Cornell University supports University-wide mail on a set of Sun Enterprise servers using Sendmail as the MTA, and cyrus for POP and IMAP service. I don't know offhand what the current count of active users on the system is, but generally speaking the Cornell community is 30,000 people, almost all of whom use electronic mail on a regular basis.
Our service is primarily POP-based. We support IMAP connections, but not the full suite of IMAP services that would truly support a centralized mail storage facility. (Just not enough hardware for that at the moment.)
In any case, we've developed an extremely reliable service at this level, and the only part of the system that isn't open source is the OS.
If anyone would like more details on how we do what we do, or why we made certain choices (i.e. which MTA to use, which IMAPd/POPd, etc.), feel free to contact me.
I work for Enterprise Rent A Car and we have 3,500 store locations that have over 35,000 employees and one Central Corporate Office in St. Louis MO that has another 3,000 eimployees. We used to use the UNIX pop servers ( pop3.ibm.net, pop4.ibm.net....pop6.ibm.net) and we had a terrible problem with attachments currupting the users pop3 inbox. And a host of other problems with IBM's servers being overloaded from our pop requests. 1.5 years ago we went to MS Excange server ( one server ) to handle the 38,000 employees and it worked greate. Not only were they able to send Internet email but the groupware stuff was awsome. It was seamless with the other products like those that ran on our 27 AS400 computers. We've never had any exchange server problems. It delivers at a minimum of 1,000,000 email messages a day. Some our Internal Excange messages and some are Internet email messages. What's cool about it is that it automatically builds address books for all the users so they can lookup any employee in thier address book. A user is automatically added to everyone's address book when they get added as a user on the NT domain. Also the pop connections to the server are all encypted ( check the secure authentication option in Outlook ) so that password sniffers do not work. We could not emplament this on the UNIX pop servers for some reason. IBM said that they didn't support that option. I thought that was lame.
Well - we've been using it for a while and it looks that it does a really good job (though the Admin interface sucks boulders through the garden hose - Java + HotJava browser ... go figure!!). You have to thoruoghly read the admin manuals and check the anti-relay rules which are *NOT* enabled by default (so you may end up a SIMS Powered Spammer (tm) >:) ). :) (Can you spell (s)Exchange for Unix?)
Other than this - prepare to spend fsking big bucks on the licenses and on the machine that does the mail.
Mind you that related to the size of the sucked boulders - I don't think there's anything in the UNIX world that can compete with Netscape's MAIL
Cheers,
INO!~
PS: If you will use SIMS - do bloody apply the latest patches (has at least one script that go beserk in approximatively 24 hours)
The easiest solution, assuming you got some dough to spend: A $4999 450 MHz PowerPC G3 running OS X Server, with the Stalker software Communigate Pro Email server. This is what this software was made to do > 500 users. Nothing opensource running on linux could handle this, believe me.
My friend was hire full time by a large computer company, they have many people in many remote sites. He has now been there two years and still does not have exchange working correctly, even for the one main site of 1000 people. So if you want to spend the rest of your professional career futzing with exchange, or paying someone, then go ahead. Otherwise, UNIX is the way, I see no reason that an open source soltion. Remember split up the load, boxes for sending, boxes to handle pop, and one big ass box to store all of this.
Currently the ISP I work has 50,000+ mail accounts on a multiserver solution... NetApp 720 for the file store on a private segment, A DellPowerEdge 2300 for the incoming mail server, and a 450 Pentium II with 256M ram for the POP box, MySQL POP Authentication tables, and a standalone MX20/Relay on another Micron Pentium II (I believe, haven't had to mess with it much)...
;-)
;-)
If you need IMAP, it gets tough... Except for IMAP I'd recommend Qmail.. it's the most robust thing, besides the NetApp that we have.. With NetApp and a RAID0 backed queue drive it screams...
You could all of the above with Penguins or VARs for pretty cheap... at guess 10,000 excluding the NetApp... If you use a beefy linux box with a fast raid 5 for the NFS server back end you'll also allow your servers to "load gracefully"
If you need IMAP but on a single domain, use the UW IMAP server... It even comes as an RPM and looks great... If you need multiple virtual domains like we do... IMAP looks pretty grim...
As it is when we go to web based email it looks like we're going to have to do a WebBased POP client...
Careful though, IMAP can get *WAY* more abused though... With IMAP you have the tendency for people to park and use more space... With POP it's just grab and go..
Resource wise POP is a better bargain and most clients can deal with it just fine... IMAP isn't worth the server load IMHO... Use qmail anyway ya can...
"Don't fear death... fear not living..." -me
Carnegie Mellon has a nice link that will help anybody out that needs info on email servers...
x t.html
http://andrew2.andrew.cmu.edu/cyrus/email/servt
SpamapS -- Undernet #Linuxhelp
Sendmail can do it but it's fairly inefficient. You may want to look into PostFix. Qmail is also popular. I believe you can use db or dbm files with to route mail to other servers in at least the first two cases.
Note: this is not for the faint of heart, and probably involves overhauling your mail setup.
;-) I have yet to implement it (I'm going slowly and testing every step. Right now I'm converting my user base to LDAP, then I'll update Cyrus, etc).
I've also looked into setting up something like that. It came down to using something like LDAP which is scaleable, standard, and OS-nonspecific for mail users. Then I had Qmail and Cyrus on the backend. I applied the Qmail LDAP patch, allowing Qmail to use LDAP for its user list. For Cyrus, there is another pwcheck file here. It adds the ability for it to authenticate against an LDAP server. Finally (yay!) we need the Qmail/Cyrus glue (as Qmail uses a slow mail format compared to Cyrus, and has no IMAP support [yick!]).
Incoming mail goes to Qmail, which uses scripts to deliver to Cyrus (users LDAP listed). User logins go through Cyrus server programs (which use LDAP auth), and can get their messages that way. This should also support virtual domains.
All in all, it sounds good
Have fun.
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
I know for a fact that Lockheed Martin uses Exchange for it's LM-Xpress email program that handles 175,000+ users. How's that for capable? The two important characteristics that they were looking at when selecting an email system were security and reliability (i.e. no lost emails), and Exchange has been a real success.
On a related note, anyone who says, "Don't use Exchange because Micro$soft sucks," really needs to be a little more open-minded. Sure, they're not a perfect company, that they do produce some solid products... exchange, office, and internet explorer come to mind. If only there were linux ports...
Dave
If you had about 3,000 users, exchange would work. However, it's not very scalable (contrary to their propiganda). The staff here was forced by the college network to switch over to it so that we could read and use the documents students were sending to us. To it's credit it's pretty flexible, but when the time came to switch the students over to it Microsoft said that it couldn'
t support 20,000+ users. Also from what I've seen of the budget it was a pretty hefty price, even with an educational discount!
There have been many posts on this already, but here is my two cents.
I'm not completley expierienced with making mail servers which are THAT big, but I can guarantee you Exchange will not handle it. You're looking for a FreeBSD machine, dual, maybe quad, processor, RAM galore (> 512), and, of course, the hard drive space to go with it.
For the SMTP agent, look at qmail, or if you're gutty, sendmail. You might look at a third party software package for this.
For the POP3/IMAP4 agent, look into UW-IMAP4. I've had it work with over 100 users on a single dinky Pentium 166 w/32 MB of RAM running FreeBSD.
check out
www.earthlink.net/about/papers/mailarch.html
It's how Earthlink does their Mail system with Sendmail, and Qualcomm's POP daemon version 2.2.
Not really open source right now (but probably will be), but my company has developed a mail/webmail/POP server solution, first outsourcing email for free similar to free homepages, and eventually offer single-click outsourcing, as long as you point your MX at us. Keeping it closed source doesn't benefit us, since the real value of outsourcing is our hardware/network setup with large RAIDs, and big pipes.
I have 10,000 messages in my INBOX right now (too many damn mailinglists), and many with large >2mb attachments, and the system runs fine. Similar to Sun's internet mail server, mailboxes are stored in a very scalable fashion, with hash indexes and preparsed MIME headers. As a comparison, the standard Linux IMAP/POP servers choke on my mailbox once it reaches about 3000 messages (meaning, lots of paging, and slow performance)
There is an integrated address book, with color coding, special mailing list detection, and an anti-spam system that uses a Visual Turing Test to eliminate bulk machine-generated spam.
Basically, it's an SMTP, POP, Webmail, and address book server, written from the ground up to be scalable, and it's pure Java. And for the naysayers, I'll tell you that the average time the SMTP daemon spends processing a message in he queue is 10ms. I develop it on Linux, and deploy it on a cluster of big Solaris boxes and it runs very nicely with no change.
(by being scalable, I mean that all resource utilization is kept at a minimum, in terms of disk thrashing, socket usage, and object allocation, so that for instance, 1000 concurrent users can be handled on a very cheap box)
If there is significant community interest for a pure-Java set of servers, I'll spend more time putting together an open-source package of the the MTA, POP, and IMAP servers in the near future.
If interested, send me mail.
Cheers,
-Ray
I read about a company (about 1300 users) in the Philippines that chose a Linux solution over MS Exchange. Follow the link to read about it.t em2/index.html
http://www.computerworld.com.ph/headlines/newsi
Why not try a Compaq DS10 or DS20 with Tru64 UNIX as it comes with a ton of Open Source stuff (with binaries) and kicks the crap out of any NT solution?
I've seen mail servers based around them that are cheap to put together and have excellent performance in terms of POP3, IMAP, and Sendmail applications. Check out this URL for the description of the software and this URL for the boxes themselves.
This is my opinion. To make sure you don't steal it, it's covered by the DMCA.
We've currently got our mail occupancy capped at 15M, and we've got several SSA RAID arrays to support the installation. About 140 gigs of mailboxes, mqueue, and cyrus housekeeping goop.
There is a Zope component that connects to an IMAP server to provide a web oriented front end that can be accessed via HTTP, XML-RPC, and even FTP. The most common configuration is HTTP (via a fairly functional but very minimal web forms based interface).
I jokingly call it NotMail. Since being hired by the authors of Zope, I have not be able to pursue it to hard core, but Python, is as most of you know, is vey easy. Most of the core is done, there would just have to be some improvements made to get it to work with the new Zope version, 2.0b1, some threadsafing issues (2.0 is fully threaded) and of course, some cosmetic changes to the HTTP interface.
For an example of a Zope powered site, check out Technocrat.net.
-Michel PelletierMachine-wise, PC hardware should handle this nicely. Take a few PCs and put them in 2U rackmount chassis with a hardware RAID adapter mirroring (RAID 1) the system disk. Put a layer 4 switch, such as a Foundry ServerIron or Alteon AceDirector in front of these machines. Need to take a machine down because of upgrades or hardware failure? Want to add more machines to the cluster to improve performance? No problem. Take the machine down and the switch automatically removes the downed machine from the available pool of machines.
Mount mail spools from a Netapp Filer. Put a few hot spares in the Filer and now you've got redundancy and fault-tolerance for your mail spools, too. Plus, it'll be fun if you give anyone tours of your facilities. Imagine their reaction when you nonchalantly yank a disk out of a Filer taking that kind of load, and then watch the Filer automatically rebuild the drive on the hot spares you have in it. :-)
You can also cluster Netapp Filers ( more info), which would allow you to have two Netapps that would automatically sync their contents. If one fails, the other takes over transparently.
Lastly, if you're going to be having all of this NFS activity with that size a user base, I would highly recommend putting a second NIC in each of your server PCs. Link these second NICs in each of the PCs into a physically separate network from the one the users will be using to retrieve their mail. Gigabit Ethernet may also be an option here depending on the traffic demands of NFS in your situation. There are two advantages to this separate network. 1) It separates your NFS traffic from your user requests and data transfers, thus preventing the network from reaching its saturation point as rapidly and 2) you can secure the NFS network and allow only NFS requests and other management processes to use this network. If your Filers are only homed to this NFS network, it would take a break-in to one of the PCs just to gain a chance at administrative access to the Filer holding all of your user data.
The only downside to all of this is that Qmail doesn't have a daemon to serve IMAP. I don't have any experience with it, but I've seen Cyrus recommended a lot for IMAP service. There are patches on qmail.org that patch Cyrus to authenticate against a CDB, the file format that qmail can use for authentication and other lookups. You might be able to do something along the lines of creating a cron job that checks for a timestamp on the LDAP entries, and updates the CDB entry for a user if the LDAP info has changed since the last invocation. Maildir support might be dicier; I only spent a few minutes on it, but I couldn't find any info on getting Cyrus to deliver to a maildir.
The paper is available here (in postscript).
At the talk I had the impression that the softwware was free. I cannot find it on their (skimpy) web site though.
From their description, 25,000 users wouldn't begin to make it sweat.
i work for an isp in the uk and we run our mail on mcis (microsoft's commercial internet server) in conjunction with LDAP and SQL. it works fine with around 300,000 users (adminttedly with clustering, but allunder nt using compaq h/w).
i would recommend against using ms exchange as i know many people who have had loads of trouble with it.
A company that a buddy of mine works for is going to attempt 100,000 users on an NT box (as in a single box, dual PII-450's). Using software called MailSite . It doesn't have anything to do with exchange....
So, I dunno if it will work, I am skeptical myself...
8-bit monkeys, 4-bit ducks
We are currently using multiple Sendmail servers with the same MX priority. If one server is down / overloaded another is chosen in its place.
/var/spool/mail directory goes down no server can receive mail until the NFS share is available again.
/var/spool/mail directories on each server without the need for NFS mounting (using a custom / modified local mailer program that prevents loops and cascades mail to all spool directories).
You must mount the mail spool directory across all participating servers. Mind you if the server that is exporting the
There is another Sendmail solution my collegues and I are working on that will update all
Hope this helps, and for your own sake avoid Exchange - the license issue will kill you alone.
We deploy cross platform solutions and have a lot of business converting sites form NT to Linux or FreeBSD. NT...fast...hmmm???
I have seen NT systems that were installed by very well known companies. They were run basically out of the box and had numerous problems, including the BOD, on a mail server running Exchange.
Everytime I compare FreeBSD/Linux uptimes with any Microsoft solution (even at different,unrelated sites), the former always wins.
Go to:
a tion.html
ftp://koobera.math.uic.edu/www/qmail/faq/orient
A little dittie on Exchange, Qmail, and HotMail.
See it at the LinuxWorld expo later this month, or look at http://www.hp.com/go/openmail
HP's newest baby check it out... "Lowest TOC, scales to millions of users". http://www.ice.hp.com/cyc/om/00/index.html
When the source is open, the possibilities are endless.