Slashdot Mirror
Behind the Scenes at Hotmail
mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."
"Between them, they process billions of emails per day and are overseen by hundreds of administrators."
And how does the NSA process all that email? Now THAT would be an interesting technical challenge!
From the article:
/.
Hotmail relies on less than 100 system administrators to manage it all.
From the summary:
Between them, they process billions of emails per day and are overseen by hundreds of administrators.
Brought to you by the high quality control here at
Read about it
"Those who don't understand UNIX are doomed to reinvent it, poorly."
From the article and elaborating on the
Q: Are there scaling reasons to think about the benefits of a command line for managing over a GUI, or are there other things to think about?
A: Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line. That's the only way you're going to be able to execute scripts and gather the results over thousands of machines.
Also, we all remember the scaling issues that MS had when they took over hotmail and initially tried to switch from freebsd to Windows.
MS had to port over cron jobs because its not something that is installed and used by default under windows like UNIX. They had to rewrite the "inefficient" perl code that ran fine on FreeBSD to C++. They had to redo the memory allocation to prevent memory leaks in the new C++ code. Read about it from the goat's mouth http://www.microsoft.com/technet/interopmigration
I can't wait until FreeBSD and other inferior OSes get tools to find memory leaks. One day....
(That last line was sarcasm and not a flame).
Yes. Hotmail was originally run on clusters of E3500 and E4500's running Solaris 2.5.1. After they got bought by Microsoft, a major initiative to migrate all boxes to Windows was undertaken in 2000. Hotmail has been 99.9% Windows for over 3 years now. The remaining 0.1% are some legacy solaris boxes used to handle backups for clusters... and even they are being phased out slowly.
--Amoeba (who no longer works there)
Do not taunt Happy-Fun Ball
I read you subject as "KISS my hotmale body".
This is only half true. The _front end_ runs on Windows with IIS. The _back end_, where the email data is stored (the User Stores), are Solaris. The front end machines dont mean much. If one or twenty go down, there are tons more to take their place. They are simply removed from the load balancing and marked as "admin plz fix this some day". The back end machines however, are super critical, as each user lives one one, and only one, user store. That machine goes down, and hundreds of thousands, to millions, of Hotmail users cant get to their mail. And thats why those machines run Solaris.