Behind the Scenes at Hotmail

mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."

Comments

From the immortal words of Henry Spencer

[hackstraw]

"Those who don't understand UNIX are doomed to reinvent it, poorly."

From the article and elaborating on the /. summary (It has a print version that consolidates the 4 pages together if you want):

Q: Are there scaling reasons to think about the benefits of a command line for managing over a GUI, or are there other things to think about?

A: Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line. That's the only way you're going to be able to execute scripts and gather the results over thousands of machines.

Also, we all remember the scaling issues that MS had when they took over hotmail and initially tried to switch from freebsd to Windows.

MS had to port over cron jobs because its not something that is installed and used by default under windows like UNIX. They had to rewrite the "inefficient" perl code that ran fine on FreeBSD to C++. They had to redo the memory allocation to prevent memory leaks in the new C++ code. Read about it from the goat's mouth http://www.microsoft.com/technet/interopmigration/ case/hotmail/default.mspx.

I can't wait until FreeBSD and other inferior OSes get tools to find memory leaks. One day....

(That last line was sarcasm and not a flame).