Slashdot Mirror
Behind the Scenes at Hotmail
mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."
"Those who don't understand UNIX are doomed to reinvent it, poorly."
From the article and elaborating on the
Q: Are there scaling reasons to think about the benefits of a command line for managing over a GUI, or are there other things to think about?
A: Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line. That's the only way you're going to be able to execute scripts and gather the results over thousands of machines.
Also, we all remember the scaling issues that MS had when they took over hotmail and initially tried to switch from freebsd to Windows.
MS had to port over cron jobs because its not something that is installed and used by default under windows like UNIX. They had to rewrite the "inefficient" perl code that ran fine on FreeBSD to C++. They had to redo the memory allocation to prevent memory leaks in the new C++ code. Read about it from the goat's mouth http://www.microsoft.com/technet/interopmigration
I can't wait until FreeBSD and other inferior OSes get tools to find memory leaks. One day....
(That last line was sarcasm and not a flame).
Why does he keep mistaking the word "use" for the word "leverage" ? The only possible advantage I can see in substituting the word "leverage" is that it sort of implies they are making the best use of these tools that they can in which case you would think that most people would have already assumed they are not making the worst possible use they could of the tools and it's interesting that the author feels it necessary to make that distinction.
I think running a mail server is a bit more complicated than a webserver or a streaming server for video
It sounds to me like you don't understand what it is that Akamai does. They're not just running web & streaming servers on their 15k machines. They're distributing content in real time in a way thtat vastly improves user access all around the world. You may have heard when Victorias Secret held their first video-streaming lingerie show. Well their servers couldn't handle the load because of all the people trying to watch it. They became an Akamai customer, and Akamai was able to redistribute their streams in real-time all over the globe. To be able to take video (or just web content) from a single source and distribute it quickly and efficiently to thousands of distributed users in real-time is a huge undertaking. Akamai has some very impressive technology to be able to do this.
I'm not saying that running a mail service like Hotmail is a piece of cake, but I do think that what Akamai does is a lot more difficult and impressive when you think about it. If Akamai's distributed environment were to drop off the net then you probably wouldn't be able to access any of the on-line services of most of their customers. (And that's just a small subset of their customer base) The ability to keep websites like those of Microsoft, eBay, Fed Ex, Red Hat, etc. all highly responsive to end users is not a simple feat by any stretch of the imagination.