Slashdot Mirror

← Back to Stories (view on slashdot.org)

Behind the Scenes at Hotmail

Posted by Zonk on from the that's-a-big-datacenter-you-have-grandma dept.

mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."

3 of 292 comments (clear)

  1. UNIX? by IAmTheDave · · Score: 4, Interesting

    If I recall correctly, wasn't Hotmail originally run on UNIX boxes?

    --
    Excuse my speling.
    Making The Bar Project
  2. Fairly Impressive by eldavojohn · · Score: 4, Interesting
    I don't know about everyone else but this article was shocking to me.

    Not only are the questions well picked but the some of the answers are quite interesting. For instance Phil on scalability:
    The problems are those of basic client-server programming--that is, figuring out the browser/http/server data-access patterns and optimizing the protocols, extending these protocols as new functionality is introduced, and ensuring that these protocols work across geo-distributed data centers when the speed of light becomes a factor. Designing applications with built-in redundancy so that they are resilient to abuse is also a challenge.
    Before reading this article, I always had hotmail pegged as a hacked together e-mail system less organized than a monkey sh*tfight but if Phil speaks the truth, I've underestimated them. They're a hacked togethor server mess with a lot of effort put into staying afloat--and they have been doing well for a long time.

    I guess I've always taken my free Hotmail account for granted.
    --
    My work here is dung.
  3. Hundreds of administrators by Iphtashu+Fitz · · Score: 4, Interesting

    more than 10,000 servers spread around the globe ... are overseen by hundreds of administrators.

    Heh. I used to work at Akamai which provides content delivery services for many of the biggest sites on the web. They have somewhere over 15,000 servers that are managed by tens of administrators, not hundreds. In fact, a typical NOCC (yes, 2 'C's for Akamai) shift at Akamai is only staffed by 8 or so people, with only a couple of senior level admins on call. And they're delivering all sorts of web-based content, including streaming, not just e-mail.
    But then Akamai runs them all on linux, whereas I belive Hotmail is all Windows based. You do the math.