Slashdot Mirror


Behind the Scenes at Hotmail

mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."

20 of 292 comments (clear)

  1. High level of QC! by Anonymous Coward · · Score: 5, Informative

    From the article:
    Hotmail relies on less than 100 system administrators to manage it all.

    From the summary:
    Between them, they process billions of emails per day and are overseen by hundreds of administrators.

    Brought to you by the high quality control here at /.

  2. Re:UNIX? by Kraegar · · Score: 5, Informative
    It used to be on FreeBSD w/Apache, now it runs on Windows w/IIS. It's not exchange based.

    Read about it

  3. Re:Does anyone know... by jcaldwel · · Score: 2, Informative

    [root@jboss html]# wget --save-headers -q -O- http://www.hotmail.com/ | grep "^Server:" 2>/dev/null Server: Microsoft-IIS/6.0

  4. Re:UNIX? by ThinkFr33ly · · Score: 2, Informative

    While there were initial problems migrating to Windows, 100% of Hotmail now runs on Windows.

    Also, Exchange was never involved in the migration. Hotmail is a combination of C++ ISAPI filters, COM+ (ATL) Enterprise Components, and SQL Server.

  5. Coral Cache by OctaneZ · · Score: 3, Informative

    Looks like the site is down, it is however there is, however, a Coral Cache copy.

  6. Re:Fairly Impressive by mekkab · · Score: 3, Informative

    Not only are the questions well picked

    The interviewer is ACM Queue editorial baord member Ben Fried, who is the managing director of Morgan Stanley's worldwide IT deptartment.

    --
    In the future, I would want to not be isolated from my friends in the Space Station.
  7. Re:UNIX? by Amoeba · · Score: 5, Informative

    Yes. Hotmail was originally run on clusters of E3500 and E4500's running Solaris 2.5.1. After they got bought by Microsoft, a major initiative to migrate all boxes to Windows was undertaken in 2000. Hotmail has been 99.9% Windows for over 3 years now. The remaining 0.1% are some legacy solaris boxes used to handle backups for clusters... and even they are being phased out slowly.

    --Amoeba (who no longer works there)

    --
    Do not taunt Happy-Fun Ball
  8. Re:Phil Smoot??? by Red+Flayer · · Score: 2, Informative

    Where do you think the unit of measurement came from? I visited my brother at MIT in '88 or '89, when the bridge (and the Smootlines) had been rebuilt... I thought it was the best thing about MIT (I learned differently later). on the origin of the Smoot.

    --
    "Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
  9. Re:Spam improvment, but not perfect yet by mccdyl001 · · Score: 2, Informative

    Yes it will. It will iterate through all your zip files. And if you password protect, gmail wont process it. So how do you send an exe or zip? Just give it some other arb extension (like bmp or dat or xxx or anything) and then attach it. On the other end give it back its proper extension, and you're done.

  10. Re:Fairly Impressive by xtracto · · Score: 2, Informative

    Man, it is the Association for Computing Machinery magazine, I mean, it is not any PC-Weekly WalMart mag.

    If you don't know about ACM publications, here are other interesting ones:

    Ubiquity: IT opinion magazine and forum
    TechNews: News Gathering Service for IT Professionals
    eLearn: Distance learning magazine
    MemberNet: Your Key to the World of ACM...and Beyond
    Computers in Entertainment: New ACM online magazine

    P.s. Sorry for the K.B.

    --
    Ubuntu is an African word meaning 'I can't configure Debian'
  11. Re:F**Kin Speak English ! by Tet · · Score: 2, Informative
    The only possible advantage I can see in substituting the word "leverage" is that it sort of implies they are making the best use of these tools that they can

    No, in fact it just makes no sense at all. The word "leverage" is a noun. The verb he was looking for is "lever", at which point it would at least have been grammatically correct. Of course, "use" would still have been a better option.

    --
    "The invisible and the non-existent look very much alike." -- Delos B. McKown
  12. more Re:The SMOOT as unit of Length by Alien54 · · Score: 2, Informative

    The Boston police have been known to use smoot markers to indicate accident locations on the bridge. Apparently Smoot's experience as a unit of measurement led to a life-long career; he eventually became Chairman of the Board of the American National Standards Institute, and later President of the International Organization for Standardization.

    --
    "It is a greater offense to steal men's labor, than their clothes"
  13. Re:Hundreds of administrators by SpotTheCat · · Score: 3, Informative

    From TFA:
    What's interesting is that despite this enormous amount of traffic, Hotmail relies on less than 100 system administrators to manage it all.

  14. Re:F**Kin Speak English ! by angelo · · Score: 2, Informative

    Please don't "don't disagree" with people. It's like an abstract double negative. Instead use "I could agree with" or similar constructs.

  15. Re:UNIX? by enantiodromia · · Score: 5, Informative

    This is only half true. The _front end_ runs on Windows with IIS. The _back end_, where the email data is stored (the User Stores), are Solaris. The front end machines dont mean much. If one or twenty go down, there are tons more to take their place. They are simply removed from the load balancing and marked as "admin plz fix this some day". The back end machines however, are super critical, as each user lives one one, and only one, user store. That machine goes down, and hundreds of thousands, to millions, of Hotmail users cant get to their mail. And thats why those machines run Solaris.

  16. Re:Hundreds of administrators by Anonymous Coward · · Score: 2, Informative

    RTFA, not the /. summary. Hotmail runs on tens of administrators as well. The /. summary got it wrong.

  17. Re:The article is fine...but by MannyO · · Score: 2, Informative

    You have to take into account less developed countries than the US.
    I travel a lot to Mexico and it amazes me that *everyone* has a hotmail account there. They advertise it on fliers, on business cards, etc....
    Some people will have (own) a domain like http://www.muchostacos.com.mx/ and *still* print their muchostacos@hotmail.com email.

    It kills me....

    I think this is because of the proliferation of internet cafes back when having internet (or a computer) at home was prohibitive.
    All those machines with their homepages set to msn.com and nothing but windows messenger as the IM client...

  18. When *I* worked there.. by EvilStein · · Score: 2, Informative

    (just a little over a year ago) At a fairly new data center, Hotmail's backend still had a LOT of Sun Enterprise 4500 boxes running Solaris. None of them were being phased out at all. But, all of the boxes that were being brought online were HP/Compaq boxes running Windows.

  19. Re:UNIX? by Bing+Tsher+E · · Score: 2, Informative

    And you know this, authoritatively?

    The Hotmail service has changed considerably. Maybe the backend is still Solaris. But you didn't provide a cite.