Slashdot Mirror
Behind the Scenes at Hotmail
mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."
If I recall correctly, wasn't Hotmail originally run on UNIX boxes?
Excuse my speling.
Making The Bar Project
Not only are the questions well picked but the some of the answers are quite interesting. For instance Phil on scalability: Before reading this article, I always had hotmail pegged as a hacked together e-mail system less organized than a monkey sh*tfight but if Phil speaks the truth, I've underestimated them. They're a hacked togethor server mess with a lot of effort put into staying afloat--and they have been doing well for a long time.
I guess I've always taken my free Hotmail account for granted.
My work here is dung.
I used to get about 35 spam a day in my primary hotmail account that I'd had since 1997. Now it gets about 4 a day so things have improved, but my biggest concern about Hotmail is that its virus scanning is horrible. There have been several times when it would have let me download a virus attachment, or allowed multiple obvious virus messages through. They've switched to Trend from McAfee, but I think the problem still remains.
Saskboy's blog is good. 9 out of 10 dentists agree.
more than 10,000 servers spread around the globe ... are overseen by hundreds of administrators.
Heh. I used to work at Akamai which provides content delivery services for many of the biggest sites on the web. They have somewhere over 15,000 servers that are managed by tens of administrators, not hundreds. In fact, a typical NOCC (yes, 2 'C's for Akamai) shift at Akamai is only staffed by 8 or so people, with only a couple of senior level admins on call. And they're delivering all sorts of web-based content, including streaming, not just e-mail.
But then Akamai runs them all on linux, whereas I belive Hotmail is all Windows based. You do the math.
Compared to Google clusters, they seem to be light years behind. As a software developer, I can tell you that the key to rolling out applications quickly, is to have a decent framework in place. Whatever that framework might be (from shell scripts to java monstrosities), once its in place, developing apps on top of it are easy. Similarly a well thought out app execution environment is golden.
If you ever check out Google's MapReduce, you'll see what I mean. It's just so well thought out and so elegant, that its easy to believe that they can scale outwards forever. You'd not be too far off if you thought that Microsoft were rethinking their whole production environment to compete with Google.
There's no way that Microsoft can quickly and easily roll out vast new applications that scale, because that whole clustering framework is completely opposite to what Windows provides.
Newsfollow.com
I was a strong hotmail user before Microsoft took it down, uh, I mean took it over.
It was a great service! One of the first, and probably the best.
Microsoft took it over and there was no advancement or innovation for years (a decade?). Spam ate up my tiny inbox while Microsoft just threw MSN graphics all over the place.
When Gmail came out, I gave it a try. It was everything Hotmail could have been years ago if it hadn't been bought by MS! (Well, it COULD have been out of business, so I've got to give them that I suppose).
They forced Microsoft to pay a little attention to features. They gave out a little more storage and started blocking some spam, but it was too little too late.
In order to write this I decided to visit my hotmail inbox, I haven't been there for a while. 136 emails, and 43 have been detected as junk. They are ALL junk--A party invite from "heather", a Cola Quiz, etc. 136 undetected junk emails out of 179.
And even at that, they still only give 1/8 the amount of storage that Google does.
Crap, on top of that I just looked at a spam with pictures in it and it didn't auto-block them like Google does. Now I'm probably infected.
Thanks Microsoft!
From,
The guy who used to argue the advantages of Microsoft to the Unix admins...