Slashdot Mirror
Behind the Scenes at Hotmail
mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."
The overriding philosophy seems to be KISS.
Don't try to tell me that the guys at Hotmail only want to Rock & Roll all night and party every day?!?
He who knows best knows how little he knows. - Thomas Jefferson
..who they call for support? :-)
If I recall correctly, wasn't Hotmail originally run on UNIX boxes?
Excuse my speling.
Making The Bar Project
"Between them, they process billions of emails per day and are overseen by hundreds of administrators."
And how does the NSA process all that email? Now THAT would be an interesting technical challenge!
What OS it runs on and which web server? I am not trying to be funny.
One ring to bind them - should probably have more fiber and less rings in their diet.
Not only are the questions well picked but the some of the answers are quite interesting. For instance Phil on scalability: Before reading this article, I always had hotmail pegged as a hacked together e-mail system less organized than a monkey sh*tfight but if Phil speaks the truth, I've underestimated them. They're a hacked togethor server mess with a lot of effort put into staying afloat--and they have been doing well for a long time.
I guess I've always taken my free Hotmail account for granted.
My work here is dung.
I used to get about 35 spam a day in my primary hotmail account that I'd had since 1997. Now it gets about 4 a day so things have improved, but my biggest concern about Hotmail is that its virus scanning is horrible. There have been several times when it would have let me download a virus attachment, or allowed multiple obvious virus messages through. They've switched to Trend from McAfee, but I think the problem still remains.
Saskboy's blog is good. 9 out of 10 dentists agree.
From the article:
/.
Hotmail relies on less than 100 system administrators to manage it all.
From the summary:
Between them, they process billions of emails per day and are overseen by hundreds of administrators.
Brought to you by the high quality control here at
> To do that they have returned to the command line.
Absolutely.
I'm currently in the process of trying to change our company culture away from legacy GUI tools and toward command-line tools.
Scriptability is a highly under-rated goal. I'm not against GUI tools -- but they need to be built on top of scriptable utilities.
I always thought that the command line was a user interface. You know, interfacing between a user and a computer.
It's hard to picture using a computer without any sort of user interface. I'm pretty sure that, in order to call it "using" a computer, some sort of interface must exist, be it keyboard mouse and monitor, binary switch, light gun, real gun, neural link, telekinesis, or whatever. Otherwise, you're not using it, are you?
On the other hand, maybe the article is correct- a lot of operations group probably don't want to use "any sort of user interface" to communicate with their computers. They want to be sitting on a beach in tahiti drinking daiquiris, thousands of miles away from the computers they're supposed to maintain.
Can anyone tell me how to set my sig on Slashdot?
In the landscape of today's megaservices, Hotmail just might be Mount Everest
Is this true? I thought Google might be the Everest. Anyway, speaking from personal experience, in my university every student has multiple yahoo/gmail accounts but just a handful use Hotmail. Can someone throw light on the actual number of users all over?
It's like one commercial after another. 'See how great we are!!'
Right... it's always more interesting to read article after article about only unsuccessful operations run by people who aren't proud of what they do, and don't face huge, global challenges.
You're cranky because it's MS. If exactly the same article ran, substituting "gmail" and "google" for all of the other names, you'd say, "cool!"
Don't disappoint your bird dog. Go to the range.
"Those who don't understand UNIX are doomed to reinvent it, poorly."
From the article and elaborating on the
Q: Are there scaling reasons to think about the benefits of a command line for managing over a GUI, or are there other things to think about?
A: Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line. That's the only way you're going to be able to execute scripts and gather the results over thousands of machines.
Also, we all remember the scaling issues that MS had when they took over hotmail and initially tried to switch from freebsd to Windows.
MS had to port over cron jobs because its not something that is installed and used by default under windows like UNIX. They had to rewrite the "inefficient" perl code that ran fine on FreeBSD to C++. They had to redo the memory allocation to prevent memory leaks in the new C++ code. Read about it from the goat's mouth http://www.microsoft.com/technet/interopmigration
I can't wait until FreeBSD and other inferior OSes get tools to find memory leaks. One day....
(That last line was sarcasm and not a flame).
Looks like the site is down, it is however there is, however, a Coral Cache copy.
Stabilized at 100% of bandwidth.
Boys from the City. Not yet caught by the Whirlwind of Progress. Feed soda pop to the thirsty pigs.
Where do you think the unit of measurement came from? I visited my brother at MIT in '88 or '89, when the bridge (and the Smootlines) had been rebuilt... I thought it was the best thing about MIT (I learned differently later). on the origin of the Smoot.
"Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
Why does he keep mistaking the word "use" for the word "leverage" ? The only possible advantage I can see in substituting the word "leverage" is that it sort of implies they are making the best use of these tools that they can in which case you would think that most people would have already assumed they are not making the worst possible use they could of the tools and it's interesting that the author feels it necessary to make that distinction.
Could anyone suggest a better rhyme for spam?
There once was a young man name Sam,
Who spent his whole day sending spam.
But at night he went drinking,
And this led him to thinking,
That maybe he would end up being damned!
God invented whiskey so the Irish would not rule the world.
It's interesting, but for some specific uses, IIS does a great job of handling traffic. For example, streaming video from servers seem to run a lot better on IIS and seem to be a little less resource intensive. I'm not sure about the overall use of Hotmail, though.
[%] Cingular Ringtones
more than 10,000 servers spread around the globe ... are overseen by hundreds of administrators.
Heh. I used to work at Akamai which provides content delivery services for many of the biggest sites on the web. They have somewhere over 15,000 servers that are managed by tens of administrators, not hundreds. In fact, a typical NOCC (yes, 2 'C's for Akamai) shift at Akamai is only staffed by 8 or so people, with only a couple of senior level admins on call. And they're delivering all sorts of web-based content, including streaming, not just e-mail.
But then Akamai runs them all on linux, whereas I belive Hotmail is all Windows based. You do the math.
AT first, it was BSD running on a bunch of identical custom-made sub-1U servers. But No! Then it was replaced by windows boxes . . . racks and racks of 99c Fry's keyboards velcroed to the backs and fronts of racks, with miles of small-gauge track, upon which ran diabolical steam-powered robots, each with a single arm and with fingers at the end, forever fixed at the precise spacing to stab the keyboards' CTRL-ALT-DEL keys. Noisily the robots rumbled back and forth on their appointed rounds . . .
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
He submits, of course, that any program can be written in any reasonable language -- for they all are, after all Turing machine's equivalents. But the quality of the tools can make a difference between a feature being added next week and not at all.
If Hotmail's admins are back to command line and scripting anyway, maybe, they should've stuck with FreeBSD.
Look at how quickly Google is rolling new things out -- their platform allows them to.
In Soviet Washington the swamp drains you.
The Boston police have been known to use smoot markers to indicate accident locations on the bridge. Apparently Smoot's experience as a unit of measurement led to a life-long career; he eventually became Chairman of the Board of the American National Standards Institute, and later President of the International Organization for Standardization.
"It is a greater offense to steal men's labor, than their clothes"
Hotmail can no longer be used with Outlook because Microsoft admits it's easier to block everyone and charge for access rather than cancel the accounts of spammers.
Yet despite the talented people working on Hotmail, they still fall flat on their face in two apparently challenging areas:
1.) Logging in. You would think that since I already typed hotmail.com in the address bar, I wouldn't have to type "@hotmail.com" in the log-in form, but alas, the solution has aluded them. In fact, it seems to have escaped them altogether, since it used to be that way. Apparently having seperate hotmail.com and msnmail.com, storing a cookie, or even just having a radio button is beyond the limits of their servers. The extra 12 characters I have to type wouldn't bug me so much, except for the fact that there's no logical reason for it.
2.) Logging out. The msn.com page that you're redirected to when you log out ranks almost as low as the AOL page that pops up when you log into AIM (a seperate problem that can be solved by using GAIM) as far as usefullness. The pointless crap they pass of as news on that site drives me up the wall (TomKat Wedding Colors to Include Fuscia, Poll: Will an asteroid hit the earth in 2029? blah blah blah). All I want to do is delete my spam, but I have to put up with this in order to do it.
With crap like that, I'm often tempted to ditch hotmail. If they can't take being a dust bin for email lists that I don't care about seriously, I see no reason why I should bother to use their 250 mB of email storage. Oh wait, it's free...right.
Compared to Google clusters, they seem to be light years behind. As a software developer, I can tell you that the key to rolling out applications quickly, is to have a decent framework in place. Whatever that framework might be (from shell scripts to java monstrosities), once its in place, developing apps on top of it are easy. Similarly a well thought out app execution environment is golden.
If you ever check out Google's MapReduce, you'll see what I mean. It's just so well thought out and so elegant, that its easy to believe that they can scale outwards forever. You'd not be too far off if you thought that Microsoft were rethinking their whole production environment to compete with Google.
There's no way that Microsoft can quickly and easily roll out vast new applications that scale, because that whole clustering framework is completely opposite to what Windows provides.
Newsfollow.com
I was a strong hotmail user before Microsoft took it down, uh, I mean took it over.
It was a great service! One of the first, and probably the best.
Microsoft took it over and there was no advancement or innovation for years (a decade?). Spam ate up my tiny inbox while Microsoft just threw MSN graphics all over the place.
When Gmail came out, I gave it a try. It was everything Hotmail could have been years ago if it hadn't been bought by MS! (Well, it COULD have been out of business, so I've got to give them that I suppose).
They forced Microsoft to pay a little attention to features. They gave out a little more storage and started blocking some spam, but it was too little too late.
In order to write this I decided to visit my hotmail inbox, I haven't been there for a while. 136 emails, and 43 have been detected as junk. They are ALL junk--A party invite from "heather", a Cola Quiz, etc. 136 undetected junk emails out of 179.
And even at that, they still only give 1/8 the amount of storage that Google does.
Crap, on top of that I just looked at a spam with pictures in it and it didn't auto-block them like Google does. Now I'm probably infected.
Thanks Microsoft!
From,
The guy who used to argue the advantages of Microsoft to the Unix admins...
(just a little over a year ago) At a fairly new data center, Hotmail's backend still had a LOT of Sun Enterprise 4500 boxes running Solaris. None of them were being phased out at all. But, all of the boxes that were being brought online were HP/Compaq boxes running Windows.