Hotmail about to collapse under load

An AC submitted this interesting tidbit from those folks over at NetCraft. To quote from the page: "HotMail has commenced its much awaited migration to a Microsoft operating system. Some Windows 2000 machines have recently been moved into the load balancing pool, with currently between 90-95% of requests being served by the established FreeBSD/Apache platform, and 5-10% from Windows 2000." This is not the first time MS are believed to have attempted this (but I'd appreciate hard evidence confirming that, instead of the more normal rumours and whispers).

Comments

I can verify this

[Anonymous Coward]

I worked with Hotmail and Microsoft for a while and can tell you they've been working on getting everything to Windows for quite a while. They were working on designing just about everything so they could port it over, however hotmail has a *LOT* of proprietary code, 'databases', servers, meta-servers, frontdoor code, etc, so it has obviously taken some time. I expect the change to happen soon.

From the horse's mouth:

[talks_to_birds]

Dogfood

"HotMail has commenced its much awaited migration to a Microsoft operating system. Some Windows 2000 machines have recently been moved into the load balancing pool, with currently between 90-95% of requests being served by the established FreeBSD/Apache platform, and 5-10% from Windows 2000. The Hotmail site infrastructure is enormous, and even if everything runs smoothly, a migration will likely take several weeks."

I'm not sure why several people have gone off on /. for attributing this story to an AC, saying that it's FUD, or that /. is about to collapse under bias..

C'mon folks, this is what Netcraft has said; /. is merely quoting them.

Click on the link and go read it, and deal with it..

It's about a third of the way down the page, under Around the Net

t_t_b
--
I think not; therefore I ain't®

Re:Netcraft Result

[ruud]

I went to Netcraft's site and this was the response back from a request to Hotmail.

www.hotmail.com is running Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b on Windows 2000

I am not one to jump to conclusions but something strange seems to be going on (or is it just me). Unix version of Apache on Windows 2000????

Conspiracy theorists will have a field day with this one.

It's quite simple actually. The machine that accepts the TCP connection (the load balancer) forwards it on to one of a pool of webservers. Sometimes also called reverse proxying. Obviously the load balancer and the webservers do not need to run the same OS, as you see in this example.

--

Re:Some Real Data: 79.8% Win2K

[waldoj]

To be honest, I grabbed the hostname from somebody else's post. (Like I said -- don't know nuthin' about Microsoft's system.) I think I got that address from Spock the Vulcan's post, which is a single head dump from Lynx. Also, JOKane posted saying that 6.1% of his (?) 1,000 wgets were processed by the IIS server.

I wonder if the login server isn't different from the actual mail servers? Hotmail does, after all, immediately push you to one of their law.hotmail.msn.com servers. That was my assumption, though perhaps flawed, when I used the lw7fd.law7.hotmail.msn.com address. Is anybody familiar with their topology?

Anyhow, I repeated the experiment, this time on lc2.law5.hotmail.passport.com, which is the server that www.hotmail.com pushes to. My numbers there more closely matched yours:

• 953 "Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b"
  
• 47 "Microsoft-IIS/5.0"
  

4.7% W2K. That's closer the the results that I'd *like* to see. :) I hope some Slashdotter knows more about MSN's load-balancing setup that we do!

-Waldo
-------------------

For $3300 it better be

[ch-chuck]

if one can install it and get it up and running quickly like a business machine should that'd be great - altho for 200 servers that comes to, mmmm, $660,000 (zowie!). If I have to start putzing around trying to trick it to work, read thick manuals, grep TechNet for workarounds and edit registry keys then I'd just rather use an open platform in the first place - at least it *worth* learning, and isn't just BG's flavor on the month that will be obsolete, useless knowledge in the next cycle - e.g., I'm glad I didn't commit much time to learning NT security domains, other than just enough to get by and get paid, now that there's a completely different system in 2K.

Re:Slashdot FUD

[Col. Klink (retired)]

> their most successful net service has been running on Unix since day one.

That's not quite true. They did attempt to convert to NT once before, but it failed under the load.

That's the justification for the headline.

uhm

[Blue Lang]

it doesn't say how many w2k boxes are being used to replace the BSD boxes. if they're putting in 400 w2k machines in place of 200 BSD ones, of course it will prolly work, but that isn't much of a reason for using w2k.

--
blue

Re:uhm

[Ralph Wiggam]

It's a risky PR move. As it stands, geeks laugh at them because a site that should be the flagship of the MSN empire doesn't run MS software. But then geeks laugh at them anyway.

If/when they move to Win2K (I would assume Datacenter, does anyone know for sure?), and it works, then the marketing folks can point at it and say, "HotMail runs Win2K and it will surely work for your smaller site". The danger is there because the whole site could just crumble if Win2K isn't up to the task. If that happens, mainstream press like the Wall Street Journal will run front page articles saying that Win2K choked in the face of major hits. That damage could be irreperable. I know the the adoption of 2000 has been modest (to put it nicely). This could be a very important move for MS.

-B

Re:uhm

[egore]

Has anyone stopped to think that it's a really good thing that Microsoft uses a Microsoft server platform for all of their servers whether or not it's the right tool for the job? If you think about it, what better way is there to improve your OS than to start using it on a large scale. Microsoft is smart enough to realize that, if they want people to use Win2k for e-mail servers, they should use it for an e-mail server as well. That way, in making it a great e-mail server for themselves, everyone else benefits because MS will be making their product that much better for that task. MS has the right approach here, IMHO.

Just for the record, I would not mind seeing MS use more of Linux because they can definately learn some things from Unix-derived OSes (like Linux and *BSD and Unix itself).

- Alex

NT's problem is cost, not lack of features

[jetson123]

NT can, of course, handle Hotmail. That's not the issue. And Microsoft has to do this if they want to appear credible at all. They also have to do it in order to figure out what is wrong with Windows 2000.

The question is whether it's cost effective for customers to deploy Windows 2000 in that way. There are several components to the cost:

• The Microsoft software licenses. Expensive for business customers, a non-issue for Microsoft.
• Software licenses for 3rd party tools that plug holes. Probably also given almost freely to Microsoft by companies hoping to get acquired.
• Hardware costs. Windows 2000, in practice, probably needs more hardware to achieve similar levels of reliability and performance as UNIX machines (I'm not talking raw hits/second).
• Maintenance and administration costs. Windows 2000 scores very poorly in that regard in my opinion. But for a Windows-only shop like Microsoft, it's probably cheaper to go with Windows than with BSD.

So, can it be done? Sure. And Microsoft needs to do it if they want to play at all. But it is not a convincing demonstration that it's a cost-effective solution.

If I were to start another big web project, I'd still not pick Windows 2000--except for niche server applications, I believe it's still too expensive to license the required MS and non-MS software, and it requires too much manpower to administer. It also doesn't have any place to grow right now: a multiprocessor Xeon is it.

Actually, it shows us one thing: the fact that Microsoft has been playing around with this for, what, three years, suggests that you can't easily create the software for a Hotmail-like service rom scratch and with complete specifications on the NT platform within that time period even if you have unlimited amounts of money and all the Windows expertise in the world. Perhaps that's the most important lesson of that exercise, and something aspiring web startups should take note of.

Re:Some Real Data: 79.8% Win2K

[tramm]

I ran the same experiment, but used www.hotmail.com for the hostname rather than the one that you selected (how did you come up with that one?)

My results:
240 Server: Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b
15 Server: Microsoft-IIS/5.0

94% Apache, 6% IIS. Much closer to the 5% numbers quoted in the article.

Windows 2000 performance

[Taral]

Although everyone here will go "it'll never work" or "it'll fail instantly", my experience with Windows 2000 is that, if properly set up, it can be quite a stable platform. We should all be watching closely, since this will be a real test to see whether Windows 2000 can meet or exceed an equivalent UNIX+Apache system.

Re:Bit of insight to Hotmail

[AJWM]

The system there is divided into "Clusters" each cluster having about 400 machines. There are about 12 clusters there, which mean about 4,800 machines. [...] they are having to add machines and clusters as fast as they can because of all the increased volume/new users.

An earlier poster said something about using "the right tool for the job". Those 4800 machines are about one ninth (1/9) of the number of virtual Linux machines that an IBM S/390 can run simultaneously, and at several times the cost of that S/390.

This isn't about the right tool for the job at all, Hotmail should be hosted on Big Iron. (To bad for MS that NT or W2K won't run on 390 hardware.) I would hope that the difference in cost (of the 4800 x86 boxes vs an S/390) is coming out of Marketing's budget. (But of course it isn't.)

Re:Why would it go down?

[stab]

Adding nodes can only make the system faster, regardless of whether the new nodes are Windows or BSD

Definitely not true. If the additional machines are significantly slower and/or unreliable, then you destabilise the overall quality of service of Hotmail.

Think about it ... if 10% of the machines suddenly buckle under the load, but in such a way as to escape automatic removal, then 10% of URL requests will die mysteriously.

This is a pretty positive move from Microsoft's point of view though - after that initial burp, they've been very careful from a system integration point of view, and seem to be quite sane about the way they are migrating to 2000 now.

Re:Slashdot ain't all that hot either.

[synx]

Well, MySQL is MySQL, but I find its being used for many projects which I think is totally inappropriate. I mean anything above the most simple site which is done as a hobby, I think MySQL is not good enough... what happens when your RDBMS fall down go boom? I find Ramus's explination why he thinks Foreign Key's are "bad" just too funny for words (ok, not quite, but still quite funny).

I'd like to point out that I think MySQL has a niche, but I think people are using MySQL in many places outside of that niche, and that takes away energy from more worthy projects ie: PostgreSQL.

Yes, There are many sites I'd suggest MySQL over Oracle, but I wouldnt use MySQL for a site I was paid to do ever again.

Re:What the hell was that headline about

[Izaak]

"Collapse under load" what are you talking about? Any chance of a link to anything that might substantiate that...?

Well, I can't speak definitively about Win2000, but I know for a fact that Windows NT can not come close to FreeBSD/Apache for web serving. I used to be a partner in a small ISP that tried to run NT/IIS... it fell over big time. We put FreeBSD and Apache on the exact same hardware and it scaled up with no problems. Dynamic content seemed to be the real problem area for NT/IIS.

I imagine MS has made some improvements in that area with Windows 2000, but I am not about to bet MY business on it.

Thad

Hotmail doesn't use sendmail; rather, qmail

[tmoertel]

Sendmail would crumble under that kind of load. Hotmail, rather sanely, uses qmail for outgoing deliveries. Here's the Message-ID from a mail I received from a friend who uses Hotmail:

<20000428205548.12433.qmail@hotmail.com>

Note the qmail part.

Re:Speaking of buckling under a load...

[MikeBabcock]

Quite frankly, I don't care how many millions of hits per second it requires to prove that IIS on NT is supposedly faster than Apache on Linux, *nix or *BSD.

I care about stability. The fact that my Apache on Linux system doesn't crash, doesn't give in, doesn't care ... that's why I use it. So IIS can hit millions more hits per second ...then fall down. ;-)

At least I don't have pieces to put back together.

Hotmail HAS tried this before

[fence]

I was a consultant at an online yellow pages company from 1998->2000.

Our director of engineering and one of the VPs were courted by Microsoft in late 1998 or early 1999 to consider using Windows NT to serve our pages and do our directory searches.

As evidence of Microsoft's ability to handle large loads, they were shown racks upon racks of rack-mounted NT boxes. Hundreds of boxes. The idea being that when some fall over, there are plenty to take up the load.

Our VP was told that these machines were to be the new Hotmail servers.

The director and VP came back to town all excited and wanted us to look into getting rid of our pesky Sun Enterprise boxes.

About a month after they got back, we showed them an article about Microsoft's failed conversion of Hotmail to NT, and how they had to roll back to FreeBSD and Apache.

to see how the story worked out, check out what your Directory EXpert is using today.
---
Interested in the Colorado Lottery?

Actual Percentages

[JOKane]

I used wget 1,000 times and checked the "Server" header. Only 61 of these requests were processed by IIS servers.

This suggests that the 10% figure that's been thrown around is (from an MS standpoint) very optimistic. 5-6% seems much more reasonable.

Dictionary spamming

[ars]

It's called dictionary spamming. They take every combination of one or two words in the dictionary plus 0-2 digits at the start end and between every word.

They also simply try randomly every single combination of letters and numbers, up to arround 5 letters, more then that would take too long.

So in short, when you create a hotmail address make it long, and don't use words from the dictionary and you won't get too much spam.

Re:How the hell are you going to /. Hotmail?

[passion]

By using those 41,500 Linux servers on one S/390 mainframe... that I bought for $45.00 - that's how :)

Or, I could convince people to join team slashdot at distributed.net for sending ping of deaths to hotmail just as M$ finishes switching over to Winblows!

Just me and my beowulfed slashdot community

Re:Here goes

[Ranger Bob]

$lynx -head -dump http://lc5.law5.hotmail.passport.com/cgi-bino
HTTP/1.1 200 OK
Date: Tue, 01 Aug 2000 18:37:33 GMT
Server: Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b
Cache-Control: no-cache
Expires: Mon, 01 Jan 1999 00:00:00 GMT
Pragma: no-cache
Set-Cookie: BrowserTest=Success%3f; domain=.passport.com; path=/
Connection: close
Content-Type: text/html

$

Intel Wins!

[dpilot]

If BSD is replaced, the BSD doesn't win.

We're all presuming that Win2k will take at least double the servers to handle the same load. So while Microsoft will claim victory, they're (presumably) paying a BUNCH of money for hardware for this showcase. So (presumably) MS doesn't win, either.

Margins are so small on computing hardware that the the boxmaker doesn't win, either. Once upon a time, the CPU and hard drive were the only really profitable parts. Given hard drive price erosion, lately, is that list down to the CPU? In that case, Intel emerges as the only clear winner in this whole thing. (I presume these are not AMD CPUs they're fielding.)

I say wait and see

[Fervent]

The TCP/IP stack *is* different, as somebody else mentioned - I definitely notice improved performance on both the Win2000 clients and the one Win2000 server lying around. Memory usage is better too - out of 256 megs on one of the clients only 60 is being used by the system directly (that's a lot better than KDE, which eats up around 180).

Not to say that an NT-based system will auction best the Linux and FreeBSD's of the world, but from what I've seen (despite the still extraorbinant-price MS charges) it's a pretty good, very reliable system.

Here goes

[Spock the Vulcan]

$ lynx -head -dump http://lw7fd.law7.hotmail.msn.com/
HTTP/1.1 302 Redirected
Server: Microsoft-IIS/5.0
Date: Tue, 01 Aug 2000 18:32:24 GMT
Location: http://lc5.law5.hotmail.passport.com/cgi-bin/login

$

Re:didn't they try this before?

[ColdN]

After buying Hotmail in the end of 1997 they tried to run it on NT. Here's an article about it.

Good for them!

[update()]

I'm sure there'll be a lot of sneering here but Microsoft has a good habit of eating their own dog food. In the early days of NT, they started using it internally as much as possible. They had a slow, buggy email system that lost a lot of their mail -- but NT and their servers got better. I've read that Solaris only started to get really usable when Sun forced their engineers to use it instead of SunOS.

Maybe if Motorola hadn't gotten rid of all their Macs they'd have improved the G4 in the last year.

Predictions

[banky]

1. It will fail miserably, and the BSD community will cry *SEE! WE'RE BETTER!* and it will disappear in the mists of time. How many MCSE's do you know that talk openly about the previous Hotmail efforts? None, that I know, anyway.
Or,
2. It will succeed, tremendously, and then MS will use it as a massive PR campaign, how they replaced the "superior" BSD. The other side (thats us, I guess) will grumble "yeah, with double the number of machines/many times the cost/lots of effort/etc" and we'll go back to telling the boss that its NT and not Samba.
Or,
3. It will be a partial success, MS will Service Pack and Hotfix away, and both sides will claim victory, anyway.

But you already knew that.

Data point

[FascDot Killed My Pr]

I setup a hotmail account long ago just to get my /. login. I've never posted that address anywhere else, and I rarely even check the mail there. But I generally have 20-30 spams in 6 months. How can that be? Either /. is leaking addresses or spammers are trying names at random.

Anyway, back ontopic: I just went and tried to get in. It took SEVERAL seconds to load each page. That's slower than I've ever seen it. And don't tell me it's the Slashdot Effect--something the size of hotmail should handle that.
--

Re:Slashdot ain't all that hot either.

[synx]

Consider it a failure of MySQL ;-)

heh, yes, I'm rabidly anti-mysql, and after studying the mysql docs for many years and seeing how the mysql guys do things, I've decided they are fucked in the head to put it mildly. They've decided that 20 years+ of RDBMS research is just plain wrong and decided that table-level locks is the way to go and that transactions are not a good way to do things. Not to mention that foreign keys are just a hassle...

I wouldnt mind so much except everyone's pushing MySQL as a oracle-replacement! I mean jeez, sure its fast... as long as you keep your concurrency low...

which brings me to the last point, slashdot is slow because they're using mysql, the table concurrency is killing them... they used to generate the static-comment page once a minute with a little daemon thingy because they couldnt get performance from multiple-readers and multiple-writers to the same table.

FUCK!

Its like the last 20 years of good research and hard work hasn't ever happened... the multiple-readers/writers with good performance problem has been licked so many times, that its just sad to see software which still cant get it right... ;-(

ok.

bye for now.

How the hell are you going to /. Hotmail?

[joshamania]

I've seen a couple of comments here suggesting that Hotmail be slashdotted. How the hell are you going to accomplish this? How many users is Hotmail up to now? Last I heard it was over 40 million....how do you figure the couple of hundred thousand (that's being VERY generous) /. readers are even going to make Hotmail's servers even hiccup? You'd have about as much luck as /.-ing Yahoo...

Netcraft Result

[HoserEh]

I went to Netcraft's site and this was the response back from a request to Hotmail.

www.hotmail.com is running Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b on Windows 2000

I am not one to jump to conclusions but something strange seems to be going on (or is it just me). Unix version of Apache on Windows 2000????

Conspiracy theorists will have a field day with this one.

Re:Gimme a break.

[bad-badtz-maru]

=====
2.) NT (not to mention 2k) can handle just as many hits as Solaris, or any other Unix platform. This has been shown time and time again, but people seem to like to ignore facts and concentrate on a three year old story about poorly written back end code
=====

This is flat out untrue. NT particularly has, time and time again, shown itself to have a feeble TCP stack that buckles under load. I am not talking about the "lets slam it with a zillion connections for 15 minutes" tests. Show me a high trafficked NT box that has been up for longer than 60 days, particularly prior to SP6a. The Microsoft solution is clustering, that way when one of the machines craps out after being up for a week, it can be rebooted without affecting site availability. The NT stack (Win 2K inclusive) is just now, within the last 12 months, starting to achieve acceptable levels of reliability. I guess it's better late than never, but don't act like the reputation is unwarranted.

maru

The Real Problem With Switching

[Pinball Wizard]

It amazes me to see all the comparisons between NT and BSD and between IIS and Apache. I've used all of the above and yes, whatever BSD/Apache can do, so can NT/IIS. You probably need 5X as much processor speed and RAM to do the same things, but Windows & IIS can still pretty much do the same work as BSD and Apache.

The real problem IMHO is that Microsoft has nothing that even remotely compares to Sendmail. Without a world-class SMTP server(and Sendmail is the only one that I know of) I just don't see how they could handle a project of this magnitude.

I know there is a Sendmail for NT, but is it as solid and reliable as the UNIX version? My experience working with SMTP on NT tells me it wouldn't be, because it doesn't integrate with the OS as nicely as it does with UNIX. Correct me if I'm wrong.

Percentages

[SnoopDobbyDobb]

I'm willing to bet that 5-10% of the mail doesn't get through! ;)

The Register have it

[HeUnique]

Here is the Link

Some Real Data: 79.8% Win2K

[waldoj]

I ran the following shell script:

#!/bin/bash
i=1
while [ "$i" -lt 253 ]
do
lynx -head -dump http://lw7fd.law7.hotmail.msn.com/ |grep Server >> /var/tmp/hotmail
let i="$i"+1
done

I got the following results:

• 202 "Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b"
  
• 798 "Microsoft-IIS/5.0"
  

Disclaimer: I know nothing about Microsoft's load-balancing setup, or if I skewed the results in any way as a result of my choice of server. So I reproduce all data here.

-Waldo
-------------------

Slashdot ain't all that hot either.

[DHartung]

I wouldn't be so smug: Slashdot just spent 92 seconds opening this edit page. Before that I waited almost three minutes for the article page with comments to load. And before that, I was surfing in directly to a specific single comment from an external link -- that never opened at all. I finally gave up on that window.

I'm sure Linux/Apache (or whatever you guys are running on over here, I don't follow that gossip) does have an overall stability edge over Redmond product, but NT was never the joke it's made out to be around here and 2000 is even more competitive.
----

This might not be a bad thing...

[AdamHaun]

If it proves that Win2k and BSD can cooperate in the same environment, even temporarily. Think about it. All along we've been trying to convince businesses to introduce Linux/BSD into their computing environments. What better ammunition to use on them than this?

"But boss, Microsoft is doing it..."

"Collapse under load"

[Hard_Code]

Am I reading the right page, because I don't see anything about Hotmail about to collapse under load. Can we please try to stay away from catchy but misleading news titles?

Gimme a break.

[CoolAss]

Let's get some facts straight.

1.) The reason Hotmail crapped out the first time in 1997 on NT was not because it couldn't handle the 10mil + users, but because the software was written in a way that was not happy on NT. In fact, the software was designed by the same people who designed the original back end code for the Solaris version of Hotmail. Basically, they just ported their code, that hardly ever works right.

2.) NT (not to mention 2k) can handle just as many hits as Solaris, or any other Unix platform. This has been shown time and time again, but people seem to like to ignore facts and concentrate on a three year old story about poorly written back end code.

3.) The reason they are doing it step by step (as in not just going, BOOM... all 20mil users on Win2k now,) is for debugging reasons. If a few thousand accounts get screwed, that's much easier to fix than a few million.

There are MANY sites on the net that get far more traffic than hotmail (the MSN homepage for instance) and they handle the load just fine. Doesn't that make you think?

It's not the number of *accounts* that matters, it's the number of simultaneous users.

Slashdot FUD

[Kook9]

There is nothing related here to justify the headline. Pure FUD. I can understand the move on Microsoft's part though -- it's got to stick in their craw that their most successful net service has been running on Unix since day one. I wonder if they expect any benefits (besides marketing) from the "upgrade"?

While I'm on the topic of misleading Win2000 figures, allow me to quote Microsoft's latest full-page newspaper ad:

"When all the numbers are in, we expect Microsoft Windows 2000 Professional to help increase sales-force productivity by about 5%, while reducing IT costs by over 12%."

That means nothing, of course, since the numbers aren't in. Wouldn't expect them to wait, though.

Kook9 out.

Once all the results are in, I expect to be heralded the greatest lover on the planet.

not that it's the best, but....

[Kailden]

From what I have seen, Win2000 is not your father's NT. I've had lots of trouble keeping windows NT running my web apps, but windows 2000 seems more stable. I still have my doubts about it being better than any unix derivative, and so I moving all my code to platform independence, and it will probably end up on AIX (I am trying to get some linux/FreeBSD boxen up and running, but I have to clear off the servers running NT right now. (there are too many other employees who readily jump into the easy but proprietary trap where I work))

Come on, people, this is a Good Thing.

[Chiasmus_]

What better way could there possibly be to test how a product holds up under high stress than to attach it to a giant e-mail network, first attempting to take 5% of the load, and then slowly incrementing it to see if and when it will choke?

And if you don't particularly want to be a beta tester, maybe you shouldn't use a giant, unruly, insecure, slow, free e-mail account as your primary mail provider. Sheesh.

As much as most of us hate Microsoft, this experiment can only do harm to hotmail. It can't really do harm to the software being tested, and it might actually end up improving it.

Why would it go down?

[wadetemp]

The story, as posted, said that Microsoft has moved machines into its load-balancing pool. It made no mention of removing the BSD machines. Adding nodes can only make the system faster, regardless of whether the new nodes are Windows or BSD.

Perhaps they could have made a better choice of OS, for *name your favorite reason here*. But hey, it's Microsoft, and they're in love with thier own stuff! Aren't we all? :)