The Setup Behind Microsoft.com

Toreo asesino writes "Jeff Alexander gives an insight into how Microsoft runs its main sites. Interesting details include having no firewall, having to manage 650 GB of IIS logs every day, and the use of their yet unreleased Windows Server 2008 in a production environment.

Comments

Mostly how they run it

[Anonymous Coward]

is have some crazy sys admins throw chairs around.

Firewall Schmirewall

[mrtroy]

No firewall? Of course not!

Microsoft servers are notorious for their invulnerability.

Re:Firewall Schmirewall

[great_snoopy]

Of course they have a firewall, just watch the difference between a tcptraceroute to a public port (like 80) and tcptraceroute to the same ip but some other port (like 110 pop3 for example). You'll see that packets get dropped at some point indicating a firewall. It's not a RST (port closed) it's just dropping packets for nonpublic services. That is a packet filtering firewall.

Re:Firewall Schmirewall

[rasputin465]

Storing their logs in /dev/nul is the most likely way they deal with 650 GB of logs.

Well geez.. in that case I sure hope they do regular backups of /dev/null! ;-)

Re:Firewall Schmirewall

[AK Marc]

Actually you're wrong. They're blocking ports. Port blocking != firewall.

Ah, the little children. Do you know what the first firewalls were? Routers with access lists. Anything that blocks anything from going to one place from another is a firewall. Port blocking is a firewall, and there exists no firewall I know of that can't be configured to do nothing other than port blocking. You don't have to inspect packets, track flows, or any of those other things to be a firewall, all you have to do is offer some means of restricting traffic. And blocking ports does that.

Re:Firewall Schmirewall

[somersault]

Inventing some binary format is pointless I'm guessing you have no prior experience with Microsoft Office then..

Re:Firewall Schmirewall

[lena_10326]

My question is why are the logs in ASCII text format? When all you want is say the IP [4 bytes], time of day [4 bytes], URI, referrer and return code [do you really care about their browser strings? You are MS after all, just assume it's IE]. Storing an IP as text requires on average 15 bytes, so right there you can shave off 11 bytes with a binary IP. Time of day is worse, a date+time string is like 25 chars. Doesn't seem like much, but multiply the 32 bytes per entry you save by say 50 million hits and that's 1.5Gbyte you saved. That's not counting the white space you can remove, and a simple huffman code you could apply to the URL/referrer.

Logging in fixed format is not more efficient than variable format text files (unless we're talking about transactions but we're not). Let's assume you're logging the basics: IP address, Timestamp, Return code, URI and we'll look at logging in fixed format then variable format.

[abcd] [timestmap] [code] [URI]
4 bytes 8 bytes 1 byte 50 bytes (you actually need 2 bytes for HTTP return code, but let's ignore that)

Every record will require 63 bytes and we'll round up to 64 for proper word alignment). So, if we log 1000 messages, we will consume 64,000 bytes total.

Ok. Now for text logging with space delimiters. We have 3 options below, each requiring slightly less space than the previous. We'll run totals for each.

123.567.890.123 YYYYMMDDHHMMSS x URI...............\n
16 bytes 15 bytes 2 bytes 50 bytes 1 byte

123.567.890.123 1197572382 x URI...............\n (UNIX time)
16 bytes 11 bytes 2 bytes 50 bytes 1 byte

1235678901231197572382xURI...............\n (UNIX time)
12 bytes 10 bytes 1 bytes 50 bytes 1 byte

16 + 15 + 2 + 50 + 1 = 84 bytes * 1000 = 84,000 bytes
16 + 11 + 2 + 50 + 1 = 80 bytes * 1000 = 80,000 bytes
12 + 10 + 1 + 50 + 1 = 74 bytes * 1000 = 74,000 bytes

Wow. Fixed binary format kicks variable text format's ass. Wrong. This assumes the URI (or message) block will always occupy 50 bytes. It will not. Let's go right down the middle and assume it averages 25 bytes and we'll recalculate.

16 + 15 + 2 + 25 + 1 = 59 bytes * 1000 = 59,000 bytes
16 + 11 + 2 + 25 + 1 = 55 bytes * 1000 = 55,000 bytes
12 + 10 + 1 + 25 + 1 = 49 bytes * 1000 = 49,000 bytes

Variable text format almost always beats fixed binary format for logging. That's why Microsoft (and the rest of the world) stores log files as text. Plus, it's far easier to manage and debug when you can slice and dice the files with standard command line tools.

One more thing. I know what you might be thinking. We're logging URLS, which will probably consume the majority of the 50 byte allotment. Most developers will calculate an average width size and double it, so no matter what we'll still be filling about 50% of the message section.

Last point. If I were to use your example, the savings with text logging would even be greater. 2 URLS would be stored, both consuming about 50% of their data block. IP address, timestamp, URI, Referrer URI, Return Code. There's also a bunch of other little optimizations you can do such as storing the domain, year, month, and day in the filename rather than in the data or dropping the least significant byte in the HTTP return code.

Re:Beta in production environment.

[EvanED]

Vista was never meant as a server. Same as XP isn't used as a server, it's Server 2003.

Eating dogfood is good

[ReallyEvilCanine]

How can anyone complain that they're running Server 2008? My company's software quality dropped considerably when we stopped eating our own dogfood two years ago. When techs, engineers and everyone else is stuck with the same problems as the future ell-users, shit gets fixed a lot faster and a lot better.

Re:Supporting

[plague3106]

How many times have you seen the microsoft.com website down / hacked?

Re:Beta in production environment.

[schnikies79]

Funny, but you're wrong. Pro is for networking enviorments where you need RDP, policies, ability to join a domain, file encryption, etc. Home lacks these.

Re:Beta in production environment.

[JCSoRocks]

Tis a sad day when the fanbois can't even get their insults right. shameful.

Swimming in acronym soup...

[thatseattleguy]

Could someone with more Microsoft Kool-Aid in their veins stick their fork in the acronym salad that is this article? ACL (Access Control Lists - which technically are a firewall), DoS (denial of service attacks) and IPS (intrusion protection services) I all know, but WTF are:

HBI?
GFS (is the G for "Ghost")?
NBI?
NLB?
ACE?

TIA :),
/tsg/

Re:Swimming in acronym soup...

[Anonymous Coward]

GFS: Global Foundation Services. Microsoft's big internal network management thing. It's the people who keep the servers up and running for everything facing outward.

HBI: High Business Impact. Social Security numbers ,Passport accounts, etc.

NLB: Network Load Balancer.

AV: AntiVirus.
DoS: Denial of Service
IIS: Internet Information Services. 'httpd' for Windows.

Perhaps the only ones who can do it "right"

[teebob21]

Let's set aside the natural urge to bash MS into oblivion. Let's (just for now) ignore conventional advice about network security and firewall use. Now, not only are these guys a Microsoft shop...they ARE Microsoft. MS claims their software is stable and secure. Perhaps it is -- when was the last time microsoft.com was taken down by malevolent hackers?

That said, with their closed source and closed-doors policy to revealing details about the inner workings of the OS, _Microsoft_ may be the only company that can successfully deploy a 100% Microsoft powered solution. How many registry changes, service daemon modifications, and other tweaks have been made to get their config running this way? The world may never know. It's probably impossible for the consumer world to ever have that level on knowledge about the Windows environment, and thus run it at peak security levels. For most consumers and businesses, a Linux OS with properly implemented firewalls is much more secure than an out-of-the-box Windows deployment and router ACLs.

Re:Beta in production environment.

[vtscott]

And of course it's already been modded up (at least only as funny). To clarify why the GP is wrong, from the wikipedia entry on Windows server 2008:

Windows Server 2008 introduces most of the new features from Windows Vista to Windows Server. This is a similar relationship to that between Windows Server 2003 and Windows XP.

Gotta give credit to MS for eating their own dog food...

Allow incoming connection on port 80? Confirm/deny

But generally..

[Junta]

Router ACLs are in place to block unnecessary ports
Cisco Guards for DoS detection and automated response In other words, they don't use firewalling where you have administrator defined rules to control traffic flow, they use networking equipment that accept administrator defined rules to control traffic flow .... totally different..

What in the world do *you* perceive the difference being between a 'firewall' and a router blocking ports based on source and destination being compared with a set of rules (aka ACLs)? Generally, firewall rules *can* get more complex than that, but mere port blocking by an intermediate router has been considered a firewall, even if it doesn't log violating or accepted packets, even if it doesn't have complex rules about connection state. Even if it doesn't have the word 'firewall' emblazened on the chassis somewhere.

Re:Microsoft brainwashing

[jez9999]

Uh, didn't I read an article not too long ago about how the update.microsoft.com site was broken into?

Link, please? http://update.microsoft.com/

Re:Beta in production environment.

[misleb]

Dude, if you can't hack that right now, how are you dealing with unix instead?

Because at least Unix has conventions.

If any platform's based on a standard of bizarre naming due to space saving stupidity, that's it.

Really? Ok, lets open up C:\Windows on one of our Windows servers. Hmmm a folder named "$hf_mig$". I suppose you know what that means or what convention that follows? Or C:\Windows\adam. Kinda looks like it might be some directory tools. Maybe ADAM = Active Directory AdMinistration? What's that doing there anyway? I could keep going down the list. I suppose there is a very good reason why there are .BMP files in C:\Windows? Desktop wallpapers? Come on. I wonder if they're related the other brilliantly named files such as SET2.tmp and SET3.tmp in that same directory. And don't get me started on the insanity that is C:\Windows\System32. Hardly a single file/folder that doesn't use 8.3 naming. I haven't clue what have that stuff is doing there.

Infact, name any mature platform that's based on reasonable standards for it's underlying API's and structure.

First of all, I was only talking about superficial organization. And if you want to see something nice, have a look at OS X some time. Not only is the System (/System) well organized, but most applications are neatly self contained in /Applications/Some.app. They usually don't spew files all over the place when installed. You know where the term DLL Hell comes from, don't you?

Didn't think you could. While it's true that things like the FHS are helping on the unix side, try telling an oldschool developer like oracle that they need to follow it. They'll laugh. and laugh.

I could give fuck-all what Oracle thinks. My Debian systems are very well organized, thank you very much. I don't find desktop wallpapers in /usr/lib. I don't find temporary files for applications in /usr/bin. FreeBSD is even cleaner. The system files never change unless I explicitly do an upgrade. All supplementary software (ports, mostly) goes in /usr/local. With Windows, on the other hand, who knows what strange and wonderful new files I might find dumped in C:\Windows tomorrow. Maybe $hf_mig2$. WHich would be version 2.0 of whtever that is, i guess.

-matthew