Slashdot Mirror
Security Issues with Windows 2000 Datacenter?
"My company is currently looking to cluster our SQL 7 servers. We're
considering Win2000 advanced server or datacenter. Around a month ago I sat in a meeting with our VP of IT, and the rest of the network admins I work with. Compaq tried to pitch their Windows 2000 Datacenter or Advanced Server solution. Here is the way the compaq people explained it:
You get datacenter only from an OEM. They look at the apps you're running
and customize a solution for you in their lab. Every datacenter implementation is different, and every datacenter CD is different. Since we would be using an EMC SAN as our clustered storage system they said our implementation would take special customization. They would have to contact EMC engineers and work together. Once you deploy it, the OEM monitors it. And you can't install any service packs or anything without getting an OK from your OEM. Any service packs are customized for your enviroment. The SLA guarantees a 99.999% uptime or your money back. Part of your money at least. Datacenter isn't an OS, but a program in their words.
Now here is the problem. With Code Red and Nimda, how do you patch IIS
running on datacenter in a timely manner? The reason IIS servers became
infected was because the admins didn't patch them in the first place. So say
a new worm comes out in a few months and it takes a few days for MS to
create a hotfix. Datacenter admins can't install it until they get their
customized copy from their OEM. And almost every 2000 server runs IIS for
terminal server. It can take a few days and in the meantime your servers
could be down. And I don't see the SLA covering a situation like this. Meanwhile you're explaining to your CEO how this $500K supposedly guaranteed solution is sitting dead in the water and you can't do a thing about.
Is there something I'm missing, or did Microsoft look over something like
this? Especially when they are trying to push Datacenter as 'Big Iron'."
Ask the vendor to modify the SLA to specifically cover the contingency of exploits and how they will be dealt with. Your vendor might try to claim that the 99.9999 uptime would cover this, but I'd counter that a server which is up but exploited is useless.
No one ever had to evacuate a city because the solar panels broke!
First of all if your company is wealthy enough to be using Datacenter as a web server I hope they are paying you a decent salary. :)
Its a waste to use Datacenter as a web server or front end machine for applications, its best use is for big honking SQL applications like MS SQL server. Datacenter is a waste for Oracle/NT because Oracle on NT is the worst implementation of Oracle in existence. If you want a big honking box to do oracle for gods sake get a Solaris/HPUX/AIX monster. Big ass database servers should never be directly exposed to the internet anyways, the connectivity should be happening thru a balls to the wall firewall.
"And almost every 2000 server runs IIS for terminal server"
Erm, I work for a Citrix Gold partner and I've never encountered this before. Installing Terminal Server does not require IIS.
In fact, according to M$ recommendations, you should minimise the services running on the TS box.....That means no IIS.
Also, the "smaller but more servers vs fewer 8 way servers" for TS debate has been done and dusted, and the recommendation certainly isn't for having fewer large servers. The "sweet spot" is a farm of dual processor servers with 1.5Gb of RAM, thus you wouldn't need Data Center anyway - normal W2k Server would be more than adequate.
"Mary had a crypto key, she kept it in escrow, and everything that Mary said, the Feds were sure to know."
Straight from http://www.microsoft.com/windows2000/datacenter/ev aluation/business/overview/default.asp:
From http://www.microsoft.com/windows2000/advancedserv
Other pieces of information not listed in that blurb about AS: supports up to 8-way SMP and 8 GB of RAM (compared to DC's 32-way and 64GB).
You're obviously not going to have a DataCenter machine sitting underneath your desk at work, but it's quite possible to do so with Advanced Server.
Keep these SQL apps behind the firewall.. turn off all IIS features on the sql boxes.. and at least Nimda should not be able to get at it. Any web interface would hopefully not use Datacenter, and use standard Advanced Server, which is easily patchable. If sql was available on the front line, well, they almost deserve it.
-=-Ze End-=-
If you aren't allowed to patch your server, then you should isolate it behind a firewall of some sort, so that the chances of infection are minimized. This may not work well for IIS (beyond simply not running it), but it will serve you well in the general case.
Reboot macht Frei.
Is a locked down version of Windows. What happens when you lock it down? Well, intensive testing occurs first to determine what is being done with the box and what possible problems could arrise. Then those problems are solved. Also, only certain applications are certified to run on a datacenter box. The goal here is to achieve five nines. That is have this box up and running for 99.999% of the year. Without thorough testing of applications this level of availability would be impossible.
Part of what you get with a Datacenter purchase is a premier level of support. This includes a named engineer for support, and automatic escalation to the highest level for any support needs. It also includes any updates and or fixes on a priority basis - if you have a Datacenter server you get patches, updates, etc. before anyone else does.
Get the vendor to patch your servers within 12 hours of Microsoft issuing a hotfix/patch. If they will not put that into the contract, tell them they're not professional enough. If they cannot do something as easy as that, would you really want them running truly business critical solutions for you?
Stop the brainwash
Because. if you do *anything* not certified by the vendor, the 99.999% agreement is void, and they are not responsible for downtime.
Datacenter is more of a custom solution package than a version of windows. Yes.. it's a version of windows 2000.. but it's really a whole package.
In other words, it's a version of windows used by vendors to create huge custom solutions, usually for databases.
Then you can negotiate all the details. And remember: 99.999% uptime does not mean that your server stay up that long, but that you have only an unscheduled downtime of 0.001% or less. Applying a patch is, in nearly every case, a scheduled downtime and does not count.
Now imagine you really, really need this patch: you can urge your OEM to install it and keep him free from all responsibility (e. g. a server crash after this does not count to the unscheduled downtime, because it was your decision to apply it). If you trust him to play fair, that's fair for both of you.If the OEM is trustworthy, he'll do what you order him to do, but in that case you will be responsible for the outcome as well.You can't burden someone with responsibility if he can't make the decision (unless you don't play fair).
You found a sword: +4 damage, +5 moderator points
I can't find any info on MS's site right now, but I'm sure that OEMs that supply W2k datacenter are required to have a support team ONSITE at MS's campus 24/7.
;)
This article raises a very good point, but Microsoft's idea behind datacenter was they hat total control over the hardware environment, and they made sure OEMs would stand behind it too, so I'd be very surprised (and dissapointed) if the OEM didn't contact their customers *immediately* with patches whenever there was a hole (and I'd guess they are pretty busy too
I can see you haven't worked with Microsoft software very much, so I'll give you the solution: Reinstall your machine.
It's *just that simple*, can you believe it? Every time Nimda hits your machine, just wipe out the system drives, reformat and re-install! Easy, right? Sure you may have to reinstall 40 or 50 times a day, but again, if you are familiar with M$ software, you'll know you need tons of backup machines that you can swap out as needed with your infected machines. Make an assembly line of it. Have one guy reformatting, another guy reinstalling and a third guy disconnecting the infected boxes and plugging the fresh machines into the network!
Now, where do you want to go today?
"Your superior intellect is no match for our puny weapons!"
This is commonly refered to as the Mainframe Mentality: these systems are so critical to a business, you don't make any changes to them unless these changes are a. absolutely critical and b. have been tested extensively in the exact configuration you'll be running them.
Now, it may seem that this would cause every Windows 2000 Datacenter server to be instantly infected with Code Red and friends, but in reality this will not be the case, because:
1. You don't expose your Datacenter servers to the Internet -- never. No matter if you're running Microsoft, AIX, Solaris or Linux: only trusted systems should have strict "need to know" access to your server;
2. Datacenter-type servers typically don't run HTTP servers. You would scale out HTTPDs (more boxes), not scale them up (bigger boxes). Also see rule 1;
3. The config of your Datacenter server is the bare minimum. So, in the case of Windows 2000, you would not ever run IIS or Index Server (the true culprit in case Code Red et al...) on it, just your database server and perhaps your business logic (although that, again, tends to scale out better than it scales up).
In summary: security hotfixes and Datacenter-type environments tend to be mutually exclusive. If you need a patch to your Datacenter server, it pretty much needs to be custom-developed for you. Fortunately, since Datacenter setups are not typically designed by the clueless individuals that gave Code Red free reign, this tends not to be an issue in real life.
If anyone out there is running Win2k Datacenter, I've got an important question I've been trying to find the answer to, with no luck so far. Can someone finally give me an answer? The question is this:
Does Windows 2000 Datacenter ship with 3-D Pinball installed by default? If so, is it in the Start menu?
That's all. Thanks.
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
Comment removed based on user account deletion
Nimda did go behind firewalls. It came in via e-mail or external consultants with laptops that attached to the LAN, and then attacked all intranet servers. As the story says, IIS is used for administering these servers, so they are indeed in a very vulnerable position and need to be patched.
"I don't know of one bank that uses a non-IIS platform."
You need to look harder then. The first 5 banks I could be bothered to look at:
"Mary had a crypto key, she kept it in escrow, and everything that Mary said, the Feds were sure to know."
Since you're paying microsoft a shitload of money, I'm sure that something can be worked out. All the friggin losers who were hitting my box with (a la Code Red) were on DSL / @home lines.
Incidentally, the iis vunerability was known since iis 4.0 was released. It was kept secret by MS because of the "If no one knows about it, no one will exploit it". I'm thinking the data center people get the patches that home users don't - sort of like netware's support, there is a $200 per support issue, but they will forward the problem all the way up to the guy who coded the section you are having a problem with.
The lame fuck of the day is 24.202.127.156
1q2w3e4r5t6y7u8i9o0pqawsedrftgthyjukilo;p'azsxdcf
Specs are hard to write and all vendors have weasel clauses. Just look at insurance policies - damage due to acts of war are generally excluded. With cracking being described as a "terrorist act" you could end up with exploits not being covered.
A big common exclusion is "unscheduled" downtime. One of our vendors would see a router or firewall machine starting to act funny and then quickly "schedule" some emergency downtime that night to reboot it thus avoiding having to pay.
I have not had good experience with outsourcing - never forget that these are the same bunch of folks who are getting skewered for lousy tech support for poor end-users who have paid extra for support packages. Attitudes don't change much across corporations.
Before I would spend the bucks for any sort of "managed services" I would make sure that the vendor guaranteed 100% availibility without exception. Availibility must be defined as a maximum latency (ie. no end user will wait more than 750ms for a response or whatever is needed).
Rationale? Any app that requires this type of support must be available to the end user without fail. That's why you pay the bucks.
OS is "up" but web server is compromised or down? It's no good to the user. The downtime was scheduled? End user doesn't care.
Why 100%? Why not. They are already guaranteeing less than 316 seconds per year of downtime. Let them work their payments for that downtime into the contract cost. I don't want to have to total up downtime and argue over when the year started. I want the vendor to know that any downtime costs them bucks. No argument, no weasel clauses, no exceptions (better keep those machines maintained, protected and patched).
Been there - been burned. We moved our servers from a "managed solution provider" to a generic server farm and got far better service for one tenth the cost.
Put the datacenter server behind a firewall, preferably with some string matching functionality (ie watchdog).
/default.ida, filtering on global.asa is also a good idea ;-) etc ..
the later iptables have a string-patch included, which allow you to target certain port/string combo's, with this it is easy to block worms from the webserver, as long as you know what request it makes.
exampple to block cmd.exe access (taken from my own internal firewall scripts, this will block nimda)
$IPTABLES -A INPUT -p tcp -i ! $INTERNAL --dport 80 -m limit \
--limit $LIMITLEVEL -m string --string "/cmd.exe" \
-m state --state ESTABLISHED -j LOG \
--log-level $LOGLEVEL \
--log-prefix "MS IIS cmd.exe usage:"
$IPTABLES -A INPUT -p tcp -o ! $INTERNAL --dport 80 -m limit \
--limit $LIMITLEVEL -m string --string "/cmd.exe" \
m state --state ESTABLISHED -j LOG \
--log-level $LOGLEVEL \
--log-prefix "MS IIS cmd.exe usage:"
$IPTABLES -A INPUT -p tcp -i ! $INTERNAL --dport 80 -m string \
--string "/cmd.exe" -m state --state ESTABLISHED\
-j REJECT --reject-with tcp-reset
$IPTABLES -A INPUT -p tcp -o ! $INTERNAL --dport 80 -m string \
--string "/cmd.exe" -m state --state ESTABLISHED\
-j REJECT --reject-with tcp-reset
If you wanted to block codered, filter on
(see iptables docs for more info)
G'luck
This is an odd question because both code red and nimda were actually viruses that took advantage of things like directory traversal and admin tools on the system. In short most admins already knew about these issues and fixed them themseleves by disabling the dir traversing and removing the template site.
So in short to answer your question when it comes to code red or nimda you really should not have a problem if you are a good admin. The same is true in the linux world and newbie web programmers that do things like system calls without checking out what is going to be called. If you call something that the users passes to you then obviously they can do things like tracrt ip; rm -rf / and your code would let it. This is not perls fault or php's fault or any other languages fault it is the programmers fault.
As much as I dislike windows, mainly because I have been an asp programmer for a long time and I would rather use linux and do perl programming (which I do now), Microsoft is somewhat right in that a knowledgable sysadmin already had the holes fixed. At the same time they should not send out software with issues like that.
--MD--
If you follow the Linux kernel development, and read around, you'd notice that scaling to a 2-way or 4-way machine is a big leap in performance. Throw Linux or any other OS on a 6-way or 8-way machine and you will watch that increase in performance degrade (ie a 2-way machine isnt x 2 the performance of a single CPU machine, and an 8-way system isnt x 2 the performance of a 4-way machine).
This, of course, is crap. To say that "any other OS" has the same scalability problem that Linux has is simply not true.
Take IRIX, for instance. I wrote some image processing code that runs on Origin servers. The 8-processor server in my lab runs my code about four times faster than my 2-p servers. And, surprise, the 32-p server in my friend's lab runs my code about four times faster than my 8-p machine.
To generalize the problems you see on Linux and Windows to "any other" operating system is simply hogwash. Your point about Windows scalability is well taken, though.
I actually posted this question twice, and I'm glad they used this second posting with our actuall situation. The first one was more of a what if scenario.
As far as terminal server and IIS, you need IIS if you want to use the Terminal Server Advanced Client and go in through the web. I was originally taught to use TS through IE and forgot going in through the TS client.
If we do go with Datacenter, the servers will host SQL 2000 Enterprise in a clustered enviroment. We currently use SQL and have a propritery in house written app for it.
And as far as the Code Red holes being found months prior to infection, I just used this as an example. I remember in 1997 and 1998 NT had new security holes every week. Windows 2000 is slightly better. 6 months ago I remember downloading hotfixes that will appear in service pack 3.
My question still remains, if a new flaw in IIS, the kernel or any other part of the OS is found how long are we supposed to wait for a fix? I forgot the specifics, but I'm pretty sure the compaq people said they customize the source code for your enviroment. They will need a copy of our in-house app, get in touch with the EMC engineers because our EMC box will be our clustered storage and analyze everything else. Then we will get a CD with a customized copy of Windows 2000 Datacenter. Like EMC, the servers will be monitored by another company and they will most likely know of any problems before us. Every so often we will get a new CD with updates, service packs, etc customized for us. But if a new worm comes out in a few months that exploits some currently unknown flaw in Win2000 or any other part of the OS, will we be dead in the water while we wait for a patch? After September 11th we were calling EMC for tech support on our Symetrix and we were basically told get in line. They had richer customers to support first.
Comment removed based on user account deletion
Both Nimda and Code Red can be avoided by locking down the IIS 5 configuration (... as demonstrated by the MS IIS lockdown tool). No patches (not even OS service packs, i.e. no Win 2k SP1 or SP2) are required! If you add some firewalls in front of your IIS, one of those being e.g. ISA Server 2k, you could use - HTTP forward caching (where all cached requests would be handled on the "other" side of the NAT firewall) - content filtering (to block offensive code such as Nimda). If your admin knows her job, everything should be just fine with your Win 2k Datacenter (except for the noise those boxes tend to make) ...
M.
The issues mentioned in this article are null & void, as a situation like that would most likely never, ever happen. (Then again, you picked Compaq as your OEM, so maybe...*insert rim shot here*)
So say a new worm comes out in a few months and it takes a few days for MS to create a hotfix.
Is there something I'm missing?
Absolutely. You've got your timelines backwards.
Worms come out a few months after the bugs have been discovered and patches have been made available. We're talking months here. Code Red came out more than 2 months after the bug had been discovered and patches created.
Microsoft has had their patches out in the wild within a few days of a major bug being discovered. The worms however take much longer to be created/deployed/spread. Although it is possible for the worms to come out much faster, they will still be lagged behind the discovery of the bug, and the patches are issued almost immediately.
And if you have an agreement with your provider that you will have 99.999% uptime, then you better believe that they will be phoning you at 2am in the morning to tell you that they're coming over to install a new patch lest they break their contract.
If God gave us curiosity
This may not be modded up high enough for the +4 folks to see it, but I have to say that the people posting at +4 and above have some really great comments.
It's nice to see Slashdot as a technical community, not just a Linux one. I know, I know, *nix is the preferred OS of many of the readers/posters, but it's nice to see such an array of comments and extremely constructive ideas and comments. Nice Comments, all.