Slammer Worm Slams Microsofts Own

MondoMor writes "Microsoft's forgot to patch some of its own servers to protect it from the months-old vulnerability exploited by the Slammer Worm, reports C|Net. Oops. Apparently Redmond's network was hit pretty hard. Just goes to show that no matter who you are, you'd better keep your apps patched." Update: 01/29 01:59 GMT by T : And if you're running systems which might be affected, take note: whitehorse writes "The Microsoft KB article for the Slammer patch found here has an incorrect URL for 'Download the patch' referring to KB Q316333 which is only a handle leak fix. The real patch may be found later in the article."

Comments

Microsoft didn't patch all their INTERNAL servers

[wbm6k]

The article I read (on yahoo) states the unpatched servers were all on the internal network, not the internet, and that they were in use by researchers within microsoft.

Let's not jump too quickly on the bash microsoft bandwagon for that. (Of course, if they just did enough testing and didn't release buggy, vulnerable software in the first place...)

Re:Zoiks!

[Anonymous Coward]

automatic update doesn't work with SQL server. you have to do patches the "old" way (unzipping files, renaming files, prayer), which is probably why so many novice admins never applied the patches.

Re:I wonder how long...

[jamesdood]

The thing to remember is this worm infected any machine running the MSDE (A scaled down MS-SQL server) So if you were running Access or Office 2000 or MS Visual Studio 6, or even Visio 2000 you could be affected by this. Most end users don't even know that they would be vulnerable and the statement "This particular worm largely ignored home and personal computers, due to the product it infects" is false. It also seems to have had an effect on certain Cisco routers. Not fun but you can't just blame "Poor Admins" as the culprits for the virualance of the worm.

Re:Zoiks!

[questionlp]

Please mod this parent up.

There isn't only no way to get SQL Server patches from Windows Update, but (as the parent mentioned), the steps required to update SQL Server and the Desktop Engine (MSDE) is a royal bitch and some.

For example, to apply any hotfixes or cumulative patches for SQL Server 2000, you must download the package, extract it, backup the SQL Server install directory and databases, manually copy over DLL files and other updated binaries, execute the SQL query files included in the patch (one at a time, in a certain order... MSDE users need to use the command line interface for it since there is no GUI provided), then pray that everything is okay and start SQL back up.

Worm's damage surprises experts -- takes out ATMs

[aengblom]

My rejected submission -- more details, but a bit long. The big news in my mind was not the microsoft bit--it was that ATM machines were unavilable because of the worm.

~~~

The worm that slowed the internet to a crawl over the weekend apparently did more damage than most originally believed. On Monday, many companies were still struggling to clean up. Financial companies and airlines seemed to be hit most acutely. Many web sites that manage payments and check loans were inaccessible. Inexplicably--and really inexcusably--some ATMS were also unavailable. Investigators are also struggling to pinpoint the worms starting point, but are having little success because it took off so fast.

Apparently similar code was released by David Litchfield of NGS Software Inc a few months ago. Virus "author," "Lion" credited Litchfield's code.

The Washington Post has an AP story up as well as this, which is older but has some additional details. The kicker to all this--the worm hit one year after Microsoft launched its "Trustworthy Computing." That and even some of Microsoft's own computers were hit (NYT Reg. Req.).

(Yep, still bitter ;-) )

Re:Despite what the apologists say

[thebatlab]

> Want to find a list of needed patches for a Microsoft product? Hope you have a few days for searching the endless volumes of technet or msdn-- hope you find everything.

I usually just go here rather than searching endless volumes....

http://www.microsoft.com/technet/treeview/default. asp?url=/technet/security/current.asp

Re:Big Surprise?

[auferstehung]

MS was able to draw on (and some would say corrupt) the smart work of their research folks and of technologies that they acquired and "MS all over it" until it fit their sales and support model, which is one of the reasons that they could do something like go from "Internet-illiterate" to winning the browser war, practically overnight.

From about Internet Explorer: "Based on NCSA Mosaic. NCSA Mosaic(TM); was developed at the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign." Microsofts "smart work" was assimilating open-sourced non-GPL code to become internet literate. Explains why they dislike the GPL. It puts a damper on their research and innovation.

Re:Say what?

[Hektor_Troy]

Well, SQL Slammer is only a TSR-worm and isn't stored anywhere but memory, so those who actually turn off their laptops as opposed to suspend or hibernate wouldn't be carriers.

Of course - it only takes one infected client ...

Zero defects impossible; fix the fences instead

[satch89450]

Zero defects is not an attainable goal; it's too expensive and no one wants to pay for it.

This article shows just what happens when you expect zero defects in the infrastructure of a large organization like Microsoft Corporation. It's not going to happen. And before someone says I'm Microsoft-bashing, I will say that this is true for the vast majority of corporations, universities, foundations, and governments. That would include Sun, IBM, Red Hat, even the *BSD folks and LKML participants.

There is a damn good reason we won't see zero defects: employees are not measured by it. Their survival, pay raises, and promotions are based not on the number of defects they don't have, but on their contribution to the "bottom line." If you preach zero defects as Job One, then prove it by firing the people who generate defects, without exception -- including the CEO, COO, CFO, CIO, and other top brass, when they screw up.

So now that the myth of zero defects has been exposed for what it is, what do we do about it?

1. System administrators are going to have to re-think their perimeter access controls. This may require router upgrades to add processing power to support additional filtering.

2. Sysadmins who have been running "mostly-open" filter configurations may want to consider moving to a "mostly-closed" configuration: deny everything except services that have been cleared for use. Don't allow arbitrary connections. Many unknowing MS SQL servers were protected from participating in this little exercise because the firewall upstream of the desktop system wasn't allowing connections to get through, even if the desktop system had a globally-routed Internet address.

3. Computer mail order houses and computer stores should consider carefully whether they should bundle appropriate software firewall products with the computers they sell. Software configured to require the user to say "Yes, I want to make SQL server available for public access!" before 1433 and 1434 would be open.

4. We need to ask the reporters and editors of mainstream publications to be more responsible when reporting problems like Sapphire/SQL. The facts were pretty well known, and available to those who tried hard enough to get them even at the height of the packet storm, so that reporters could make their deadlines and get the facts straight. [Names of the guilty withheld, at least for now -- they know who they are.]

5. Tier 1 and Tier 2 bandwidth providers need to consider modifications to their Acceptable Use Policies to require some basic filtering of packets in both directions. These AUP changes have been discussed before; perhaps now is the time for them to go into effect:

   • Upstream packet source addresses must be verified at the perimeter such that the packet's return address points to a host in the network, and not to a random IP address or to broadcast addresses
   • Downstream packet destination addresses must be verified at the perimeter such that the packet is directed to a single host in the network, and not to a random IP address or broadcast address (other than multicast addresses, if such are allowed in the network)
   • As one drills down the levels of networks, packet source/destination verification must be done at all levels -- no exceptions (the excuse "It costs too much" doesn't wash when you consider that suitable packet filter technologies are available in both *BSD and Linux flavors, running on hardware that costs less than your standard business power lunch for four)
   • "Small services" (TCP 0:19 and UDP 0:19) must be blocked at the perimeter, both as source and destination ports.
   • A small number of other, specific ports must be blocked at the perimeter, those ports being identified as services that are intranet in nature instead of "global" services. The specific ports to be blocked should be determined co-operatively to avoid denying essential services to customers.
   • Encourate the use of VPN for interaction between two separated locations needing the above-mentioned intranet services over the Internet.
   • Encourage the use of abuse-prevention methods such as Network Address Translation on all circuits [cable operators take note] to block access to those systems that are NOT intended to be servers.

6. Update the Best Practices RFCs to incorporate some or all of these suggestions, so that Internet operators around the world can participate in solving the problem.

(N.B.: I want to point out that many USA-based cable operators are contributing to the problem by disallowing the use of NAT and VPN technologies in their apparent [alledged] quest to limit the broadband "Internet service product" to browsing and downloading files. I believe that such an attitude contributed to the problem, not the solution. I understand well the technical and business motivations for this, but I also believe that there are (U.S.) national security implications against such a policy. THINK!)

Are any of these ideas new? NO. The only new idea is to have the Lords Of The Internet use their influence over their customers to implement them more widely.

Good fences make good neighbors. The Internet is a neighborhood.

Re:Say what?

[br0ck]

Rebooting clears the worm from memory and don't most people shut their laptop off when they carry it in to work? Actually we quite a few laptops and desktops that needed patching and the users had no idea that they had MSDE or SQL server running. Windows Update doesn't notify these users that they need patches, and these users definitely include the types that would have no idea that they need to track down obscure patches that didn't even, until this weekend, have an install routine. MSDE installs by default with Visual Studio .NET, ASP.NET web matrix tool, Office XP Developer, MSDN subscription, Accesss 2002, Visual FoxPro 7/8 and can be redistributed by vendors. Even worse, MSDE and SQL server install with a blank password, although I think it warns on install now.

Useful script for keeping up-to-date with RPMS.

[caluml]

cd /raid/8.0/updates

wget -nd -nH --mirror --no-parent --passive ftp://ftp.mirror.ac.uk./sites/ftp.redhat.com/pub/r edhat/linux/updates/8.0/en/os/i386/ -o log
wget -nd -nH --mirror --no-parent --passive ftp://ftp.mirror.ac.uk./sites/ftp.redhat.com/pub/r edhat/linux/updates/8.0/en/os/i686/ -a log

saved=`grep saved log | grep -v ".listing"`

check=`rpm -K /raid/8.0/updates/*.rpm | grep -v "md5 gpg OK"`

if [ "$saved" ]
then
mail user1@domain.com user2@domain.com <<EOMAIL
New RedHat 8.0 RPMs downloaded onto `hostname`
Please update them:

$saved

$check

If there are any kernel updates, please run lilo before rebooting

EOMAIL
fi

Run this in the night some time.
When you come in, if you've got an email, run:
cd /raid/8.0/updates
rpm --freshen -vah *.i686.rpm
rpm --freshen -vah *.i386.rpm

Hey presto. Job done. And if you use Grub, you don't have to bother about running lilo.

Actually No

[ravenmoon]

Microsoft incorrectly states in bulletin MS02-061 that SQL Server 7.0 and MSDE 1.0 are also affected by the worm.

While troubleshooting an issue related to the patch w/ MS phone support, the technician told me that 7.0 is not affected and the bulletin was incorrect.

It is entirely possible he was misinformed though.

Re:SQL 7 is *not* succeptable to Slapper

[the-matt-mobile]

SQL 7 is *not* succeptable to this vulnerability. SQL 7 doesn't use port 1434 for anything. That's new in SQL 2000. However, 7.0 is vulnerable to plenty of other things.

Re:hmmm...security + patch administration

[painehope]

most of the few hundred lines of code is for a system that fetches the patches from a site that maintains a redhat mirror ( site selection by ping time ) to a local patch server, and does the comparison and selection ( w/ a somewhat optimized algorithm ), then moves old patches out of patch tree. It only took about 60 for the updating stuff. It's what I use on workstations. For servers and clusters, yes, I administer those more directly ( select an appropriate patch, drop it in a patch directory, and it is installed by autorpm ). Perhaps I should have clarified that. But I'm generally not too worried about most linux patches, as I can be sure that the patch I install for a utility or a desktop app isn't going to include something that break mysql, apache, etc. And you can use some of these package management packages ( no pun intended ) on most Unices, for example, RPM on AIX. Most of my admin experience is with Linux, but I've done some AIX + Solaris, and I'll agree that the default methods are a bitch, but they can be made more efficient and simple. And Linux is drop-dead easy to administer on workstations, pretty easy on servers, and very easy on clusters.
As to why I don't use up2date, it's because it chews up to much bandwidth ( ea. workstation hits redhat patch server ). I looked at using Ximian's stuff, but management shot it down.
And if administering w2k boxes is so damn easy, how come very few admins do it right? Proportionally quite a good deal less than their Unix counterparts? I can't verify the comments you made about w2k package management, as I quit doing windows around NT4.0/98 days, but I'll take your word on it. BTW, can this be remotely automated? But, still, how come everyone whines about how hard it is? And how come a frequent complaint ( and one I've seen w/ my site's sExchange mail servers ) is that one patch/update breaks other services/functionality?

That's the SP, not the patch!

[burgburgburg]

The service pack was released January 17, 2003, a week before Slammer hit. The patch, which does require all of that manual effort is what has been out there for the past six months.

Alot of sysadmins were waiting for the SP to be released before even approaching this one, just because the patching process is so complex. They just waited a week too long

My Experience

[0xA]

Ive read that the patch before this thing went big was a bitch. Basically it was a lot of manual this and that updating and rebooting. Basically this meant a lot of people couldnt get aproval from management to patch the server.

It's not that bad really, I think later versions of the patch even included a batch file to copy stuff around for you. Even without it, it only took 10 minutes... I mean really, if somebody can't handle this kind of stuff should they really be an admin?

Some have said they applied the patch and still were vunerable.

Yeah you have to be careful with this stuff, if you apply patches in the wrong order you can sometimesend up with the vulnerable code still there. I know a _really_ good admin that got hit with Code Red because of that. The correct order can sometimes be a bit of a mystery.

Some have said the patch fucked their server.

That's the big problem with this situation. I can understand why people don't have this patch or SP3 installed. You really never know what one of these things is going to do. It is common for amins to schedule a 3 hour downtime to roll something like this in, even if they have tested the hell out of it. You need time to get the damn thing back out if it screws stuff up. I deployed W2K SP3 onto my terminal servers a few months ago and it broke Office on every one of them. It didn't do that when I tested it, took me hours to clean it up.

Re:Problem is IPv4

[trybywrench]

actually what made this especially bad was UDP. Not many programs run on UDP ports almost always they are TCP. TCP has a VERY important feature and that is upon a non-ack'd window it throttles back the send rate. This is a way to get congestion feedback to a host and tell it to "settle down". The problem with UDP is there is no way to tell it to slow down. Also, the fact that the Internet is a "best effort" network means that no matter what the UDP incoming rate the routers will do their best to deliver the packets. This comes at the expense of all other traffic flows in the router, no way to get congestion feedback to the host means no way to limit the incoming rate. Even if the routers just dropped the packets that still increases CPU and RAM utilization and with the volume that was happening would still probably bring traffic throughput to a trickle.