Slashdot Mirror
Universities Taken Offline to Fight Worms, Viruses
chrismg2003 writes "Nationwide universities are opening their doors to new students but closing off their network services. The Blaster worm has caused universities to take drastic actions to protect their campus networks. Universities have gone as far as shutting down their entire resnet network and bringing it back up dorm-by-dorm after each computer has been certified worm-free. The ICMP ping requests alone have brought down my university's resnet multiple times and we are scrambling to clean the worm from all computers before it forces us to follow suit with other universities."
Can we get the ISPs to do this too? It'd be really great if they'd just turn off a tiny manageable chunk of infected users and wait for them to call support. Support could then tell them to patch, or upgrade, or get some other type of clue. A really with-it ISP could just replace the web page the user wanted with a page that tells them to get with it.
Problem is, any plan will cost money to support. Worse, it might prompt the users to just cancel their service. I can't imagine ISPs like that idea. At least with the universities, the students have no choice, pretty much.
A programmer is a machine for converting coffee into code.
This situation has affected me. I wonder how they will certify my Linux computer. They can't run their security checker stuff on it, as it doesn't even run windows. I may have to put up a patched XP install just to regain network access. Anyone got a spare copy to donate?
I actually am a network technician at a university right now, and basically the problem with the current issues, is that the students don't know the proper security measures, like patching their systems. The majority of students that I have disinfected, haven't run windows update, ever! They usually also have out of date anti-virus definitions, and now a firewall is looking like more of a necessity. If they would realize this, then the problems wouldn't be as wide spread.
At the University I work at, this year they are just restricting resnet students from running what are deemed "Server" services on ports below 1024, such as shared drives or telnet dameons. However, above 1024, the students can run whatever services they want, so the ones who know what they are doing will run ssh up there. Also, the school has central servers that can run things (like web pages) for the students that are quite sufficent (speaking as a former student).
Next year, however, there is discussion of implementing something like checking all the dorm machines before they are allowed on the network... We have 40,000 undergrad students, so if even 1/4 are living on campus that will be quite a chore, but it is being discussed, and will happen.
One of the computing directors even told me the only reason it wasn't done this year was because they could not get the cd's for staff cut in time. I just want to know where they are going to get the army of staff that would be needed on Labor day weekend to do this.
I posted this before but it's still relevant..
I work for tech support for a large (30,000+ students) university. This fall we're expecting as many of 30 percent of the machines coming to residence to be infected with a worm.
To defend against this we're going scan all machines over the network during the registration process and if the machine is vulnerable the browser will get redirected to a webpage with the relevant patches which the client must apply. If they don't apply the patch they won't be able to connect to anything but our internal authentication vlan.
One of the reasons our networks get hammered during any worm incident is that there are so many machines connected to the network that just aren't patched ever.. Eventually we just have to manually shut down the ports infected machines are connected to and wait till clients call to complain to explain why they've been disconnected.
I go to a decent size university (about 3000 students) they recently got hit by all the worms. Working for the computer services department, we were busy with the back to school issues and also with the worm. In creating our images, we have set the virus software to update daily around 9am (I think) with a randomization of about 3 hours. This was one defense against the worm.
Another defence was through the problem reports, since the campus provides computers for every dorm room. Upon submission of the problem, sometimes we would go reimage the system with the fix. Other times we would run some virus software to remove it and then the fix. After a few days, after we had figured out the fix, we sent out an email to the entire student body with the fix and with a removal program.
On the network end, port 139 is still currently blocked since that was one way that it spread. We have yet to totally get rid of the worms, but we are almost there.
With the other viruses, the server team quickly blocked all attachments with the pif extension, and a few others. This worm was pretty much stopped before it had a chance to grow on the network.
My university never shut down dorms or the network of any sort to stop the worm. We have maintained a active roll with virus software with our own ftp server for the definitions. Our server is also update twice a day to help prevent any more outbreaks.
Even though the worms were all acrossed campus, having many people work on the stopping and blocking the transmission of the worm, I think help keep my universitys network up.
At UCB the campus wide network (not just the resnet) is on alert for infected machines. If one is found, it is denied access until a sysadmin comes out and cleans it. They've sent several warning messages prior to doing this. The news release is here
-Ryan
AUWYHSTOT (Acronyms are Useless When You Have to Spell Them Out Too)
Tech support services are basically overhead at an ISP (as far as increased service burden, ultimately cost to you). The easier you make the service, and the less dependent on tech support, the better for its consumers.
Indeed, if you call your favorite big ISPs tech support, they are unlikely to provide real help anyway (little technical insight, low pay, high turnover). Adding the extra burden of instructing the user how to un-infect their computer on something mechanical like individual telephone tech support would not help matters.
I favor the idea of cutting off infected customers. But I think the mechanism of getting customers back online should not involve the customer having to figure out that they need to call tech support - at least not first. The better way to support them is to redirect ALL HTTP requests from these customers to a ISP-provided site, which in turn informs the customer that they are seeing this page because their network access has been lost due to a virus problem on their computer.
That's the way that AT&T got customers off their @Home services (e.g. static IP addresses, dns/nntp/pop3/imap server information, etc etc). All HTTP requests went to a canned page. All usenet newsgroups at the old NNTP server contained a single message - one that instructed the customer to reconfigure their NNTP settings. All requests from non-DHCP provided IP addresses were directed to an appropriate placeholder.
I'm at NDSU in Fargo (insert obligatory joke here), and for once ITS had a semi-intelligent solution. They found some way (haven't had a chance to ask for specifics) to find out when a computer was infected (or even vulnerable, I hear), and then they just denied that MAC address an IP from the DHCP server. Once it's cleaned up, you call or email them and they put you on the list to be reactivated. Of course, it's a bit bothersome when you have to wait overnight to get a PC back online, but it's better then losing all network access while you wait for them to check everything. (Of course, this solution only came about when they didn't get the patch rolled out in the computer clusters and most of them were shut down to getting infected.)
I'm the SysAdmin for the math department, and we're still facing sporadic infection on computers that didn't get patched when I sent out an email this summer. (Would have patched them myself, but I was 1500 miles away.) Fortunately, our lab got patched the night before Blaster was triggered, so we were safe there. Only a couple faculty members who could wait a day or two to get back online.
"You will only be remembered for two things: the problems you solve or the ones you create." Mike Murdock
I got hit with the W32.Wechia.Worm today.
.NET Passport before I can do anything.
.NET Passport, which has also been cracked, and potentially sensitive user information taken.
Yes, yes... install all patches, etc. The thing is, Microsoft is releasing security patches at an alarming rate at this point, and XP's Automatic Update seems profoundly dumb... I could swear I've downloaded the same security updates 3 times now, since it apparently either doesn't detect whether you already downloaded them (I can't always install-and-reboot in the middle of my work), or there's a ongoing stream of new revs to the patches, without them stating such.
And now, MSN Messenger keeps informing me that there's a "Critical Security Update" with a link to a download page (naturally, I can't reply to the message...), and going there informs me that I must set up a
All I want to do is turn MSN Messenger off. Close, disable, whatever. Version 7 seems to have no method of preventing it from connecting and giving me a bunch of messages when I connect to the internet. Try exiting it, it says it's in use by another application, even when I have none open. Select anything regarding its startup options in the options menu, still comes up. I've now went ahead and uninstalled it using Add/Remove Programs, though I'm reluctant to do that in case I need to communicate with a client using it at some point.
This is truly annoying. It seems that in effect, Microsoft is zealously forcing me to maintain my vulnerability to exploits, by insisting I continually use their Messenger (Yahoo IM works just fine for me, thank you...). They nicely give me the alternative of updating, to do which I need to sign up for
At least in most areas, you can choose to avoid a vulnerability-laden application. It seems the Microsoft solution to their insecure software is just to go ahead and force you to use it.
Argh. Does anyone know how I can just turn off MSN Messenger? TIA!
(Disclaimer: My personal experience, Microsoft used fictionally, MS lawyers are good people, etc...)
~ Whence do you come, slayer of men, or where are you going, conqueror of space?
Here in Mexico, at my university (ITESM), there is a scanner running every 30 minutes. If it detects you are infected with the Blaster worm, your network access is revoked. You have to go to the IT department so they can check your computer and certify it virus-free.
Also, every time you go into the school's web site, a pop-up window appears with instructions on how to install Norton AV and keep it updated.
Because of these worms/virii, the network has been down intermittently for the last 4 weeks.
> If you mean Linux, I assume you somehow are going to fund training all the students how to use it, along with getting all of the school's faculty and staff to support it, along with providing for Linux patch management efforts. Yeah, right. Back to the real world we go...
Insightful? That isn't insightful, that's just plain flamebait. Obviously you've never even tried using Linux! There's nothing difficult about it at all - KDE and Gnome look enough like Windows that anyone familiar with Windows can figure out how to use it for what they want. Let's not forget that in universities, most of the students just want to use word processing for reports and stuff. KOffice, OpenOffice, etc. really don't look much different to Microsoft Word which is what most people are used to using, so I don't see any retraining costs there. And the suggestion that perhaps staff wouldn't WANT to use Linux? You're forgetting that universities are where Linux came from! RMS started the GNU project in the labs at MIT, Linus was still a student when he started Linux. I know most of the staff at my university prefer Linux but don't use it on their destktops because stupid coroporate policy dictates that they must use Windows for their desktop!
As for computer science students - should they be made to use Linux? Yes! Unix (and thus Linux) was first designed as a programmer's OS, so if they can't figure out how to use it they sure as hell won't have a chance in their computer science course!
What was all this about again? Worms? What are they? I wouldn't know, I use Linux, never had any problems with worms, trojans, viruses, etc. Everytime I see the headline "virus causes $200 trillion damage" or some other ridiculously over-inflated estimate, I just laugh. I guess it's their fault for continuing to use an OS that has so many times caused so much trouble for them.
I work in Technical Support for a local ISP here that provides access via dial-up, DSL, and terrestrial wireless (802.11b mostly, but also Turbocell, Trango & Motorola 5GHz solutions as well for backhaul links and bigger clients), and we also supply net access to a few apartment complexes and student housing facilities in the area (college town ISP).
Ever since Welchia hit, we have been doing exactly what is being described here: kicking off individual customers and even shutting off entire chunks of our network when it is discovered that a particular user or a large group of users are infected with Welchia and spewing their worm-related ICMP crap all over creation. We've had to take down entire apartment complexes and have people go door-to-door with CDs containing the removal tools and MS patches before bringing them back up.
I'm not certain how many people outside of the ISP technical support world know just how much of a PAIN Blaster and Welchia have been FOR technical support departments. Welchia came out, what, 2-3 weeks ago?, and although for the most part the majority of people are not seeing their effects anymore, these worms *are* still alive and kicking, and I don't see the end in sight anytime soon...our incoming calls have skyrocketed ever since the worms were released and especially after we found we had to take the drastic actions that we have had to take, and they have not waned yet!
We're going to be forced to continue to deal with these annoyances (-- understatement) for a long time to come.
It's not just universities doing this. My girlfriend lives in an apartment complex (primarily students) in which they have a complex-wide wireless network (Airwave, I believe). Anyhow, their network has not worked longer than 15 minutes at a time for the past 2 weeks. The apartment managers turned off the network access to everyone this past Friday and required everyone to install patches, virus scanners, "Service Pack 1", etc., and turn in a signed affidavit that this has been done in order to get internet access back...
More power to 'em!
Anyhow, my university sucks. Our campus email is flooded by upwards of 200 emails a day with "Re: Your application" in the subject line. Why can't this type of thing be handled more appropriately by the tech people at a friggin' university?
they've done this at Brandeis. unpatched windows xp/2000 computers are banned from the network.
The UW labs in Seattle were hit real hard by the Blaster worm. Thus, the UW campus network was a mess for a bit. Main causes: First, students can use the computers for whatever they want... i.e. the computers are very open. Second, IT didn't patch the computer.
Now you may wonder why I said "computer" and not "computers". Well here is why...the UW has an imaged drive lab. So one computer is used to push updates to EVERY single computer. Everytime a student logs off a computer the hard drive is made fresh again (cleaned) by the master server. That ensures proper working order and minimum IT staff work. Anything the student installed is erased too.
Single point of failure anyone?
Life is like pants... fit in or you don't fit in.
The idea of Quarantining users in a "update" sandbox sounds really cool. As long as the ISP can locally host the patches, it sounds like the perfect solution to the virus problem. I'd think we'll see virus scanning being included with ISPs in the very near future. Unfourtnately, MS is only interested in Monopoly, not fixing the problem. Most ISPs can't afford MS solution to the problem (i.e. pay MS lots of $$$ for expensive servers that still wipe out because MS can't keep up) Until Windows Update server API is untied from Windows servers (andd secret protocols, CALS, stupid patch changed EULAs, etc) it will always be a problem because no one will pay for "protection" for an insecure OS that should have been right to begin with.
Until Windows update can be written from scratch in PHP or Perl, and hosted on Linux without any other MS "restrictions" you'll continue to see the horrible virus problem. They're still trying to tie-in to the monopoly, it's about time they were forced to give it up for security!
Colleges, like the rest of society, expect students to behave in accord with established standards, or face the consequences. Violate those standards -- steal test questions, set fire to the library, etc. -- and you will be held responsible for your behavior.
There's no reason why behavior with a computer should be exempt.
If some college kid physically damaged hardware in his school's server farm and took the network down, the school might very well sue him to recover their financial losses.
Likewise, any student who deliberately releases a virus, worm, etc., on a school network ought to be held financially responsible for the damage.
Schools (and any other institutions) should establish "standards of behavior" (e.g., required protective software, avoidance of banner servers, etc.) and hold students who violate those standards responsible for their share of the damages.
-- Slashdot: When Public Access TV Says "No"
This is true (to a point) but ignores the premise that a 'default' setup of various OS (and running the basic semi-automated maintenance most do) will have some that are more or less secure than the others. Windows is unlikely to be the most secure of these, as Microsoft simply hasn't shown themselves to be that concerned with security. Mac OS X won't be too bad... Apple's been pretty good with semi-regular security updates as things are disovered, for example. The top would, I would think, end up being one of the more paranoid *BSD variants, but I might be surprised.
Mac OS 9 drew viruses even when it was a minority. Windows is the primary target, but that's not just because it's the most used, but also reflects a lack of forethought.
upgrade to a more secure operating system.
If you mean Linux...
What if he means OS X?
Seriously, think about it. We're talking about the _education_ market here. The area in which Apple has special deals with almost everyone, you know? Why not push students towards buying Macs, which (conveniently!) are available right there at the student store. Those Universities can make some money and fix up their network.
I work for RESNet at Rochester Institute of Technology. We've implemented a pretty good solution which has stopped no-one from internet access for any extended period of time.
/release to get them off the network, installs any and all necessary patches, installs the university-licensed mcafee antivirus, updates the definitions, and prompts them to restart at appropriate moments. Also on the CD for severe cases we have all the individual updates, and the Stinger virus remover.
Every PC on our network must go to start.rit.edu (when they plug in they get a temporary 10. IP, which can only access select servers, and other machines on their subnet). At the start.rit.edu page we've coded an activex control which checks the version numbers of the RPC DCOM patched files (We compiled a list of every major windows version, every service pack, pre/post RPC DCOM patch). If the user is not patched, they are redirected to a page indicating which patches they must download/install off our server -- we also have allowed the users to access windows update through a proxy (if IE auto proxy detection is turned on).
Finally we've coded a program, and put it on a CD entitled the RIT Windows Resource Kit. The program automatically detects their OS version, and upon them clicking a button, runs ipconfig
We also have RIT servers on campus who's logs are parsed on an hourly basis, and any machine which has connected to it in an attempt to spread the worm is blocked from the network. We then have a new custom-coded web interface which correlates with our network registration database: IPEdit that we can use to look up users who can't get online, explain to them to get the CD, patch their PC, run stinger, and then we can reeanble them. Most users are back online within an hour.
So far we've distributed over 5,000 copies of the CDs to each incoming freshmen and returning upperclassmen. (15,000 students at the college). As can be seen, our bandwidth usage is very much under control. Although we've experienced a lot of call volume (300 students a day) this last weekend as 2500 freshmen moved in, I'm happy to say that over 4000 students are registered on the network, and the phone in our office hasn't rung for the last hour.
May this post be indexed by spiders, and archived for all to see as my Internet epitaph.
I'm a student and restech staff at Washington University (St. Louis - not the state school in the article). Our master plan before move-in was to program in a check for the Blaster/Welchia vulnerability as students attempted to register online for their ethernet connection. However, this caused numerous problems. Firewalls prevented us from seeing the vulnerability and forced the restech consultant for each dorm to go check individual computers. This also did nothing about already-infected computers, but we programmed in an automatic disabling system to take care of those. The biggest problem, however, was that our registration subnet turned into a cesspool of infection, as people plugged in and turned on their computers and then left them unpatched and unregistered for internet access. These quickly became infected and we didn't have anything trolling through the registration subnets to automatically disable people. The resulting campuswide infection overloaded our router so much that the network-based swipe card door locks and heating/cooling systems stopped functioning. This produced lots and lots (60-80 hrs) of unpaid overtime as the small restech staff went computer-by-computer over the course of two days with a large stack of CDs programmed to patch and disinfect computers automatically, and then reenable each individual computer. Needless to say, we're still suffering from a lot of difficulties. Welchia is particularly troublesome because the Symantec/Norton fixwelchia tool often misses copies lurking in system restore points and whatnot that reinfect computers.
At the University of Connecticut, ResNet officials actually keyed into rooms. Didn't unplug the machines from the router, didn't block the MAC address.
I'm aware that this is an awful problem, but how on earth does it justify keying into someone's room?
(I'm not kidding. dailycampus.com has the story in its 8/28 back issue. They don't take external links, though this will take you to a registration page. Also notice the article on 3/6/2003 where ResNet threatens to boot warez kiddies out of housing. Real nice fellas, these guys...)
--grendel drago
Laws do not persuade just because they threaten. --Seneca
At my medical school, a bunch of students did a free vaccine drive for inner city kids. All their mothers had to do was show up with their little ones... no fee, no hassle, no problem.
Well, one problem... only about six people showed up, and this was after they advertised beforehand, posted it in the innner-city clinics, etc.
So yes, some people could care less... it was a very eye-opening experience for a group of well-meaning young physicians.
But to address the original point, there is NO justification to sanction the whole because of the actions of the few... that's a lazy and ineffective strategy.
Even if a man chops off your hand with a sword, you still have two nice, sharp bones to stick in his eyes.
i'm one of the student techs so i've been dealing with this since move in time. what the networking people did was purge all the computer registrations from the database and updated the registration page with instructions and downloads on how to protect/fix systems and told people to run them before they registered. of course not everyone could figure it out/ bothered and got infected. to handle that they've been blocking all the problem ports across network segments to minimize the spread and traffic. then the packet sniffers have been identifying infected computers and emailing the owners notifying them that they have 72 hours to get the computer cleaned or have their ethernet jack disabled. i've been having to make a lot of dorm visits to clean up systems but so far our network hasn't taken a noticeable hit. also with the recently installed webserver, every attachment is scanned for known viruses and those are deleted, and every suspect attachement has _unknown appended to them so that they can't be "accidently" run.
I never said I was smart, I just said I was smarter than you
>Ok, first off, Lindows is a garbage OS.
.exe programs aren't going to run on their computers anymore?
Maybe it is, but it *IS* linux based, and *IS* a shining example of "desktopizing" linux. Once it's installed, it's so easy to use it's a joke.
If admins were to take some time and secure it up, while maintaining the simplicity, it'd be a great option. This blaster virus shows that admins are already taking copius amounts of time doing it for windows -- why not just do it right in the first place, once?
>Secondly, what the hell is wrong with you thinking that in school is the only place high school students use a computer, and what makes you think that if they use a computer at home that their family has enough money to afford a mac!?
Alright, no problem. I think you're just proving that computers are so easy to use that learning two OSes, one of them "untaught" (that being the windows PC at home) that expecting someone to learn another that is comparable in simplicity isn't too much to ask.
>Thrid, suppose Linux were mandatory at Universities, are you volunteering your time to explain to students how to properly secure their machines, and explain that all those little
No, and it wouldn't need to be mandatory. It wouldn't make any sense for it to be mandatory. A university is a learning institute. Learning takes place using computers of many forms, from windows, to Mac, to Linux, to mainframe. Simply offering a good linux distro as an option should be fair enough.
>And you're going to tell them personally that the $700+ office software mommy bought them can't be used?
ROTFLMAO! I run a computer store and I can tell you "mommy" is so cheap with their kid's computer that simply getting them not to pirate the OS is a task and a half itself. 90% of the systems that come through my doors for repair won't install SP1 and are going to get infected OVER and OVER because they use the windows pirate key. Sure, I turn on the firewall, but the users just shut that feature down when they get it home and kazaa seems "slower". No, I won't help them fix their pirate OS to work like a normal one. I don't assist in piracy.
I've not sold a single copy of office, despite the fact that the real price of it is about $289.99 CDN. Although I'm a new store, openoffice (Free, of course) is turning out to be a hugely popular alternative, even if the computer just gets a pirated copy of office installed by the user when it leaves the store. Even my $259.99 CDN special is a tough sell to some parents. A lot of them are buying used systems for less from me.
The fact is a fully supported, even if optional, linux install at a university will help break it into the desktop market. And that can only be a good thing for society (and my store -- my profit margins on software are next to nil, so I don't care to sell it anyways).
If you could be told what you can see or read, then it follows that you could be told what to say or think - BoC
I'm a senior at SNHU and this is what I have observed.
There was a noticable slowdown on Saturday and Sunday (when all freshmen moved in), but the network didn't go down. I imagine probably some of it was the normal freshman Internet traffic since many of them never had fast internet before, the rest was from Blaster.
Returning students arrived on Monday and Tuesday. Tuesday the network got slower and SLOWER and SLOOOOWEERRR then crashed about mid-afternoon. Didn't come up until yesterday morning.
RA's and orientation leaders were given CD's with the patch, fix tool, and virus definition files for various popular virus scanners.
Knowing this university, there will still be people unpatched come next May since no one has gone door-to-door to verify everyone's computers.
Oh and some students randomly can't get on the internet. Noticed today I had an IP address conflict, so I got a suspcion that the DHCP server has also ran out of IP addresses.
My girlfriend goes to NEC and their network has been totally down since Sunday. Basically they are going to go to each computer and patch it before they turn the network on. For some reason they insisted on attempting to patch her computer even though she showed them it was running Windows 98 SE (which isn't effected by Blaster), just like I told her to do. *sigh*
how many parents are against vaccination programs... I'm not even talking MANDATORY vaccination programs, I'm talking vaccines in general. Probably as many are motivated by fear as are motivated by religion.
There are people out there who preach that vaccines are a scam; nothing but evil, drug company money-makers. They look at the very small numbers of adverse reactions, where vaccines make people sick (a few hundred cases, generally out of millions of doses), and use those incidents to frighten parents into avoiding vaccination. Some use the logic that "if everyone else is vaccinated, you won't have to be, because you'll never come into contact with a diseased person!" Well, that might have been true before the jet age... but I've seen rare-in-the-US diseases in my ER, sometimes in immigrants, (sitting next to your child in the waiting room), sometimes not. Some vaccines don't induce an immune response in certain people, so they are potential infectious sources. Bottom line: there is always a small reservoir of people out there who can infect you. The choice of whether to get a shot or not is really up to the individual.
Personally, I'm generally a fan of vaccinations (with some exceptions)... but not all doctors are. If you meet one who's not a fan, ask him why. If he starts spieling some wide-eyed conspiracy theory stuff, RUN the other way. On the other hand, If he starts talking about odds ratios, attack rates, and slightly increased complication rates for certain age groups, he may know what he's talking about... consider listening, then check it out for yourself.
Just remember, not all doctors who are against certain vaccines are crackpots.
Even if a man chops off your hand with a sword, you still have two nice, sharp bones to stick in his eyes.
This is what my school did with Blaster...
They just pulled the fiber from the routers down in the basement (IT's standard location). We spent the next 6 days (weekend included) going from door to door with a bevy of CD's (one for each OS, created by our poor MCSE). Each CD had a little batch job that scanned the PC, removed the infection (if it existed), and then installed the appropriate patch.
This was made more complicated by the University's privacy policy, which mandates that a school employee cannot enter a student's room alone. We had to travel in teams, and with a small school's IT department, that meant we had 3 teams for 2,500+ PC's. That comes out to over $5K in manhours alone.
The infection rate was approx. 68%. I think we need a class on how to install patches.
What if this weren't a hypothetical question?
At the University I work/attend school at, we've been experiencing major problems with the load on our PIX firewall. The primary fails and rolls to the secondary a couple dozen times per day. I would assume that this is happening in many places.
This summer has been very very busy (fun) for us. In the middle of a MAJOR Cisco IOS upgrade, several worms get unleashed. Then while combatting those things, we get hit by the massive power failure that reveals that some of Cisco's new code doesn't recover perfectly after a power failure... as in... DOESN'T WORK. Ah woohoo!
"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman
In retrospect, we should have been a little more proactive like GMU was. At 1:52:28 on Saturday, an infected machine was connected. Before 2:00, all the dormitory subnets were basically unreachable, and I was seeing broadcast traffic (the arp storm from all the pings) rates approaching 2000/second. We got lucky and our institutional machines (the Windows ones anyway, we have a lot of Suns and Linux boxes and such around) were already patched, due to vigilance on the part of our technicians. I managed to get the academic buildings back by temporarily stopping all ICMP at the building routers (I had to go around to most of them with a laptop and a serial cable). We have a "class B" IP range (yep, we're one of the evil institutions causing the shortage in China or wherever it is) which fits exactly into what Nachi starts to scan, so this actually helped. I finally ended up bringing down all eighteen dormitories down once we figured out for certain what it was (I thought it was a failed router or something spewing garbage onto its segments at first), then we brought up one building at a time, and sent a quick bit of Perl through the segment disabling all the ports with vulnerable machines on them (about 70% of the W2K/XP machines were vulnerable, and about 40% were infected).
At that point, we co-opted the resident assistant staff and had them going door-to-door with our techs (we called in all 60 or so at that point) cleaning and patching machines, and reactivating ports. Amazingly enough, we actually had everything more or less back to normal by the time classes started on Monday.
Things I learned from the experience:
I was reading through this discussion and was about to post about the work UCONN did. I think one of their admins posted the link to their page to resnet-l last week and I was impressed.
They did a very nice job containing the spread of the worm. Kudos to them.
On the other hand, the response from our office (Housing Tech Support at a school in Indiana, we just help students get online, don't deal w/ switches and routers) has been somewhere between nothing and next to nothing. I asked my boss to go buy us some blank CD's as it became apparent that Blaster was going to be a huge problem, she just ignored me. Sigh. Luckily I'll be unemployed in December (when I graduate), and won't have to deal with my boss' incompetence any more.
Am I the only one that thinks this whole strategy, the whole situation of having to shut down the entire network and clean each individual node (PC) before you start up the network again, is quite literally insane? Every time I read about something like this it reminds me of someone trying to plug up enough holes in a sieve to make it hold water. Next time some idiot (i.e., the Dean) brings in his infected personal computer and hooks up to the university's internal network, don't they just get to start this whole Chinese Fire Drill all over again?
Madness. Isn't there a better way to do things? Why does anyone in the IT world even put up with this? Why does *anyone* put up with this? Would having everyone run Linux/UNIX/MacOS X even make any difference, or would it just be a matter of time before some new worm broke out and they had to take down the whole network and clean every Linux PC the same way they're doing with Windows PCs? Or, to rephrase, if you took Microsoft out of the equation, would this situation even be possible?
I'm looking for some serious discussion, not jokes.