Universities Taken Offline to Fight Worms, Viruses

chrismg2003 writes "Nationwide universities are opening their doors to new students but closing off their network services. The Blaster worm has caused universities to take drastic actions to protect their campus networks. Universities have gone as far as shutting down their entire resnet network and bringing it back up dorm-by-dorm after each computer has been certified worm-free. The ICMP ping requests alone have brought down my university's resnet multiple times and we are scrambling to clean the worm from all computers before it forces us to follow suit with other universities."

Comments

Can ISPs get with it too?

[inertia187]

Can we get the ISPs to do this too? It'd be really great if they'd just turn off a tiny manageable chunk of infected users and wait for them to call support. Support could then tell them to patch, or upgrade, or get some other type of clue. A really with-it ISP could just replace the web page the user wanted with a page that tells them to get with it.

Problem is, any plan will cost money to support. Worse, it might prompt the users to just cancel their service. I can't imagine ISPs like that idea. At least with the universities, the students have no choice, pretty much.

Re:Can ISPs get with it too?

[colinramsay]

Here in the UK, NTL did just that. I'd taken down our firewall for about five minutes and in that time we contracted Blaster, which promptly got eaten by Welchia. I scanned for Blaster and applied the MS patch but didn't scan for Welchia...

Next day, we try and go online only to be redirected to http://outbreak.ntli.net/ which told us they'd found that we were transmitting loads of data... they gave us links to blaster and welchia scanners and the MS patch. Until we stopped transmitting we weren't going to be allowed onto the net at large.

Upon removing Welchia we were promptly allowed back online. I've never been very impressed with NTL before, but this sort of decisive action was very impressive.

Re:Can ISPs get with it too?

[dazk]

Where's the problem to shut people down but allow them to reach a server where all the relevant patches for the malware causing a shutdown is available? Might even be a proxy to official MS sites.

Re:Can ISPs get with it too?

[Abcd1234]

A few hours. You honestly think it'd take just a few hours to 1) take all the calls from a bunch of people who's net connection are shut down and 2) instruct them (and potentially walk them through) how to disinfect and patch their systems? Really. Frankly, I find it remarkable how naive you are...

I absolutely agree with the original poster... if some idiot doesn't patch his box, I shouldn't suffer. If anything, set up rules at the upstream router to shut down his, and only his, connection (hell, you could automate this if you wanted). But don't you dare touch mine.

Linux

[Anonymous Coward]

This situation has affected me. I wonder how they will certify my Linux computer. They can't run their security checker stuff on it, as it doesn't even run windows. I may have to put up a patched XP install just to regain network access. Anyone got a spare copy to donate?

Our Solution

[RedSynapse]

I posted this before but it's still relevant..

I work for tech support for a large (30,000+ students) university. This fall we're expecting as many of 30 percent of the machines coming to residence to be infected with a worm.

To defend against this we're going scan all machines over the network during the registration process and if the machine is vulnerable the browser will get redirected to a webpage with the relevant patches which the client must apply. If they don't apply the patch they won't be able to connect to anything but our internal authentication vlan.

One of the reasons our networks get hammered during any worm incident is that there are so many machines connected to the network that just aren't patched ever.. Eventually we just have to manually shut down the ports infected machines are connected to and wait till clients call to complain to explain why they've been disconnected.

Too mechanical

[Tor]

Tech support services are basically overhead at an ISP (as far as increased service burden, ultimately cost to you). The easier you make the service, and the less dependent on tech support, the better for its consumers.

Indeed, if you call your favorite big ISPs tech support, they are unlikely to provide real help anyway (little technical insight, low pay, high turnover). Adding the extra burden of instructing the user how to un-infect their computer on something mechanical like individual telephone tech support would not help matters.

I favor the idea of cutting off infected customers. But I think the mechanism of getting customers back online should not involve the customer having to figure out that they need to call tech support - at least not first. The better way to support them is to redirect ALL HTTP requests from these customers to a ISP-provided site, which in turn informs the customer that they are seeing this page because their network access has been lost due to a virus problem on their computer.

That's the way that AT&T got customers off their @Home services (e.g. static IP addresses, dns/nntp/pop3/imap server information, etc etc). All HTTP requests went to a canned page. All usenet newsgroups at the old NNTP server contained a single message - one that instructed the customer to reconfigure their NNTP settings. All requests from non-DHCP provided IP addresses were directed to an appropriate placeholder.

This ISP does

[nathana]

I work in Technical Support for a local ISP here that provides access via dial-up, DSL, and terrestrial wireless (802.11b mostly, but also Turbocell, Trango & Motorola 5GHz solutions as well for backhaul links and bigger clients), and we also supply net access to a few apartment complexes and student housing facilities in the area (college town ISP).

Ever since Welchia hit, we have been doing exactly what is being described here: kicking off individual customers and even shutting off entire chunks of our network when it is discovered that a particular user or a large group of users are infected with Welchia and spewing their worm-related ICMP crap all over creation. We've had to take down entire apartment complexes and have people go door-to-door with CDs containing the removal tools and MS patches before bringing them back up.

I'm not certain how many people outside of the ISP technical support world know just how much of a PAIN Blaster and Welchia have been FOR technical support departments. Welchia came out, what, 2-3 weeks ago?, and although for the most part the majority of people are not seeing their effects anymore, these worms *are* still alive and kicking, and I don't see the end in sight anytime soon...our incoming calls have skyrocketed ever since the worms were released and especially after we found we had to take the drastic actions that we have had to take, and they have not waned yet!

We're going to be forced to continue to deal with these annoyances (-- understatement) for a long time to come.

UW Labs

[jeeryg_flashaccess]

The UW labs in Seattle were hit real hard by the Blaster worm. Thus, the UW campus network was a mess for a bit. Main causes: First, students can use the computers for whatever they want... i.e. the computers are very open. Second, IT didn't patch the computer.

Now you may wonder why I said "computer" and not "computers". Well here is why...the UW has an imaged drive lab. So one computer is used to push updates to EVERY single computer. Everytime a student logs off a computer the hard drive is made fresh again (cleaned) by the master server. That ensures proper working order and minimum IT staff work. Anything the student installed is erased too.

Single point of failure anyone?

RIT's Solution -- Working well

[LogicX]

I work for RESNet at Rochester Institute of Technology. We've implemented a pretty good solution which has stopped no-one from internet access for any extended period of time.

Every PC on our network must go to start.rit.edu (when they plug in they get a temporary 10. IP, which can only access select servers, and other machines on their subnet). At the start.rit.edu page we've coded an activex control which checks the version numbers of the RPC DCOM patched files (We compiled a list of every major windows version, every service pack, pre/post RPC DCOM patch). If the user is not patched, they are redirected to a page indicating which patches they must download/install off our server -- we also have allowed the users to access windows update through a proxy (if IE auto proxy detection is turned on).

Finally we've coded a program, and put it on a CD entitled the RIT Windows Resource Kit. The program automatically detects their OS version, and upon them clicking a button, runs ipconfig /release to get them off the network, installs any and all necessary patches, installs the university-licensed mcafee antivirus, updates the definitions, and prompts them to restart at appropriate moments. Also on the CD for severe cases we have all the individual updates, and the Stinger virus remover.

We also have RIT servers on campus who's logs are parsed on an hourly basis, and any machine which has connected to it in an attempt to spread the worm is blocked from the network. We then have a new custom-coded web interface which correlates with our network registration database: IPEdit that we can use to look up users who can't get online, explain to them to get the CD, patch their PC, run stinger, and then we can reeanble them. Most users are back online within an hour.

So far we've distributed over 5,000 copies of the CDs to each incoming freshmen and returning upperclassmen. (15,000 students at the college). As can be seen, our bandwidth usage is very much under control. Although we've experienced a lot of call volume (300 students a day) this last weekend as 2500 freshmen moved in, I'm happy to say that over 4000 students are registered on the network, and the phone in our office hasn't rung for the last hour.