Slashdot Mirror
Universities Taken Offline to Fight Worms, Viruses
chrismg2003 writes "Nationwide universities are opening their doors to new students but closing off their network services. The Blaster worm has caused universities to take drastic actions to protect their campus networks. Universities have gone as far as shutting down their entire resnet network and bringing it back up dorm-by-dorm after each computer has been certified worm-free. The ICMP ping requests alone have brought down my university's resnet multiple times and we are scrambling to clean the worm from all computers before it forces us to follow suit with other universities."
Can we get the ISPs to do this too? It'd be really great if they'd just turn off a tiny manageable chunk of infected users and wait for them to call support. Support could then tell them to patch, or upgrade, or get some other type of clue. A really with-it ISP could just replace the web page the user wanted with a page that tells them to get with it.
Problem is, any plan will cost money to support. Worse, it might prompt the users to just cancel their service. I can't imagine ISPs like that idea. At least with the universities, the students have no choice, pretty much.
A programmer is a machine for converting coffee into code.
This situation has affected me. I wonder how they will certify my Linux computer. They can't run their security checker stuff on it, as it doesn't even run windows. I may have to put up a patched XP install just to regain network access. Anyone got a spare copy to donate?
> upgrade to a more secure operating system. If you mean Linux, I assume you somehow are going to fund training all the students how to use it, along with getting all of the school's faculty and staff to support it, along with providing for Linux patch management efforts. Yeah, right. Back to the real world we go...
You should get a partial tuition refund if you don't use Windows, and thus the university's IT doesn't have to worry about you.
At the University I work at, this year they are just restricting resnet students from running what are deemed "Server" services on ports below 1024, such as shared drives or telnet dameons. However, above 1024, the students can run whatever services they want, so the ones who know what they are doing will run ssh up there. Also, the school has central servers that can run things (like web pages) for the students that are quite sufficent (speaking as a former student).
Next year, however, there is discussion of implementing something like checking all the dorm machines before they are allowed on the network... We have 40,000 undergrad students, so if even 1/4 are living on campus that will be quite a chore, but it is being discussed, and will happen.
One of the computing directors even told me the only reason it wasn't done this year was because they could not get the cd's for staff cut in time. I just want to know where they are going to get the army of staff that would be needed on Labor day weekend to do this.
I posted this before but it's still relevant..
I work for tech support for a large (30,000+ students) university. This fall we're expecting as many of 30 percent of the machines coming to residence to be infected with a worm.
To defend against this we're going scan all machines over the network during the registration process and if the machine is vulnerable the browser will get redirected to a webpage with the relevant patches which the client must apply. If they don't apply the patch they won't be able to connect to anything but our internal authentication vlan.
One of the reasons our networks get hammered during any worm incident is that there are so many machines connected to the network that just aren't patched ever.. Eventually we just have to manually shut down the ports infected machines are connected to and wait till clients call to complain to explain why they've been disconnected.
Except that most students weren't around in July. You can't make students apply patches while they are off for the summer.
Of course you can try to educate them so that they will understand the need for these patches and apply them on their own, but actually achieving that goal is not a trivial task (and perhaps drastic actions like kicking machines off university networks are the first step in a tough love approach that might just work).
At the university where I work, the main campus is in the middle of an XP rollout, and the builds being installed didn't have the patch applied. Hosed the network so badly that remote updating wasn't possible - all the techs have been frantically running around with patch disks for the last few days.
Fortunately, the campus where I'm based is mostly on Win 9x, and we managed to get most of the rest of them patched before many were infected. We thought that we'd got them all, but we were still seeing ridiculous ICMP traffic. The networking people checked the traffic logs, and the PCs were identified.
They belonged to two of the Technical Support staff.
At UCB the campus wide network (not just the resnet) is on alert for infected machines. If one is found, it is denied access until a sysadmin comes out and cleans it. They've sent several warning messages prior to doing this. The news release is here
-Ryan
AUWYHSTOT (Acronyms are Useless When You Have to Spell Them Out Too)
Comment removed based on user account deletion
Tech support services are basically overhead at an ISP (as far as increased service burden, ultimately cost to you). The easier you make the service, and the less dependent on tech support, the better for its consumers.
Indeed, if you call your favorite big ISPs tech support, they are unlikely to provide real help anyway (little technical insight, low pay, high turnover). Adding the extra burden of instructing the user how to un-infect their computer on something mechanical like individual telephone tech support would not help matters.
I favor the idea of cutting off infected customers. But I think the mechanism of getting customers back online should not involve the customer having to figure out that they need to call tech support - at least not first. The better way to support them is to redirect ALL HTTP requests from these customers to a ISP-provided site, which in turn informs the customer that they are seeing this page because their network access has been lost due to a virus problem on their computer.
That's the way that AT&T got customers off their @Home services (e.g. static IP addresses, dns/nntp/pop3/imap server information, etc etc). All HTTP requests went to a canned page. All usenet newsgroups at the old NNTP server contained a single message - one that instructed the customer to reconfigure their NNTP settings. All requests from non-DHCP provided IP addresses were directed to an appropriate placeholder.
The action seems perfectly reasonable to me:
To get the school's message across, all students were asked to sign a document confirming that their computers were updated with all the needed security upgrades. Not enough students confirmed that their machines were updated, prompting the GMU action today. Administrators said they would try later today to reconnect dorms, weeding out students with infected PCs. Students living off campus can continue to dial in to the campus computer network.
Looks like the kids are getting a decent deal on virus-removal and system updates too:
Students are being charged $30 if a university technician is called in to clean an infected machine, a school spokesman said. Students can go to off-campus experts for a fix but must certify that their computers are updated with the latest security fixes before being allowed to access the campus network.
Hmph, I can't find anything wrong here. Of course, there are a couple of choice quotes from the kids who, I believe, are our future:
Kimberly Borchert, a 19-year-old sophomore, said her computer "freaked out" as soon as she plugged it into the school's network last week.
Freshman Andrew Canose was one of several GMU students who encountered problems after installing the university-provided anti-virus software. Canose found the new program conflicted with an older anti-virus program already on his computer. "My computer is like at war with itself and won't work," he said.
But my favorite lines are from the admins, such as this gem:
"I think we really need to groom a new type of student who is responsible for their computer security," said Kathy Gillette, manager of George Mason University's beleaguered tech support center. "A lot of them lived at home and mom or dad took care of the computer so they've never learned how to fix them, but hopefully we'll be able to teach them that too."
And the classic:
"There were a certain percentage of students that wouldn't listen to us unless we hit them upside the head with a lockout," he said. "You simply can't deal with these problems until you've got your network under control."
everything in moderation
I got hit with the W32.Wechia.Worm today.
.NET Passport before I can do anything.
.NET Passport, which has also been cracked, and potentially sensitive user information taken.
Yes, yes... install all patches, etc. The thing is, Microsoft is releasing security patches at an alarming rate at this point, and XP's Automatic Update seems profoundly dumb... I could swear I've downloaded the same security updates 3 times now, since it apparently either doesn't detect whether you already downloaded them (I can't always install-and-reboot in the middle of my work), or there's a ongoing stream of new revs to the patches, without them stating such.
And now, MSN Messenger keeps informing me that there's a "Critical Security Update" with a link to a download page (naturally, I can't reply to the message...), and going there informs me that I must set up a
All I want to do is turn MSN Messenger off. Close, disable, whatever. Version 7 seems to have no method of preventing it from connecting and giving me a bunch of messages when I connect to the internet. Try exiting it, it says it's in use by another application, even when I have none open. Select anything regarding its startup options in the options menu, still comes up. I've now went ahead and uninstalled it using Add/Remove Programs, though I'm reluctant to do that in case I need to communicate with a client using it at some point.
This is truly annoying. It seems that in effect, Microsoft is zealously forcing me to maintain my vulnerability to exploits, by insisting I continually use their Messenger (Yahoo IM works just fine for me, thank you...). They nicely give me the alternative of updating, to do which I need to sign up for
At least in most areas, you can choose to avoid a vulnerability-laden application. It seems the Microsoft solution to their insecure software is just to go ahead and force you to use it.
Argh. Does anyone know how I can just turn off MSN Messenger? TIA!
(Disclaimer: My personal experience, Microsoft used fictionally, MS lawyers are good people, etc...)
~ Whence do you come, slayer of men, or where are you going, conqueror of space?
I work in Technical Support for a local ISP here that provides access via dial-up, DSL, and terrestrial wireless (802.11b mostly, but also Turbocell, Trango & Motorola 5GHz solutions as well for backhaul links and bigger clients), and we also supply net access to a few apartment complexes and student housing facilities in the area (college town ISP).
Ever since Welchia hit, we have been doing exactly what is being described here: kicking off individual customers and even shutting off entire chunks of our network when it is discovered that a particular user or a large group of users are infected with Welchia and spewing their worm-related ICMP crap all over creation. We've had to take down entire apartment complexes and have people go door-to-door with CDs containing the removal tools and MS patches before bringing them back up.
I'm not certain how many people outside of the ISP technical support world know just how much of a PAIN Blaster and Welchia have been FOR technical support departments. Welchia came out, what, 2-3 weeks ago?, and although for the most part the majority of people are not seeing their effects anymore, these worms *are* still alive and kicking, and I don't see the end in sight anytime soon...our incoming calls have skyrocketed ever since the worms were released and especially after we found we had to take the drastic actions that we have had to take, and they have not waned yet!
We're going to be forced to continue to deal with these annoyances (-- understatement) for a long time to come.
The UW labs in Seattle were hit real hard by the Blaster worm. Thus, the UW campus network was a mess for a bit. Main causes: First, students can use the computers for whatever they want... i.e. the computers are very open. Second, IT didn't patch the computer.
Now you may wonder why I said "computer" and not "computers". Well here is why...the UW has an imaged drive lab. So one computer is used to push updates to EVERY single computer. Everytime a student logs off a computer the hard drive is made fresh again (cleaned) by the master server. That ensures proper working order and minimum IT staff work. Anything the student installed is erased too.
Single point of failure anyone?
Life is like pants... fit in or you don't fit in.
And far FAR easier than "switching" to Linux.
Anyone "retarded" enough to get infected with a virus on Windows is FAR too "retarded" to not get their linux box rooted. Especially with the blaster virus. It could be blocked by two compeltely seperate and simple prevention schemes.
If you have your linux box, unsecured on the net, then you are the "retarded" one. You have either been rooted already and don't know it or it will happen soon.
If you HAVE secured it, I guarantee you did more work to do so that it would have taken anyone to prevent being infected with Blaster.
Contrary to popular belief, coding is not all free blow-jobs and beer. Those things cost MONEY!
sometimes the techs are so harried for time that they don't get around to patching their own shit.
Sometimes they are so lame they can't be bothered to wipe their own asses, either...
Still, what a professional embarassment!
I work for RESNet at Rochester Institute of Technology. We've implemented a pretty good solution which has stopped no-one from internet access for any extended period of time.
/release to get them off the network, installs any and all necessary patches, installs the university-licensed mcafee antivirus, updates the definitions, and prompts them to restart at appropriate moments. Also on the CD for severe cases we have all the individual updates, and the Stinger virus remover.
Every PC on our network must go to start.rit.edu (when they plug in they get a temporary 10. IP, which can only access select servers, and other machines on their subnet). At the start.rit.edu page we've coded an activex control which checks the version numbers of the RPC DCOM patched files (We compiled a list of every major windows version, every service pack, pre/post RPC DCOM patch). If the user is not patched, they are redirected to a page indicating which patches they must download/install off our server -- we also have allowed the users to access windows update through a proxy (if IE auto proxy detection is turned on).
Finally we've coded a program, and put it on a CD entitled the RIT Windows Resource Kit. The program automatically detects their OS version, and upon them clicking a button, runs ipconfig
We also have RIT servers on campus who's logs are parsed on an hourly basis, and any machine which has connected to it in an attempt to spread the worm is blocked from the network. We then have a new custom-coded web interface which correlates with our network registration database: IPEdit that we can use to look up users who can't get online, explain to them to get the CD, patch their PC, run stinger, and then we can reeanble them. Most users are back online within an hour.
So far we've distributed over 5,000 copies of the CDs to each incoming freshmen and returning upperclassmen. (15,000 students at the college). As can be seen, our bandwidth usage is very much under control. Although we've experienced a lot of call volume (300 students a day) this last weekend as 2500 freshmen moved in, I'm happy to say that over 4000 students are registered on the network, and the phone in our office hasn't rung for the last hour.
May this post be indexed by spiders, and archived for all to see as my Internet epitaph.
I'm only on page 3 of 7.. but think I have made enough comments to show that we should take this article with more than a grain of salt. I'm going to read the rest of the article now.
-tor
I think this is a brilliant world. Unfortunately, there are already some sleazy companies who have pop-up ads that say the same thing. (ie. "You're infected with MSBlaster, patch your machine, then protect yourself permanently with (whatever the company's product is called.)"
You could also exploit a common NT hole by sending an NTMESSENGER message to them. (ie. "Message from Root@yourdomain.com: Your machine has been infected with a virus, please visit Windows Update to apply the patch ASAP.)
Who did what now?
Is all the extra work that these worms and what not are causing for us IT folks, good for our industry in general? Certainly it keeps us busy just keeping everything running, and that's gotta keep a few people on the payroll.
If that's the case, I'd like to send a shout-out to all the virus and worm authors out there: you infect my computer and I'll pop a cap in yo azz, but as long as you just infect the clueless newbies, and it helps me separate them from their cash, I give you the thumbs up.
Synergy is your friend
At my medical school, a bunch of students did a free vaccine drive for inner city kids. All their mothers had to do was show up with their little ones... no fee, no hassle, no problem.
Well, one problem... only about six people showed up, and this was after they advertised beforehand, posted it in the innner-city clinics, etc.
So yes, some people could care less... it was a very eye-opening experience for a group of well-meaning young physicians.
But to address the original point, there is NO justification to sanction the whole because of the actions of the few... that's a lazy and ineffective strategy.
Even if a man chops off your hand with a sword, you still have two nice, sharp bones to stick in his eyes.
Here at Denison University, we were lucky enough to catch wind of this perl script, written by Josh Richard of the University of Minnesota-Duluth and enhanced by Mike Lang of the University of Connecticut enhanced it. We modified our standard registration web page (unknown mac-addresses are handed a dummy ip and all traffic redirects to a registration page. Once they register, DHCP hands them a "real" ip) to scan for the DCOM vulnerability using the UCONN script. Users that fail the test are redirected to a page offering links to the patches. Users that pass are directed to the standard registration page, including virus scanning downloads. UConn also includes handy suggestions for using TCP dump to listen on port 135 and for ICMP, note it in a log, giving you a great list of IPs that need to be cleaned. Read UConn's entire summary page here. It saved us.
This comment was not generated by Uber Elephants...