Slashdot Mirror
Handling User Grown Machines on a Large Network?
matth asks: "Recently with the outbreak of the MSBLASTER worm and the startup of the college semester here in the US we've been hit by a big problem here where I work. Many students are bringing in machines from home, often times infected. The infections are so bad that they bring the whole network to a crawl. Yes, you can install ACLs on edge routers and put a router between the dorms and the rest of your network, but it still brings the dorm to a crawl. You can make sure people install the patches, but what if someone re-installs Windows, or brings in another machine, and what about NEXT year? From the Slashdot community, how have sysadmins out there dealt with this? How can you manage each machine in a network such as a college, where people are bringing their own machines in from the outside? ACLs on routers... but what about for the segmented network?"
Force them to login to an Active Directory domain and hand out updates...
You talk better than you fool!
You can only separate networks so much.
If you make them bear some financial responsibility for not checking their machines first this might help.
just ban users from your network.
At my university, at least for the public machines, when you logon to the domain, a script executes that automatically patches your machine and runs fixblast and fixwelch. you might want to investigate into something like that
If you can track down where the traffic is coming from (which I believe you can with MSBLASTER, at least to the extent of IP address and from there, MAC address), block their port until they fix their machine. Once they've (a) patched up and (b) removed MSBLASTER, let them back on. Having an A4 sheet detailing where to get the patch and removal tool (possibly mirrored locally) would be a good idea too.
Do some intrusion detection on the network--possibly through Snort. If any machine is spamming out MSBlast messages or Sobig emails, drop their connection via MAC address and refuse to give them another DHCP lease. Then, when the person comes in to complain, let them know their computer was infected and flooding the network, and give them a floppy with the proper security patch on it.
It might be a bit annoying to automate the process (except for handing out floppies) at first, but it seems like it could significantly help, while at the same time educating users to update their patches.
No comment.
...tell students at registration that Windows machines are not allowed on the network, and that they must install Linux. This will not only clean up your network problems, but it will also give the students a sense of doing the right thing for their computers. Along with their free condoms, give 'em free Linux CDs.
Here at my school, for the last week, starting about a day before freshman move in, they have had flyers *everyewhere* telling people not to hook up the network until they install this patch provided by the IT dept. Of course, there are still the bozo's that don't pay heed to the warnings....but there are lots of them in the world anyways.
Ensure that home machines (ones that you haven't configured) get IPs in a VLAN group which you've bandwidth throttled on the routers/switches along the say so the rest of the VLANs don't get choked by home-grown disasters.
Machines you have control over can get IPs in another VLAN which isn't throttled, or at least not as much as your "uncontrollable" VLAN. At the router where the VLANs can meet have strong ACLs and traffic flow control.
Just because you give them access with their own machines doesn't mean you have to give them unrestrained access.
Trolling is a art,
assuming your network is switched, and your switch are "manageables" (ie you can log in them remotely)
you could have an IDS (or similar) with a rule looking for specific attacks (ie blaster). when you detect such an attack, fire off a script that shuts down the user's port on the switch. they'll bitch and moan that they can't access the net but you'll know who they are now and charge them a cleanup fee (make sure to include it in the terms of use)
another solution is to require anyone bringing a computer from home to have it inspected by your techs, block access based on mac address and only give them access once they passed the test. it does require more ressources tho, and ideally you'd still need the first option (in case where someone reinstall windows)
I think this was one of the approaches Stanford was going to take. No DNS for your machine until you get it checked out by their IT department.
Chris
... from another point of view.
I'm a student at a university whose dorm network got nailed by blaster something fierce. Almost as bad as it was Klezed a couple years before. Anyways, because of all of this, the sys admins decided to completely eliminate the dorm network from the upper campus one - also cutting off 'net access - during school hours. This is a real big pain in the butt, and I'm actually hoping there are some great answers in this topic so I can give them to my sys admin.
Of course, compounding the situation are seemingly (dunno if they actually are or not considering I've never even SEEN one before) incompetant dorm techs taking an entire day to clear out just one dorm building of ~50 rooms (2 people per room, but often less than 2 PCs per room...). Considering Blaster only affects 2000/XP/2003 machines, that means that the roughly 50 computers running those took 8 hours to clean? Something seems wrong here.
I'm just annoyed because my room (along with my entire hall since I'm the resident 'hey, call him!' computer geek and have patched everyone) is completely free of blaster and its ilk, yet I have to deal with the people who either don't know to patch Windows often, or don't care.
How about this one: What can a STUDENT at one of these schools do to help? I've tried teaching as many people as possible about computer safety (take a health classes' STD safety course, apply to computers basically), and I'm ineligable to become a dorm tech right now... anyone?
--- Ãther SPOON!
"Along with their free condoms, give 'em free Linux CDs."
"Here. You'll never use this first item if you choose to use the second item. Have fun, and welcome to college."
You are sooooo fired.
You ought to be able to tweak your DHCP so you can block machines that are broadcasting this badly by telling them their default gateway is localhost.
My problem? I was perfectly gruntled, until some numbnuts came by and dissed me.
Assuming you can identify the port from which the infected traffic is coming, post a list of all infected rooms on the front door of the dorms, with an explanation that "these computers are causing your network to suck."
The problem will be fixed.
Moderate drunk! It's more fun that way!
... when you go to a university where you do not log on to a domain in dorms.
I've found that to be very common (including the Uni that I'm typing this at) since it is MUCH easier to set freshman up on movein day.
Also, certain things do not work when you start logging onto domains. Example: XP's fast user switching. You'd have students complaining about the administration restricting their rights to their own computer, blah blah blah... then on top of it, automatically patching something. Legal nightmare. Works great for lab PCs, horrid for dorm PCs.
--- Ãther SPOON!
First they came for the menial jobs. I never spoke out because I didn't have a menial job.
Somebody has obviously made a serious mistake then. Can I suggest you apply at the sign of the Golden Arches to find something more commensurate with your intellectual abilities?
When the blaster worm hit, we had to work for a few days to clear the thing from the staff network.
Now that we well and truly cleared it after much scanning to make sure, we've moved on to the on-campus student's network.
We have to physically go to each room, patch and scan to remove both blaster and welchier.
It's both an annoyance for us and the students who pretty much treat us like unwanted guests on their pcs.
Be you Admins? nay, we are but lusers!
For years, the last thing the admins at my university wanted to do was inspect each computer before it was permitted to be on the network. This year they have broken down and are doing so, to be connected (wired or wirelessly) one of their employees must inspect the computer and make sure that they are not only completely patched, but also that they are running antiviral software (Norton ONLY).
This is of course great in theory, until a week later when someone formats, 'forgets' to patch, brings their computer home, gets re-infected and comes back to school.
Until patches become mandatory for many of these users, there is no way to prevent such a thing... short of finding the virus writers and skinning them alive during prime time, that might make some of these script kiddies think twice before doing what they do.
Help Brendan pay off his student loans
This is so ridiculous that I'm still thinking you're joking. Either that or you haven't been in education for a long time.
I'm a CS student. We often have the choice of buying an outdated EUR 6 hardcopy of a lecture script (without TOC or index), printing some 200 pages (on a printer quota that's sufficient for 150) or viewing the constantly updated script on-screen with search functionality. This holds true for at least four courses per semester. Without PCs, we'd be royally screwed.
In most appartment buildings for students, the network is provided by the university over a 2MBit line with at least 10% packet loss, high lag and a 650MiB/month quota (traffic inside the uni network isn't counted). Bozos who don't get the rules get blocked at the inhouse switch.
If they'd try to ban PCs they'd get only one thing: open revolt. I mean the stuff with burning administration buildings. Literally. Plus it'd be mostly unenforceable in countries with things like individual freedom. Oh, there's also the need to at least quadruple the number of terminals across the campus.
Fight hunger. Filet a politician and send him to a 3rd world country of your choice.
In our residence halls, we have about 7500 people. What we have done is make a series of VLANs, centrally administered by VMPS. We have the regular VLAN for a building's users, a quarantine VLAN, and a blackhole VLAN. As we detect users that are infected, we move them to the quarantine VLAN where we have colocated a quarantine webserver via an 802.1q trunk. This server provides them with all the patches, av software and latest DATs. Once installed, the resident "signs" with their campus ID to verify that they have installed the various fixes, and they are moved back. If someone languishes in the quarantine VLAN for too long, we move them to the blackhole VLAN (which is essentially a defined VLAN that isn't trunked anywhere so VMPS can still legally place them there).
This segmentation has helped dramatically. At one point, we were blocking nearly 800,000 icmp echo requests outbound/sec across all interfaces. Now? around 1k/sec. And that's over the last week.
Now if I could just get past the residents who:
1. Don't fix themselves because it was too much to read.
2. Don't know how to use a web browser
3. Don't know what a scroll bar is (!!!)
4. Don't contact us for help, but instead go to the President and Provost's offices.
Hang in there, segmentation helps dramatically.
Naturally, if you're the BOFH type of network admin you can skip the first part
MAC address filtering would bring out at least one privacy advocate complaining about rights, and absolute Nazi like controls won't fly at a public institution.
Everybody seems to be advocating the staff doing stuff, do they have the resources to handle every little issue a student comes up with?
VLANs with heavily controlled QoS would help. I also like a script forcing certain patches.
Could the school get a license from an AntiViri company to cover all students, force everybody to run it as policy, script the updates, IDS to ban infractions by switch port or something with would f%$k the student because it might take a week to get around to turning the port back on.
Kevin
Irrational Diversions
running Mac OS X and I haven't had to lift a finger to do much of anything for more than a year
That's what I call a boring life. Compare this to the action packed life of a Windows(tm) Admin. I can imagine the next Microsoft tagline:
Windows: Bringing Unlimited Action to bored System Admins, since 1981.
getSexySig();
You never played the lottery? Let me ask you another question.
Do you have any kind of insurance?
But surely you know that, like a lottery, insurance works because on average people pay more money into it than they receive from it. Lotteries and insurance are both gambles... except that in a lottery, you bet on good fortune. With insurance, you bet against bad fortune. In both cases, the expectancy value is less than 1, but in both cases you'll be damn glad you subscribed when your number's up.
I know I know, it's just a joke. Well, I just had to get this off my chest.
If construction was anything like programming, an incorrectly fitted lock would bring down the entire building...
I'm a student PC/Net tech at a small college (1500 students, 400 staff/admin/faculty). We use an AD domain to corral our users, so to speak.
We did some testing with the Blaster patch before we encouraged our users to download it; I always check Bugtraq, personally, before I put anything on a machine I'm responsible for. Once we decided it wasn't breaking anything (at least it didn't break anything for us) we burned it to a whole bunch of CDs (with the Symantec removal tool, the Win2k patch, the WinXP patch, and the WinNT fix). Each RA/helpkid/tech also got a corporate edition of NortonAV on a disk (we have a site license) with instructions for students on how to update their virus definitions.
Each RA got this disk. Each help desk kid (there are about 15 student help desk kids) got one, and the other five PC/net techs (other than me) got one. We marched around campus for about a week wearing very visible "TECHNOLOGY SOLUTIONS CENTER" T-shirts and essentially infiltrated dorm life with our antivirus software.
Were there huge network slowdowns? Oh yeah. For the first day and a half when students came back there was little, if any, network connectivity. But the RAs were adamant about having the kids run the patches and install NAV. Did we use guerilla tactics, like disabling network ports or confiscating network cable? No, not at all. We just made help extremely visible, and with a horde of student tech workers getting $5/hr, it was not so bad for cheap labor for the college, either.
You might bitch and moan and say that a college kid with a virus will never go talk to his RA, but we had mandatory floor meetings for every floor for every hall across campus, and when you've got 20 kids and one RA, it's pretty easy to reach the end users. Users only understand that "my computer doesnt work", and you can bet that a college kid at a small, tech-oriented campus will go see his RA if he knows his RA can help him. (If the kids think the RAs are totally bogus, then there's problems with administration that have nothing to do with computing and is for another thread entirely.)
Do these tactics make Mac/Linux users feel discriminated against? I saw some whining in the comments about this, but guess what: Even if an RA is minimally intelligent in the realm of computing, he can PROBABLY tell a Mac from a PC. Mac users get left alone (like me.)
Full network connectivity returned at about 9 in the morning on the day after move-in. (you'd be surprised how fast 30 RAs and 21 tech kids can move.)
You might also bitch and moan and say that students shouldn't have L2 domain admins. Okay, I can understand that. One kid got forcibly removed from our staff last year for leeching software off a drive he had permissions to, so no, it's not a completely perfect solution, and a lot of trust is involved. But it worked okay for us and minimized a lot of headaches.
Angry IT woman in big clompy boots. And talking lint!.
Not really an option. And an incorrectly managed linux machine on an academic network can be almost as big threat to the outer world as windows. I am speaking out of experience as I have dealt with OC3+ floods coming from zombies in student dorms long before people started to apply "voodoo" to windows machines. It was linux, bsd, solaris and other unix systems in those (pre BO) times. Quite oftent it still is.
Still, you can very easily deal with it.
1. Move dorms to private addresses so that you do not have an address space constraint as the next step will eat addresses like there is no tomorrow.
2. Subnet the network into a small salad and put each slice of the salad into a separate VLAN.
3. 802.1q the vlans up to a linux box, bsd box or a cisco that has enough grunt to filter (72xx VXR or similar comes to mind, bigger ones have a hard time filtering, smaller ones cannot handle the bandwidth).
4. Filter on all 802.1q interfaces on the linux/bsd/cisco.
As a result you contain any clap to a small subnet.
Note that everybody will hate you initially. People definitely did hate me 8+ years ago as this was one of the things I did to deal with a similar problem (one dept in the building I managed was being hacked left right and center).
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
at my University, they've started to do that. If your machine is spitting out garbage they kill your connection and call (e-mail) whoever is responsible for maintaing the system and notify them that they need to get the problem fixed before their IP will become active again.
We havn't done it in our lab (there are multiple on campus) yet as there's no impending doom if we don't, but we're looking to secure our work area with a router that blocks all ports and then use 192.168.0.* IPs behind it. Which allows us to fresh install Windows or whatever and not have to worry about getting infected before we can get them up to date.
It'd be trivial for a University to setup such an area and if a user is trouble, kill their connection and call them and tell them to bring down their system to the secured lab to be patched and fixed.
My home network which has every flavor of Windows running was completely unaffected by the Blaster worm simply because I run a router intelligently.
It's really not that hard to not get infected.
Ben
Work Safe Porn
My solution is not very large scale (only 240 ports), but works quite well. A 486 machine on top of every switch running tcpdump filtered through a perl script that uses snmp to shut down the offending port as soon as any 'suspicious' traffic starts to flow from it. The 486's are setup to netboot with the loader on CD (or floppies for the few machines that don't support CD boot), and all share the same NFS server, making managment a snap.
Of course this only works if you have managed switches/hubs, a bunch of spare 486's (pentiums would be better) and a day or so to set it up. The nice thing is that if the 486 fails (only one has so far), the network stays up.
This has stopped 99% of malicious traffic dead in it's tracks.