Slashdot Mirror
Handling User Grown Machines on a Large Network?
matth asks: "Recently with the outbreak of the MSBLASTER worm and the startup of the college semester here in the US we've been hit by a big problem here where I work. Many students are bringing in machines from home, often times infected. The infections are so bad that they bring the whole network to a crawl. Yes, you can install ACLs on edge routers and put a router between the dorms and the rest of your network, but it still brings the dorm to a crawl. You can make sure people install the patches, but what if someone re-installs Windows, or brings in another machine, and what about NEXT year? From the Slashdot community, how have sysadmins out there dealt with this? How can you manage each machine in a network such as a college, where people are bringing their own machines in from the outside? ACLs on routers... but what about for the segmented network?"
You can only separate networks so much.
If you make them bear some financial responsibility for not checking their machines first this might help.
I think this was one of the approaches Stanford was going to take. No DNS for your machine until you get it checked out by their IT department.
Chris
... from another point of view.
I'm a student at a university whose dorm network got nailed by blaster something fierce. Almost as bad as it was Klezed a couple years before. Anyways, because of all of this, the sys admins decided to completely eliminate the dorm network from the upper campus one - also cutting off 'net access - during school hours. This is a real big pain in the butt, and I'm actually hoping there are some great answers in this topic so I can give them to my sys admin.
Of course, compounding the situation are seemingly (dunno if they actually are or not considering I've never even SEEN one before) incompetant dorm techs taking an entire day to clear out just one dorm building of ~50 rooms (2 people per room, but often less than 2 PCs per room...). Considering Blaster only affects 2000/XP/2003 machines, that means that the roughly 50 computers running those took 8 hours to clean? Something seems wrong here.
I'm just annoyed because my room (along with my entire hall since I'm the resident 'hey, call him!' computer geek and have patched everyone) is completely free of blaster and its ilk, yet I have to deal with the people who either don't know to patch Windows often, or don't care.
How about this one: What can a STUDENT at one of these schools do to help? I've tried teaching as many people as possible about computer safety (take a health classes' STD safety course, apply to computers basically), and I'm ineligable to become a dorm tech right now... anyone?
--- Ãther SPOON!
... when you go to a university where you do not log on to a domain in dorms.
I've found that to be very common (including the Uni that I'm typing this at) since it is MUCH easier to set freshman up on movein day.
Also, certain things do not work when you start logging onto domains. Example: XP's fast user switching. You'd have students complaining about the administration restricting their rights to their own computer, blah blah blah... then on top of it, automatically patching something. Legal nightmare. Works great for lab PCs, horrid for dorm PCs.
--- Ãther SPOON!
For years, the last thing the admins at my university wanted to do was inspect each computer before it was permitted to be on the network. This year they have broken down and are doing so, to be connected (wired or wirelessly) one of their employees must inspect the computer and make sure that they are not only completely patched, but also that they are running antiviral software (Norton ONLY).
This is of course great in theory, until a week later when someone formats, 'forgets' to patch, brings their computer home, gets re-infected and comes back to school.
Until patches become mandatory for many of these users, there is no way to prevent such a thing... short of finding the virus writers and skinning them alive during prime time, that might make some of these script kiddies think twice before doing what they do.
Help Brendan pay off his student loans
I think that this is the perfect environment for an anti-worm. If the spread of such a worm was limited to the college's netblock, it could be easily controlled (luckily computer viruses don't spontaneously mutate) and it could be set to download all needed patches from a campus server, and destroy itself on command from the same server. Something like this could also be worthwhile on corporate networks. Why haven't antivirus companies caught on to this? They could sell customized anti-worms to small-to-medium size network owners. The problems of releasing an anti-worm on the Internet at large don't apply to smaller networks. You can get the permission of all the network admins before releasing the worm, and a central server can be used to control the infection, keeping track of which computers are patched and shutting down the worm when it has done its job.
main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
In our residence halls, we have about 7500 people. What we have done is make a series of VLANs, centrally administered by VMPS. We have the regular VLAN for a building's users, a quarantine VLAN, and a blackhole VLAN. As we detect users that are infected, we move them to the quarantine VLAN where we have colocated a quarantine webserver via an 802.1q trunk. This server provides them with all the patches, av software and latest DATs. Once installed, the resident "signs" with their campus ID to verify that they have installed the various fixes, and they are moved back. If someone languishes in the quarantine VLAN for too long, we move them to the blackhole VLAN (which is essentially a defined VLAN that isn't trunked anywhere so VMPS can still legally place them there).
This segmentation has helped dramatically. At one point, we were blocking nearly 800,000 icmp echo requests outbound/sec across all interfaces. Now? around 1k/sec. And that's over the last week.
Now if I could just get past the residents who:
1. Don't fix themselves because it was too much to read.
2. Don't know how to use a web browser
3. Don't know what a scroll bar is (!!!)
4. Don't contact us for help, but instead go to the President and Provost's offices.
Hang in there, segmentation helps dramatically.
Naturally, if you're the BOFH type of network admin you can skip the first part
MAC address filtering would bring out at least one privacy advocate complaining about rights, and absolute Nazi like controls won't fly at a public institution.
Everybody seems to be advocating the staff doing stuff, do they have the resources to handle every little issue a student comes up with?
VLANs with heavily controlled QoS would help. I also like a script forcing certain patches.
Could the school get a license from an AntiViri company to cover all students, force everybody to run it as policy, script the updates, IDS to ban infractions by switch port or something with would f%$k the student because it might take a week to get around to turning the port back on.
Kevin
Irrational Diversions
That's when you set forth the rules.
Windows 2000/XP only, if it's a Windows environment, or MAC otherwise. Any machines found online that violate the policy will be denied access, and the violaters fined.
I know of a couple of small colleges that are MAC only; they don't support Windows machines of any kind. To ensure this, you buy the computer when you start your term--it's part of your tuition and fees. This way, no one brings in anything unauthorized from home.
In America today you can murder land for private profit. You can leave the corpse for all to see, and nobody calls the c
here at Oklahoma State University, the IT department gave all the RAs in all the dorms and apartments a fix-it CD, all users must run the software on the CD regardless of whether they don't think they have msblast/sobig, etc.
Case in point -- back in 2000, even though I had about four years Linux experience by then, I managed to bring down Internet access for an entire dorm (about 900 students) for a week.
:^)
It all started when I helped a friend install Linux on his new computer. Unfortunately, in addition to installing a DHCP client on his machine, I had accidentally flagged the DHCP server to install as well. What happened was that the DHCP server software on his new Linux box was challenging the Windows DHCP server that the dorm was using, and his machine won -- even though his DHCP server wasn't properly configured to hand out IP addresses to other clients. So, all of these other 900 students would turn on their computers, which would send out a DHCP request, and they would get a response from his computer instead of the real DHCP server, thus causing their computers to give up trying to connect to the network. Ironically enough, his computer connected to the internet fine, as it was the only one connecting to the real DHCP server (I guess that explains his super-fast connection during that week).
Anyway, we had no idea that any of this was happening until we headed back to his dorm room one day, and found three network services guys looking in bewilderment at the computer (they had never used anything but Windows, so they had no idea how to fix it). They claimed that it took them a week to isolate the problem to his machine. They explained what was happening, and it then hit me that the DHCP server was also running on his machine, so I logged in, apt-get removed it, and the problem was immediately fixed. Not in their eyes though, as they made us talk to the head guy at network services... He gave us fair warning that if we did that again, our access to the network would be revoked (and rightly so!).
The obvious moral of the story is, whereas most OSes give you just enough rope to tie a knot, Linux gives you enough rope to hang about 900 people.
Slashdot's first reaction to VMware
I know the parent was meant to be funny, but believe it or not, that's what my school did. They unregistered all cards from their DHCP database and are requiring everyone to re-register on condition of passing a brief virus scan to get back on the network. Our network is set up to disallow external routing for any not-registered machines.
I guess that's what they get for forcing everyone to migrate to XP last year...
"Quoting famous computer scientists out of context is the root of all evil (or at least most of it) in programming." - K
I'm a student PC/Net tech at a small college (1500 students, 400 staff/admin/faculty). We use an AD domain to corral our users, so to speak.
We did some testing with the Blaster patch before we encouraged our users to download it; I always check Bugtraq, personally, before I put anything on a machine I'm responsible for. Once we decided it wasn't breaking anything (at least it didn't break anything for us) we burned it to a whole bunch of CDs (with the Symantec removal tool, the Win2k patch, the WinXP patch, and the WinNT fix). Each RA/helpkid/tech also got a corporate edition of NortonAV on a disk (we have a site license) with instructions for students on how to update their virus definitions.
Each RA got this disk. Each help desk kid (there are about 15 student help desk kids) got one, and the other five PC/net techs (other than me) got one. We marched around campus for about a week wearing very visible "TECHNOLOGY SOLUTIONS CENTER" T-shirts and essentially infiltrated dorm life with our antivirus software.
Were there huge network slowdowns? Oh yeah. For the first day and a half when students came back there was little, if any, network connectivity. But the RAs were adamant about having the kids run the patches and install NAV. Did we use guerilla tactics, like disabling network ports or confiscating network cable? No, not at all. We just made help extremely visible, and with a horde of student tech workers getting $5/hr, it was not so bad for cheap labor for the college, either.
You might bitch and moan and say that a college kid with a virus will never go talk to his RA, but we had mandatory floor meetings for every floor for every hall across campus, and when you've got 20 kids and one RA, it's pretty easy to reach the end users. Users only understand that "my computer doesnt work", and you can bet that a college kid at a small, tech-oriented campus will go see his RA if he knows his RA can help him. (If the kids think the RAs are totally bogus, then there's problems with administration that have nothing to do with computing and is for another thread entirely.)
Do these tactics make Mac/Linux users feel discriminated against? I saw some whining in the comments about this, but guess what: Even if an RA is minimally intelligent in the realm of computing, he can PROBABLY tell a Mac from a PC. Mac users get left alone (like me.)
Full network connectivity returned at about 9 in the morning on the day after move-in. (you'd be surprised how fast 30 RAs and 21 tech kids can move.)
You might also bitch and moan and say that students shouldn't have L2 domain admins. Okay, I can understand that. One kid got forcibly removed from our staff last year for leeching software off a drive he had permissions to, so no, it's not a completely perfect solution, and a lot of trust is involved. But it worked okay for us and minimized a lot of headaches.
Angry IT woman in big clompy boots. And talking lint!.
My solution is not very large scale (only 240 ports), but works quite well. A 486 machine on top of every switch running tcpdump filtered through a perl script that uses snmp to shut down the offending port as soon as any 'suspicious' traffic starts to flow from it. The 486's are setup to netboot with the loader on CD (or floppies for the few machines that don't support CD boot), and all share the same NFS server, making managment a snap.
Of course this only works if you have managed switches/hubs, a bunch of spare 486's (pentiums would be better) and a day or so to set it up. The nice thing is that if the 486 fails (only one has so far), the network stays up.
This has stopped 99% of malicious traffic dead in it's tracks.