Slashdot Mirror
Handling User Grown Machines on a Large Network?
matth asks: "Recently with the outbreak of the MSBLASTER worm and the startup of the college semester here in the US we've been hit by a big problem here where I work. Many students are bringing in machines from home, often times infected. The infections are so bad that they bring the whole network to a crawl. Yes, you can install ACLs on edge routers and put a router between the dorms and the rest of your network, but it still brings the dorm to a crawl. You can make sure people install the patches, but what if someone re-installs Windows, or brings in another machine, and what about NEXT year? From the Slashdot community, how have sysadmins out there dealt with this? How can you manage each machine in a network such as a college, where people are bringing their own machines in from the outside? ACLs on routers... but what about for the segmented network?"
Windows? I am seriously considering moving my smaller clients to Mac of Linux pretty soon, I'm drawing up the proposals today.
You can only separate networks so much.
If you make them bear some financial responsibility for not checking their machines first this might help.
Time to diversify so that the target infestation isn't as large. But you can't tell people what OS to run, so as for protecting the network, not allowing email attachments is pretty harsh to some people, but I think it's what will need to be done in the long run.
Email should be used for communication, not for transfering files.
CB
free ipod and free gmail!
I think this was one of the approaches Stanford was going to take. No DNS for your machine until you get it checked out by their IT department.
Chris
... from another point of view.
I'm a student at a university whose dorm network got nailed by blaster something fierce. Almost as bad as it was Klezed a couple years before. Anyways, because of all of this, the sys admins decided to completely eliminate the dorm network from the upper campus one - also cutting off 'net access - during school hours. This is a real big pain in the butt, and I'm actually hoping there are some great answers in this topic so I can give them to my sys admin.
Of course, compounding the situation are seemingly (dunno if they actually are or not considering I've never even SEEN one before) incompetant dorm techs taking an entire day to clear out just one dorm building of ~50 rooms (2 people per room, but often less than 2 PCs per room...). Considering Blaster only affects 2000/XP/2003 machines, that means that the roughly 50 computers running those took 8 hours to clean? Something seems wrong here.
I'm just annoyed because my room (along with my entire hall since I'm the resident 'hey, call him!' computer geek and have patched everyone) is completely free of blaster and its ilk, yet I have to deal with the people who either don't know to patch Windows often, or don't care.
How about this one: What can a STUDENT at one of these schools do to help? I've tried teaching as many people as possible about computer safety (take a health classes' STD safety course, apply to computers basically), and I'm ineligable to become a dorm tech right now... anyone?
--- Ãther SPOON!
Seeing as in this situation you wont be able to convince your students to switch:
1) Require all machines to register their mac address via nice gui or website. This way when you use all the rest of the stuff mentioned here (snort, etc) you can easily track the student down.
2) Run snort, router, acls, etc in a way to automatically blocks infected users. Or at the very least it should at least alert you of them. But blocking is best so that they dont spread the infection further on your network or to the internet via your fat pipe.
3) Buy a site license of the managed versions of Norton Antivirus for the dorms and hand one to every student as they walk in the door. Once they've installed it you can force the updates on to them.
As the systems admin who will test those patches in a test lab before rolling them out to people, you will make sure that will not happen if you valuie that paycheck. Blindly checking off security updates for addition to the network is studipity no matter what the platform, wther you use up2date or MS AutoUpdate. For MS systems, having a SUS server helps centralize this process since you check off what you authorize to get pushed to the network. Active Directory policies can enforce this. Those that don't want to play in the domain can piss off. If you want to keep them off the network, there is always 802.1x.
"Beware of he who would deny you access to information, for in his heart, he dreams himself your master."
... when you go to a university where you do not log on to a domain in dorms.
I've found that to be very common (including the Uni that I'm typing this at) since it is MUCH easier to set freshman up on movein day.
Also, certain things do not work when you start logging onto domains. Example: XP's fast user switching. You'd have students complaining about the administration restricting their rights to their own computer, blah blah blah... then on top of it, automatically patching something. Legal nightmare. Works great for lab PCs, horrid for dorm PCs.
--- Ãther SPOON!
I hadn't thought of this implication. Unfortunately, it's not feasible to force the users to do anything in this kind of situation - that would be an administrator's nightmare.
I'm assuming you have each computer connected to a central switch, right? What I would do is block all communication between the PCs on the network. Allow each one to get out to the internet through the firewall, but block them from connecting to each other. That would give them the ability to browse the web, check email, instant message, etc., without needing to worry about them setting up servers, file sharing, and trading viruses, etc., between each other. It's heavy handed, but at least you're still providing the service you're supposed to (internet connectivity).
Just a thought. I'm not completely sure this is even feasible with a switch, but I would think so.
"I have never let my schooling interfere with my education." - Mark Twain
I know of at least one school in my area taking a tighter approach- no machines have their access to the network turned on until they've been personally looked at by a support tech. Long delays, obviously, but at least nothing should get by.
For years, the last thing the admins at my university wanted to do was inspect each computer before it was permitted to be on the network. This year they have broken down and are doing so, to be connected (wired or wirelessly) one of their employees must inspect the computer and make sure that they are not only completely patched, but also that they are running antiviral software (Norton ONLY).
This is of course great in theory, until a week later when someone formats, 'forgets' to patch, brings their computer home, gets re-infected and comes back to school.
Until patches become mandatory for many of these users, there is no way to prevent such a thing... short of finding the virus writers and skinning them alive during prime time, that might make some of these script kiddies think twice before doing what they do.
Help Brendan pay off his student loans
I think that this is the perfect environment for an anti-worm. If the spread of such a worm was limited to the college's netblock, it could be easily controlled (luckily computer viruses don't spontaneously mutate) and it could be set to download all needed patches from a campus server, and destroy itself on command from the same server. Something like this could also be worthwhile on corporate networks. Why haven't antivirus companies caught on to this? They could sell customized anti-worms to small-to-medium size network owners. The problems of releasing an anti-worm on the Internet at large don't apply to smaller networks. You can get the permission of all the network admins before releasing the worm, and a central server can be used to control the infection, keeping track of which computers are patched and shutting down the worm when it has done its job.
main(c,r){for(r=32;r;) printf(++c>31?c=!r--,"\n":c<r?" ":~c&r?" `":" #");}
I doubt this was what the original poster meant though.
In our residence halls, we have about 7500 people. What we have done is make a series of VLANs, centrally administered by VMPS. We have the regular VLAN for a building's users, a quarantine VLAN, and a blackhole VLAN. As we detect users that are infected, we move them to the quarantine VLAN where we have colocated a quarantine webserver via an 802.1q trunk. This server provides them with all the patches, av software and latest DATs. Once installed, the resident "signs" with their campus ID to verify that they have installed the various fixes, and they are moved back. If someone languishes in the quarantine VLAN for too long, we move them to the blackhole VLAN (which is essentially a defined VLAN that isn't trunked anywhere so VMPS can still legally place them there).
This segmentation has helped dramatically. At one point, we were blocking nearly 800,000 icmp echo requests outbound/sec across all interfaces. Now? around 1k/sec. And that's over the last week.
Now if I could just get past the residents who:
1. Don't fix themselves because it was too much to read.
2. Don't know how to use a web browser
3. Don't know what a scroll bar is (!!!)
4. Don't contact us for help, but instead go to the President and Provost's offices.
Hang in there, segmentation helps dramatically.
Set up a dhcp/iptables/ LINUX firewall . I run a script that monitors the net for a rush of packets (ICMP/port 135/smurf attack) it works great! heres the algorithm in pseudocode - any net admin should be able to put it together. You basically monitor 1000 packets and count the number packets per host and find the packet count per time then dump if they are pushing 90% or more packets while (true) do t0 = timeinseconds packetlist = tcpdump -n -i -c1000 t1 = timeinseconds iplist = grep list|print ipfield| uniq -c totalscanseconds = t1-t0 totalpackets = count(packetlist) if totalpackets greater than 99% iptables -t -nat -A PREROUTING -s offendingip -d 0/0 --dport 80 -j DNAT --todestination and viola! all users flooding the net are automatically forwarded to a you are quarantine website no matter what. All packets are dumped before they go any further. I can handle easily 500 - 700 connections with a dual AMD 1800 cpu / 500meg ram dual nics setup as a dhcp server
Naturally, if you're the BOFH type of network admin you can skip the first part
Basically what we've done is burn a shitload of CD's with the Blaster patch on them, given them out to people with the worm and then encouraged them to distribute the CD's to their friends. We've also given those CD's to our local residential hall tech support people (the ones who actually go to the person's room and fix whatever problem; they are assigned by dorm).
:)
Recently, we've begun deactivated the ports of people who we've been able to trace the worm back to, having them call us, pick up the CD, install the patch and then having an RCC verify that the patch is installed before reactivating their ports. We've also closed off the ports that the worm is known to propagate through. We've still taken damage as a result of it, but I think we've managed to minimize it somewhat. In the meantime, I've been trying to convince the Mac users I support that they're not at risk. If you say, "impossible" enough times in a row, they start believing you.
When I was in the dorms, we had a really slow network, mainly because it was in the height of file-sharing. I used ntop and other network tools to find out who was using up all of our bandwidth with movie-sharing, and then organized a posse. One time a poor guy opened his door to find 20 of us telling him to be more considerate or else.
We wouldn't have done anything to him, but network performance went up a little.
Anyway, I think that the list-posting idea is ten times better than any of the other suggestions I've heard so far.
MAC address filtering would bring out at least one privacy advocate complaining about rights, and absolute Nazi like controls won't fly at a public institution.
Everybody seems to be advocating the staff doing stuff, do they have the resources to handle every little issue a student comes up with?
VLANs with heavily controlled QoS would help. I also like a script forcing certain patches.
Could the school get a license from an AntiViri company to cover all students, force everybody to run it as policy, script the updates, IDS to ban infractions by switch port or something with would f%$k the student because it might take a week to get around to turning the port back on.
Kevin
Irrational Diversions
That's when you set forth the rules.
Windows 2000/XP only, if it's a Windows environment, or MAC otherwise. Any machines found online that violate the policy will be denied access, and the violaters fined.
I know of a couple of small colleges that are MAC only; they don't support Windows machines of any kind. To ensure this, you buy the computer when you start your term--it's part of your tuition and fees. This way, no one brings in anything unauthorized from home.
In America today you can murder land for private profit. You can leave the corpse for all to see, and nobody calls the c
here at Oklahoma State University, the IT department gave all the RAs in all the dorms and apartments a fix-it CD, all users must run the software on the CD regardless of whether they don't think they have msblast/sobig, etc.
Case in point -- back in 2000, even though I had about four years Linux experience by then, I managed to bring down Internet access for an entire dorm (about 900 students) for a week.
:^)
It all started when I helped a friend install Linux on his new computer. Unfortunately, in addition to installing a DHCP client on his machine, I had accidentally flagged the DHCP server to install as well. What happened was that the DHCP server software on his new Linux box was challenging the Windows DHCP server that the dorm was using, and his machine won -- even though his DHCP server wasn't properly configured to hand out IP addresses to other clients. So, all of these other 900 students would turn on their computers, which would send out a DHCP request, and they would get a response from his computer instead of the real DHCP server, thus causing their computers to give up trying to connect to the network. Ironically enough, his computer connected to the internet fine, as it was the only one connecting to the real DHCP server (I guess that explains his super-fast connection during that week).
Anyway, we had no idea that any of this was happening until we headed back to his dorm room one day, and found three network services guys looking in bewilderment at the computer (they had never used anything but Windows, so they had no idea how to fix it). They claimed that it took them a week to isolate the problem to his machine. They explained what was happening, and it then hit me that the DHCP server was also running on his machine, so I logged in, apt-get removed it, and the problem was immediately fixed. Not in their eyes though, as they made us talk to the head guy at network services... He gave us fair warning that if we did that again, our access to the network would be revoked (and rightly so!).
The obvious moral of the story is, whereas most OSes give you just enough rope to tie a knot, Linux gives you enough rope to hang about 900 people.
Slashdot's first reaction to VMware
I know it's a pain to lose ping functionality, but in the case of Nachia, the fastest way to stop it is to put a filter on your switch. If you use Cisco 65xx's with the Policy Feature Card, you can run the following commands:
set security acl ip WORM deny icmp any any echo
set security acl ip WORM permit ip any any
commit security acl WORM
set security acl map WORM 1 (or whatever VLANs you have)
If you have some other product for LAN switches, shame on you! Well, there probably is a similar filtering capability if you have the right components.
I've been involved in cleaning up after SQLslammer and Nachia on a rather large network. In both cases, I found that router filters were difficult to implement without causing the filters to kill the routers (except on a few very new high-end routers). The PFC claims to work at wire speed. In practice, I've had a hard time proving them wrong on that.
This filtering technique will allow you to drop packets as soon as they enter the switch. Basically your doing a L3 or even a L4/L5 filter (tcp/udp with port) on a device that is really operating at L2.
A couple things to note, you can't log the packets and once you put the filter in place you probably won't be able to determine who is sending junk, but you shouldn't be patching machines for a worm by going after the infected ones... every machine in the network needs patched before you lift filters regardless of whether the worm is still in your network or not. If not, it will be back!
I know the parent was meant to be funny, but believe it or not, that's what my school did. They unregistered all cards from their DHCP database and are requiring everyone to re-register on condition of passing a brief virus scan to get back on the network. Our network is set up to disallow external routing for any not-registered machines.
I guess that's what they get for forcing everyone to migrate to XP last year...
"Quoting famous computer scientists out of context is the root of all evil (or at least most of it) in programming." - K
I don't work for MS or Symantec. I do work with over 700 different end users every year. I was mearly posting an option. It's not hard to have the user login to an AD domain, then hand out a major security update. If the user is on your network, wouldn't you want them to be secure? If I owned and ran a major campus network, I would only let users join the network on my terms. This doesn't have to be done by using an Active Directory domain, but should be done and noted that it will be done if it is. One way to look at it is the physical security on a major campus. Lets say that students need to use an ID card to enter and exit a building (domain). If they don't feel they should have to use the ID card, then I don't feel they should have to attend school there. Now, lets say a major health issue came up. Wouldn't you, as a school, want to force all the students to get some sort of immunity? If you don't make them get an immunity it will be a bigger problem. That's that. I don't know... maybe my point was lost, maybe not. I'm tired and typing and that's not a good combination.
You talk better than you fool!
...of worms, but because massive ammounts or pr0n is being downloaded and uploaded... There is so much pr0n I don't know when they would have time on thier hands... (no pun intended) ...to get infected with a virus or worm...
I'm a student PC/Net tech at a small college (1500 students, 400 staff/admin/faculty). We use an AD domain to corral our users, so to speak.
We did some testing with the Blaster patch before we encouraged our users to download it; I always check Bugtraq, personally, before I put anything on a machine I'm responsible for. Once we decided it wasn't breaking anything (at least it didn't break anything for us) we burned it to a whole bunch of CDs (with the Symantec removal tool, the Win2k patch, the WinXP patch, and the WinNT fix). Each RA/helpkid/tech also got a corporate edition of NortonAV on a disk (we have a site license) with instructions for students on how to update their virus definitions.
Each RA got this disk. Each help desk kid (there are about 15 student help desk kids) got one, and the other five PC/net techs (other than me) got one. We marched around campus for about a week wearing very visible "TECHNOLOGY SOLUTIONS CENTER" T-shirts and essentially infiltrated dorm life with our antivirus software.
Were there huge network slowdowns? Oh yeah. For the first day and a half when students came back there was little, if any, network connectivity. But the RAs were adamant about having the kids run the patches and install NAV. Did we use guerilla tactics, like disabling network ports or confiscating network cable? No, not at all. We just made help extremely visible, and with a horde of student tech workers getting $5/hr, it was not so bad for cheap labor for the college, either.
You might bitch and moan and say that a college kid with a virus will never go talk to his RA, but we had mandatory floor meetings for every floor for every hall across campus, and when you've got 20 kids and one RA, it's pretty easy to reach the end users. Users only understand that "my computer doesnt work", and you can bet that a college kid at a small, tech-oriented campus will go see his RA if he knows his RA can help him. (If the kids think the RAs are totally bogus, then there's problems with administration that have nothing to do with computing and is for another thread entirely.)
Do these tactics make Mac/Linux users feel discriminated against? I saw some whining in the comments about this, but guess what: Even if an RA is minimally intelligent in the realm of computing, he can PROBABLY tell a Mac from a PC. Mac users get left alone (like me.)
Full network connectivity returned at about 9 in the morning on the day after move-in. (you'd be surprised how fast 30 RAs and 21 tech kids can move.)
You might also bitch and moan and say that students shouldn't have L2 domain admins. Okay, I can understand that. One kid got forcibly removed from our staff last year for leeching software off a drive he had permissions to, so no, it's not a completely perfect solution, and a lot of trust is involved. But it worked okay for us and minimized a lot of headaches.
Angry IT woman in big clompy boots. And talking lint!.
This gives us the following benefits:
1. Only machines we want to have on our network are there. This usually means that we give out IP addresses in exchange for the basics - a MAC address and the location of the machine. Higher levels of management of clients has its costs, so that'll be down to the individual manager to decide (for instance - only machines running OS xyz, or only machines we have root/admin access to, only machines built with our spec/OS and connected to our auto patching architecture, etc).
This means that we can, in extreme cases, remove someone from the DHCP lists, and flag their MAC up in arpwatch. In the case of "students arriving at the start of term", there is quite a flood of applications at the start of term - combined with teaching them how to find their mac address (solved with a flier in their matriculation pack). After that, it slows to a trickle of applications.
2.With managed switches (and really, who DOESN'T use managed switches in large networks?) troublemakers can be sought and disconnected in times of strife. You have their IP address AND know which switch/port they're on (through the MAC/location registration process). It really is up to the user to come to the IT staff in the event that their connection drops. We have disabled specific ports on network switches in some cases, which is a far more useful solution than removing DHCP entries, but for public areas the DHCP block is what is needed (laptops in libraries for instance). Smart users will get around this, but it's not the smart users you're worried about. They know how to patch.
When it comes down to it, make one simple rule - network access is a priveledge, not a right. Our entire university wide IT infrastructure is built on this philosophy, and as a result the onus is on your users to behave in a responsible way.
--
Why can't we all just get along?
I just saw a presentation on a campus-wide wireless network.
Because you cannot control who uses the wireless zone, it's treated as potentially hostile or untrusted and users must authenticate to a VPN.
A nice side-effect of this is that the VPN in Windows routes all traffic via the VPN, letting them apply all sorts of policies "port 4444, I don't think so...". Blaster only affected users silly enough to bring in an infected machine.
Perhaps a similar setup for the untrusted wired network too?
"Everything is adjustable, provided you have the right tools"
A slightly less draconian measure which my school has taken is, upon detecting virus activity from a given computer on the network, it is removed from the DHCP database and kicked from the network. The owner of the computer is then notified that their computer is infected with a virus and not allowed to reconnect to the network until they have demonstrated the problem is fixed. One should note that our network has on the order of 50,000 computers attached to it, so this is definetely a scriptable solution. Also, this allows for a mixed computing environment.
There's no sig like SIGSEG
It's amazing how many students seem to have wiring problems after they crash the local nets on certain campuses. I just wish the same approach could be applied to home users.
Many of the worms and viruses that bog the net have had patches for months or even years. I say if the patch was out three months ago, cut the user off at their ISP -- permanently.
You can't drive without a license -- if you can't update, you don't know how to "drive" the internet. And no, I really don't care about the "rights" of the brain-dead to access public resources.
Even my techno-illiterate parents know enough to keep the virus files and patches up to date -- because they were taught before the machine was ever plugged in to the 'net.
I do not fail; I succeed at finding out what does not work.