Slashdot Mirror
Curing a Corporate Virus Infection
museumpeace writes "Over at Internet Storm Center Deb Hale's 'In search of the bot net' entry for September 25 recounts a grueling hunt for all the .exe's, reg entries and sources for a bot infection of a 60 server corporate network. What a nightmare! The story ends with an indictment of careless users and a suspicion that Ares, one of the sloppier Pirate2Pirate filesharing tools was the original souce of the extensive corruption that eventually even crippled the AV tools. How typical is this sort of grief? [More more frequent than reported, I would expect: the corporate victim demanded anonymity for the story to be told]."
Only slightly biased. I understand the annoyance of the admins over this screwup, but take deep breaths and count to 10 before you badmouth all P2P networks.
And security always includes usage policies.
$ su -
# uname
Linux
# iptables -P INPUT -j DENY
# iptables -A INPUT -m state --state=ESTABLISHED,RELATED -j ACCEPT
# exit
$
I want to delete my account but Slashdot doesn't allow it.
If you can use the 'protected port' option on e.g. cisco switches, TURN IT ON.
Essentially, it prevents the indicated ports on the switch from communicating with other ports that also have that protection set. Unless you have sloppy shared directories or some reason for the actual PC's to talk directly to eachother, it won't harm anything and will prevent the viruses from spreading pc-to-pc once (not when) they get in.
[disclaimer: i work for a major fortune 500 company with a large, 50+ distributed node WAN]
Everytime there's a big ass Windows vulnerabilty, there are security emails and IT manager emails basically saying "heads up, check your shit." But let's say somebody doesnt check his shit, and a site ends up infected. The WAN group watches the network, especially during times like this, and nodes are just dropped off routing from the rest of the network until they get their act back together.
I realize the article is talking more about the pains of these nasty new infections that mutilate machines, but the old saying works -- a good offense is a great defense. Assign local managers responsibility for the server boxen at their node, he/she should be keeping the machines patched, but when that fails, close the node off the network before it can damage anywhere else.
Of course the major server boxen have their own layer of network between them and the rest of the WAN, so they can be isolated if the worm is already rampant on the network. Doesn't hurt to access list transmission ports, either, icmp, tftp, foo...
Blame your own policies, not your users. Users are not IT experts and will not be even with extensive training.
Restrict privileges. Don't allow anything that is not necessary...
...a grueling hunt for all the .exe's, reg entries and sources for a bot infection...
Wrong answer. If you have a compromised system, trying to clean it is (a) likely to be really difficult, and (b) not secure.
Wipe the system, reinstall, and recover from backups. (You do keep good backups, right?) It sounds pessimistic, but in most cases an attempt to "clean" a system is going to end up with you pulling out the OS reinstall disks anyway.
Tarsnap: Online backups for the truly paranoid
There are really times when I wish you could mod a submission as "Flamebait."
"Only slightly biased. I understand the annoyance of the admins over this screwup, but take deep breaths and count to 10 before you badmouth all P2P networks."
YEAH! Let's badmouth only the ones used to transport "pirated" material.
It took more than a week to fix. Basically IT took everything down and cleansed each individual computer before it was allowed to be back on the network ... except of course for the linux boxen and even they were affected by the lack of servers.
Since I have great respect for our IT guys (they are really scrupulous about permissions and patches), it was a sobering experience.
You needn't treat them like a threat to their face, that is just rude. Most people are "too busy" or don't care enough to learn about computer security. So nod and just listen to *their* problems and lock down their system against the big threat.
... so we set out to prevent user folly. In so doing we created the IT tech's dream.
We had to deal with this more often than not
First off you start at the network layer, and make sure via firewalls that people can't get anywhere or use any application that will cause you grief.(p2p/streaming etc.) Then you transparently proxy all your traffic so that the guy who checks out classic-cars.com all day for backgrounds can do his thing and not screw everyone else.
Then you take every user system and you lock them down. You start out by moving all their dynamic data (that you wanna keep) to a file server. Mapping the winblows appdata/my documents gives you a wannabe roaming profile without all the garbage.
After you make all that effort you either impliment a mandatory PXE re-imaging overnight (too much of a headache for us) or you use something like Deep Freeze and lock down the system entirely. Due to Deep Freeze even the most zealous surfer can only horribly damage their system once a day.
Now you have an ideal environment. All changes on a system that need a reboot *must* involve a contact to the IT department, and those you think are savvy enough not to need a frozen system can do 90% of their own support.
Ok sure so your level of responsibility goes up. The pristine environment means you have plenty of opportunity to script away your work. Not to mention silly things like virus outbreaks are really limited because a frozen system need only reboot to remove the virus.
Think *pro-active.*
- Running Windows
- Not using total security throughout the network.
- Allowing Users to download any tool that they want
- I will bet that they allow CD/floppy downloads.
- Probably allow Outlook (and in an insecure fashion).
And the Blame goes to:p2p software??????
Our society really suffers from a lack of taking blame.
Anybody who runs MS should know that it takes a lot of effort and money to truely lock it down. As such they should do so. It is simply part of the total cost of running a Windows system.
I prefer the "u" in honour as it seems to be missing these days.
Yes it is IT's fault. They let users have privilages sufficient to install programs, leading to viruses. If it were a buffer overflow in a JPEG I wouldn't blame IT.
Rules are clearly stated - enforce them or if you want to let users have more freedoms then keep and monitor detailed logs on what they do with these 'rights'.
You seem to demonstrate an immature attitude and lack of respect for users - if you are an admin you are employed because you are a specialist and it is better for you to be the single point of expertise for that task - just like you couldn't calculate the accounts for a company I doubt the finance staff would be so patronising as saying "waaa, the accountant says I can't have 3 21" LCDs, waaa, the CEO says I can't take 5 months paid leave a year".
--
Slashdot: Racism against Indians OK. China bad, USA good. Blue pill in water supply.
"Over at Internet Storm Center Deb Hale's 'In search of the bot net' entry for September 25 recounts a grueling hunt for all the .exe's, reg entries and sources for a bot infection of a 60 server corporate network. What a nightmare!" ...Apple Macs and Assorted Linucen, curing .exe, registry and bot infections for 5 years and counting!
...or does this guy come across as a total ass? "Pirate2Pirate"? Blaming the users? I mean, isn't *he* paid to enable *them* to do their jobs, not the other way around? (Of course, the actual article is /.ed, so maybe it's just the summary that gives me that impression.)
Learn from history. Government legislation against spam has done squat.
Who convinced you that they were legislating *against* spam?
CAN-SPAM: It's not just a horrible backronym.
Quantum materiae materietur marmota monax si marmota monax materiam possit materiari?
Limit access to the application/web server level at the router. Isolate workstations so that they can each see the file servers but not all other systems. If someone needs direct access to servers, they should have a real good reason (or it should be obvious; admins, developers.).
Keep in mind that I'm not suggesting that the limits be so strict that people are annoyed and attempt to break or ignore security. They should be well organized, though, and monitored. Reasonable exceptions should be made immediately, and unreasonable exceptions should be granted quickly with an eye to isolating the damage of that exception as much as possible.
A firewall can not protect you from yourself. Turn off what you do not need. Do not use the firewall to do your work.
Here is an idea that seems to slip past many...
C-O-R-P-O-R-A-T-E F-I-R-E-W-A-L-L
We used to have botnet probs in our corporate network... once we installed a Zonelabs Integrity server and were able to control what programs had access to the internet and which ones did not, it was pretty easy to fix.
Funny how it's IT fault for not getting people to follow the rules (whatever happened to self-discipline?).
Self-Discipline can be overwhelmed by rules. If you tack on all the Computer Rules to all the other rules (on Harassment, on Job-Requirements, etc) you rely on someone to remember a long list of do's and don'ts.
But a healthy admin policy will restrict the user without requiring her to remember what's acceptable and what's not acceptable, and why, and all that.
Who gives diddly what you think about your screensaver. That doesn't help you do your work.
mefus
In Open Society, GPL Software frees YOU!
Excellent. But don't forget to keep administrative control from the users and limited to the a few users.
Run security audits to make sure only the chosen few have administrator rights. This is for local PCs. Domain rights should even be more tightly controlled.
Keep AV defs updated daily. Report the numbers daily to check compliance.
Remove the ability to disable AV.
Check AV logs daily. Any report should be dispatched to a tech to "fix" the PC or determine what happened to the AV and take action accordingly.
Use group policies to ban known software, P2P & Hack/hacked tools. ( Not perfect but keeps the stupid honest)
Scan all email in & Out with AV & Spam Killer.
Be perpared to shut mail off if required to protect systems. This means you will nee to provide some user with a safe external email.
Keep your PCs patched on a regular basis. After testing on several test groups for issues.
Document your system & processes.
Inform & educate your users.
Happy to report the last big virus we had hit was Melissa. It made us retool the whole AV/Patch process and take these measures and more.
At work we have 20K users in the US alone. We actualy don't have that bad of a time dealing with viruses and worms and the like.
Why? Because 98% of the users get pushed their virus updates and their OS updates. This includes the clueless people.
We also run network scans and know WHEN computers were updated. If the computer is connnected to the network, we know what updates it has or doesn't have. The only hard part is FINDING the unpatched computers.
We also have a firewall that prevets P2P connections, FTP and anything else non web browser related (gets anoying at times).
In reading this story.. I can only assign 1% of the blame on the users and 99% of the blame on the admins for not doing a proper job.
yeah, yeah, i'm sorry, you're sorry, everybody's sorry... quit blaming your users. that aside, i think this article is a little more proof that anti-virus programs like norton, are ineffective these days. the way they function needs to be re-thought badly. i hope to see the cost of devices like this one come down to more consumer friendly levels in the future. anyone have any ideas on how anti-virus can be improved?
scott king
It's even more important. Do you want to chase problems every 5 minutes and waste your weekend? I don't!
Exactly my point!
Take one thing at a time, starting with your most troublesome group or servers. Don't grab the 300 client system nightmare first; look one server and see what it depends on. Are there 10 applications running on it? Is there a way to move one or a set of them of them off and isolated that?
If you're getting pecked to death by ducks, start by killing one duck at a time! (Or find a smaller group of ducks to kill at a new job.)
Don't let upper management know that you suceeding, though. They may want to get rid of the monkey.
A firewall can not protect you from yourself. Turn off what you do not need. Do not use the firewall to do your work.
"In a world where a private corporation could create a private bridge and set strict rules of usage for that bridge, would that private corporation be responsible for its own damages if its manager of Bridge Upkeep failed to set the readily available measures to prevent paid employees to swerve around for fun, crash through side guards and park said car next to a fresh-water lobster?"
Sounds more like this guy was just looking for an excuse to submit a story and use the term "pirate2pirate."
"Making up a new plural case of a word to try to sound cool", on "haxor.dk". That's a 15 yd. penalty and loss of down.
"Ask not what your country can do for you." --John F. Kennedy
The problem with this is that most applications for Windows don't consider the "multi-user" environment. There are a lot of apps that simply don't work well when it's not run by an Administrator account. Take for example Office 2000. I've installed this before on a Windows 2000 machine. When I run it as an Administrator, there is no problem. When I run it as a User account, it keeps asking me to insert the Office 2000 CD because there are missing components. WTF? Granted I installed it with only the features I need, but why the hell should it ask for the CD in the User account and not the Administrator account?
Another case... I used to program for a corporate environment. I was the only one who programs with conditions as to who is running the software, so I could save their data into their respective "Documents and Settings" folder, under Application Data. The rest of the developers don't care. I even set the installer to make sure only an Administrator account can install (using InnoSetup, great software).
So who's to blame? Users for running as Administrator (because they have no choice a lot of times)? Developers for not developing with multi-user environment consideration? Or Microsoft, for "hacking" Windows to become a horrible multi-user environment?
Why windows users insists using admininstrator accounts, when they could use a limited account that prevents access to the system and program dirs?
It's standard practice on a Windows network not to allow users administrator access. The only system that MS has ever released that encourages users to use administrator is XP Home, which is designed for home use, where that is probably more appropriate.
I find it highly implausible that the company described in the article here allowed their users to access administrator accounts. But then, you don't need administrator access for a trojan to launch an attack over the network and break in to other computers on it. Not in Windows, nor Linux, nor any other OS I've used.
No reference at microsoft site about using a machine in limited mode to stop viruses/trojans.
What, you mean like this one:
Microsoft recommends adopting a policy that provides the fewest privileges possible to help minimize the impact of malware that relies on exploiting user privileges when it executes.
There's a lot of corporations that refuse to report a breech in security. Simply for the reasoning that people will bail out like rats, leaving the company with little to no customer base. I suspect there's an amount of identity theft involved with the whole sordid affair, and that quite a few people make the mistake of signing up with those companies.
One day, some kid working on his thesis paper will compile a list of the IDT (IDentity Theft) victims, and there will be a nasty little coincidence...
Geez, any self respecting switch has some of those features - people should learn to use them to partition the network. On a Windoze office network, very few users need to talk to each other - most only need to talk to a server.
Oh well, what the hell...
Yes it is IT's fault. They let users have privilages[sic] sufficient to install programs, leading to viruses.
...
Ok, then whose fault is this:
IT: We need to implement $securityrule.
CEO: No.
IT: But it will prevent $securityproblem.
CEO: No.
IT:
Or this:
IT: $User violated a security rule. They should be reprimanded.
CEO: No, we don't want to piss them off.
IT: But it was in the employee handbook, and they signed a statement saying they'd follow the rule.
CEO: Get back to work, shouldn't you have a microchip to renoberate or something?
If it were a buffer overflow in a JPEG I wouldn't blame IT.
You're in a very small minority of people who actually have a working knowledge of network security. Everyone else blames IT for everything from global warming to their coffee getting cold. The mantra is "Don't understand it? It's not important. Blame IT."
Never underestimate the power of stupid people in large groups.
How much of the work is truly original? Most artists draw heavily upon a shared cultural heritage and public domain to create new works. It's a bit hypocritical to make use of that heritage and then scream "It's mine! All mine! Nobody else can ever look at it or listen to it without paying me for the privilege."
Mea navis aericumbens anguillis abundat
...because it verges on flamebait for responses that will not be entirely on topic [I thought the /. gods did a good thing recatorizing the story as IT] but the sparks have been kind of flying and I do enjoy
fireworks. The sad truth is that there are valid
points being made by both Calamormine and Quaters.
Consider how some small time software developers try
to make a living with share ware or the "free" trial
version that, if you like it but want all the bells
and whistles, you have to pony up 59.95 to get a
licencse key [and of course, those poor guys are at the mercy of people who pass around key-gen
programs]. Point being that products that benefit
from word-of-keyboard marketing CAN take advantage of pervasive sharing. You could learn a lot from
reading Dan Bricklin's article on how the right
license can make or break a small company's
fortunes.
BTW, My oldest son is a fairly creative musician but though he still spends hours per day composing or improvising, has chosen to study molecular biology, abandoning an
idea he had in high school to put his compositions up on his web site. Why? When he comes home from college, I unplug the rest of our computers from the cable modem, he plugs his laptop in so he can keep picking "stuff" up with Ares. I let him have a nice wallow in the information sewer highway and point out the keylog files on his hard drive at the end of his visit. Within a few days the weird protocol/port combinations bouncing off my firewall drop down to normal levels. Why? You have to ask someone his age I guess.
I can't tell you how fervently I wish I could make a living in a cabin off the grid with a few hot PCs and a solar powered satellite dish serving up fairly priced tricks and treats you all would not mind paying to have on your computers but I can't think of any way to protect it. I have resigned myself to working in a soulless megacorp, writing software I can't tell anyone about because megacorps have the means to get customers by the short hairs and hang on.
SLASHDOT: news for people who can't concentrate on work or have no life at all and got tired of yelling back at the TV.
Is that "Don't let (upper) management know you're succeeding" as in "Go around replacing the operating systems on your company's servers without permission?"
I don't know of many faster ways to get fired. I don't know how it is in the shop where you work (if you work in IT or ever have) but in the shops where I worked, I did not own the servers or any of the other equipment. Neither did my boss. Those things were the property of the company, and even in shops where we had incredible leeway over what we did and how we did it, going around and replacing OSes with other ones required at least approval from the CTO. That was in the liberal places. In the conservative places, approval for such things may be higher than that. When customers depend on your systems operating, stability is job one and they aren't going to allow you to take a potentially de-stabilizing action without approval. Even if you succeed in every way, you may still be fired for acting without authorization.
Now, about this time, some of you might be saying "Well, if it's stability they want, they should get *nix in and Windows out as fast as possible."
While I couldn't agree (in principle) with that sentiment more, and am glad that in my present position in email security (I miss being an admin, but I sure don't miss carrying a pager!) I am grateful that I have sufficient leeway over my tools that my workstation is one of the handful on our network that is not running Windows (Ubuntu, a Debian-based distro. Quite nice; but I digress). However, the fact remains that in any properly run shop (yes, properly run, as hard as that may be for anyone with little or no experience - especially in big operations - to accept, have controls in place is the proper way to do things), permission is required to go around re-architecting major systems and replacing OSes.
In smaller networks, the decision may go no higher than the CTO, and if further approval is formally required, whatever the CTO asks for is rubber-stamped.
In larger shops, such things will typically require a general management decision, requiring the COO, the CEO, and often the CFO (and maybe others) to sign off on it. Why the CFO? These things cost money directly, and if there are failures, those cost money too. Especially if you have SLAs with your customers.
So yes, we may know a better way (and we do run our hundreds of servers on Linux, thank you), it's not enough to know a better way. If you want to change to it, you have to make the business case, present it professionally, and get approval and support for it. If you go ahead without following these steps, in most shops you're onto a good way to find yourself unemployed.