Slashdot Mirror
What to Do When Your Security is Breached
ancientribe writes "When you've got a full-blown security breach on your hands, what do you do? If you've been smart, you'll already have a computer security incident response team — and a plan — in place. But many companies are too resource-strapped to have a full-blown, fully-tested incident response strategy. DarkReading has some tips on what to do — and what not to do."
When your security is breached by a handful of thugs you must immediately run out and attack a random neighbor's house.
many IT managers decide to purchase Microsoft so when something happens, well, "we couldn't go wrong with Microsoft" or "it's Microsoft, not us". Unfortunately, that's the extent of their plan, after pulling the network cable, i.e. cover their asses.
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
But since ours is a relatively small company, we went with the open-source Thai fighters.
Innovation makes enemies of all those who prospered under the old regime... -- Machiavelli
Kent Brockman: So, professor, would you say it's time for everyone to panic?
Professor: Yes I would, Kent.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
what to do if you burn your hand:
1. first, remove your hand from the burning stove.
2. use ice to cool your hand
3. seek medical attention.
wow. Thanks. I never would have figured any of that out on my own.
All businesses should have contingency plans for all disasters.
For most disasters, whether it's an IT disaster, a natural disaster, a non-natural physical disaster like a fire, a real or frivolous patent lawsuit, employee or company malfeasance, or what not, you need a plan.
For "terminal" disasters, like a nuclear blast that kills all employees and destroys all company assets, folding up shop may be the right business plan. For small businesses, extreme disasters like car wreck that kills all the employees might also be terminal in a slightly less catastrophic way. In these cases, at least you can plan to sell your business or its assets to another entity, so your customers have continuity.
Basically, divide your disasters into categories, and plan and insure accordingly:
0) end of the world, big asteroid or global thermonuclear war
1) major catastrophe, we are dead, forget about the customer, nuclear detonation event
2) end of the company, save the customer, Enron
3) end of the management team, save the company, MCI
4) we can recover from this but it's gonna hurt a lot, Vonage(?)
5) it's a flesh wound, CEO dies of heart attack
6) mosquito bite, SCO sues IBM
7) what? something happened? I didn't even notice, {if I had an example it would be #6}
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
The appropriate response is to shoot the lieutenant responsible for security. Then promote another ambitious, yet expendable underling to his/her place. Come on - this is Evil Overlord 101-level stuff.
'Loose' is when your pants are three sizes too big. 'Lose' is when you misuse 'loose'.
I've dealt with a couple security breaches in the past. It's never easy, and there's always that feeling of being violated as well. The important thing is to not lose your head about it, or you'll make mistakes that could lead to another or worse breach.
First, find out the extent of the breach. Analyze your log files. Find out what time it happened. Find out who was logged in at the time, and find out any log messages from any system services that can help you figure out what the problem was. If you can't figure out what the scope of the breach was with a high level of confidence, then you have to assume the worst: the entire network is compromised.
Second, salvage what you can. Again, be very careful about doing this. Hopefully you have a backup somewhere which would allow you to avoid or shorten this step as much as possible. In essence, do what you have to do to the compromised machine to avoid losing work, but always be concious of the fact that the machine is compromised, and may be transmitting or recording keylogs or other sensitive information. If possible, disconnect the compromised machines from the Internet and isolate it from the rest of your LAN.
Third, plan for the future. How would this breach be avoided in the future? Was it an OS problem? If so, then maybe you need to install OpenBSD instead. Was it a problem with a particular package you were using? Choose a different package. Can you configure your firewall or server to prevent or limit the abuse that caused the problem in the first place (e.g. fail2ban to deal with SSH phishing attacks) or install monitoring software to alert you of a problem (e.g. an IDS like Snort)? Do your users need further training? Does your password policy allow weak passwords? Etc.
Finally, take a deep breath. Unless you've been totally negligent in your job, there wasn't much you could do to prevent it. Don't worry about the fact that you don't have enough to go to the police; most Network Administrators don't have the hardware, training or certification to present evidence in a courtroom anyway. If you can go to the cops, then bully for you! Make that black-hat asshole pay!
"Please describe the scientific nature of the 'whammy'" - Agent Scully
Windows XP: What's security?
Windows Vista: This wouldn't happen to me anyway, I'm the Most Secure OS (tm)!
Mac OS X: I never get any viruses!
GNU/Linux: Me neither!
Windows Vista User Access Control: You are entering a conversation with flaming probability 89%. Cancel or Allow?
Windows Vista: [to Vista UAC] Allow. [to the others] That's because nobody uses you!
GNU/Linux: Oh yeah...
Mac OS X: That's because only elite people use Mac OS X. Because you're not worth them.
GNU/Linux: Wait! Windows Vista, you lie! Lot's of people from all around the world use me! In fact, they even improve me! That's because we believe that...
Mac OS X and Windows Vista: [at the same time] Shut up Linux.
Windows Vista: [to Mac OS X] But anyway, even if there were a "Security Breach", it's not like they'd be able to mess anything up!
Mac OS X: That's because it's impossible to do anything in Vista.
Windows Vista User Access Control: [to Vista] You are coming to a sad realization... Cancel or Allow?
NB: the views or opinions expressed by any of the characters do not necessarily resemble the views or opinions of the author.
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C1 bottles of beer on the wall. Take one down, pass it round... Oh, umm...
I got done reading this, and it's pretty dumb.
"If you're a big company, you already have a security team. If not, hire one." DOH!
That smacks me of the same kind of response from slashdot about legal advice... "Im being sued by the RIAA, should I ignore it?"
Still, why not gander around and see what the the real security experts and such say about such matters:
The Coroners Toolkit Tools for Unix
Nagios detection suite
Honeypots for 'sticking hackers'
And there's the wonderful tools in the Linux kernel for bridges and such that can be made to monitor data as if there was no computer there at all. Also, PF in FreeBSD can route and filter based on much more criteria than Linux netfilter can (like via OS).
You should have a secure layout of your network along with a respectable sensor network. The Sensornet should be separate from the general network.
If you already work in IT, these things should be obvious, as it is the similar measures required for data recovery on non-hack problems.
If you are 0wned, don't trust anything the box self-reports.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
OpenBSD: [walks into room, looks around, walks out, shaking his head not understanding why everyone can't be as secure as he is]
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
It was an open FTP server. Some kind soul put about 14Gb of movies on one of our servers, then we noticed the hole (mainly because of the space) and shut down access to that server.
So in our case the response was:
1. Stop access.
2. Buy beer and popcorn
3. Watch movies.
Once I was a four stone apology. Now I am two separate gorillas.
I've considered it, but there's a lot of barriers. First, you need enough evidence for a subpoena. That means that the chain of custody has to be preserved, and the crime scene needs to be secured by the police. Usually that means giving the compromised machines, relevant logs from monitoring equipment, etc. over to Law Enforcement for an indeterminate amount of time. I know I can't live without my servers for that long.
You need to get the subpoena to identify the person behind the attack. That assumes that your evidence actually points to a specific suspect. Unless your attacker was a complete moron, or your network logs are incredibly voluminous, that's not very likely. Once the subpoena is served and you've got your suspect and laid charges, you need to present evidence. That requires an expert witness. If you're lucky, YOU are the expert witness, but there's training and certification involved in that process. Otherwise, you get to hire an expert witness, and that won't be cheap. Your opponent will probably hire an opposing expert, just to confuse everybody.
Overall, I'd say that chances of success are incredibly low. Legal fees will be very high, and you have to turn over a fair chunk of your network assets to Law Enforcement. Basically, if you aren't really, really sure that you've got your man, it's really not worth the time and effort to find out who it was. That effort is much better spent allowing you to sleep at night knowing that people aren't getting in, IMO.
"Please describe the scientific nature of the 'whammy'" - Agent Scully
When in confusion
or in doubt
Run in circles
scream and shout.
And yeah, pull the ethernet cables out.
The most secure OS in the world wont protect you from a poorly-coded app. How many people are trying to crack your server at the OS level vs. the number of people looking for SQL injection vulnerabilities?
I don't know, their approach seems kinda... dangerous to me, but maybe that just shows that they're the big security gurus and I'm just a lowly coder. Maybe I can learn something from them. Or maybe they're talking out the ass, I dunno.
For starters the advice to wait until the whole team is assembled, including the accountants, lawyers, etc, then holding meetings to determine your strategy, etc, before even unplugging the damn thing... dunno, it seems to me bordering on criminal. Yes, you don't want to let one lone cowboy handle it from end to end, but a trained admin could at the very least be able to unplug the computer from the network and isolate the damage before it goes any worse. Or know enough to decide if it has to be unplugged. But if he thinks it is, it should be step #1 not IIRC step #4 after you're done holding your meetings and informing the employees and having PR draft the vaguely worded announcement that tries to make it sound unimportant to your customers.
Waiting for the designated accountant, and the designated lawyer, and the HR guy, and God knows who else to arrive at the middle of the night and hold their meeting while a breach is in progress and someone is downloading your productive database, seems to me dumb to the extreme. To reuse your example, it's like saying you should keep your hand in the stove until you talked to your lawyer and your doctor and a designated family member, make sure you have a strategy, and only then pull the hand out. By that time, it could be burned to a crisp.
I mean, by the elder Gods, especially when you include such non-techies... surely you've seen these guys when they have to give you a spec for a program. If you wait for them to hold a meeting on such technical issues as "are we in aggreement that we need to unplug the server?", at least one goes into responsibility avoidance mode and refuses to be remembered as the one who took any decision, at least one goes into alpha-dog-pissing-on-everything-to-mark-his-territ ory mode, etc. It's a meeting that could well take hours without going anywhere.
Frankly. I'd rather just trust the "cowboy" admin to know his job well enough, and know whether he needs to unplug the servers because of a serious breach, or just let it be if it's just a DDOS, while the non-techies deal with their own domain of competence. There is _nothing_ a non-techie can add that's meaningful to that kind of an inherently techie decision. Just like you don't have the admins tell the company lawyers what to do, have the decency to not have the admin hang around and wait for the lawyers to tell him what to do. It's not only a better use of the admins' time, it's also a better use of the lawyers' time, who could be doing something that's a better use of _their_ skills in that time.
I'll aggree, though, that the advice at step 1 seems to be dangerously content free. It's something which, although it may sound otherwise, actually noone ever actually did as such. Even if one "cowboy" admin did offer to contain the incident, it's not like someone let him deal with the _whole_ affair, including the HR, legal and financial aspects. Which is the domains they mention that you need on that team. More likely the "cowboy" just dealt with the servers, while the lawyer did his own job, the HR guy did his own, etc. I don't think (m)any people let the admin draft the press release too, for example. So the whole "don't let one 'cowboy' deal with it all" advice is basically like saying "don't try to fly on a broomstick off a bridge": you weren't actually planning to do that anyway, and it's not really giving you any insight you didn't already have.
Finally, I don't know, maybe I'm just paranoid by trade, but the whole thing looks more like PR and a bit of an IT-for-PHBs magazine than anything actually serious about security or IT. It reads like little more than an advertisment for the three companies they mention, with a bit of a scare theme to make you contact them ASAP, than anything else. I'm also a tad cir
A polar bear is a cartesian bear after a coordinate transform.
1. Keep the suits and incompetent people the hell out!
Once a compromise happened, there is no time to listen to lawyers or marketing executives. If they have anything to say, they would write a document where they list all recommendations they can care about -- for example, how "This site is pwn3d" web page is supposed to look like, whether it is a good thing to send all users a letter "please cancel your credit card", or what information can be released to authorities. If they didn't do that already, let them write those things while sysadmins are working.
This, of course, means that if there is only one sysadmin competent enough to investigate and fix the problem, then he would have to work on it alone.
2. Shut it down and investigate changes made by the attackers.
Before doing any investigation or recovery, shut the compromised and potentially compromised devices down. No malicious code should remain running. Whatever services should remain, must run in the minimal mode on separate hardware. For example, keep email running on a newly installed box. All investigation should be done in a clean environment -- drives moved to dedicated "clean" machines, or original servers booted from clean images (CD, PXE, replacement drives) on a private subnet, not accessible to anyone but people involved in incident response. Make full images of compromised hosts' storage whenever possible.
Backups are your friend. IDS logs are, too, but make sure that your IDS isn't compromised, and actually recorded something meaningful.
3. Don't worry about the person who originated the the attack.
Find its results and, if possible, method. Likely there will be at least one person within the company (malicious or more likely negligent) and at least one outside. Screw them both, they don't have access to your network anymore because it's off.
4. Immediately restore known-clean backups, perform audit on potentially compromised data and update the systems.
Backup is "known-clean" if investigation shown that it is from a state before the attack and does not contain vulnerable versions of software or compromised authentication information that allowed attack to happen. Usually some data has to be restored from a compromised system because it's more recent than backup (or because you are an idiot and forgot to back it up). Audits are supposed to be painful. Once data is in place, update software and configuration. Erase all compromised authentication keys and tokens.
5. Document the process.
I mean, technical details.
6. Tell everyone that they are screwed.
Explain to every office drone that they are going to get new passwords. They won't like it, so keep your LART ready.
Oh, btw:
http://abelits.livejournal.com/30214.html
http://abelits.livejournal.com/30681.html
http://abelits.livejournal.com/30872.html
Contrary to the popular belief, there indeed is no God.
Enter the FBI, who showed up in my roomate's lab asking about his computer (amoung other things). Picture yourself a grad student answering his lab door to find men in suits (an uncommon experience) who say they're part of the FBI (also uncommon), and mean it (still less common). After some questions, it was hesitantly established that my roomate was not the hacker serving root kits from his home computer.
From there, the FBI (with our permission) bugged our appartment. They put a "tap" in our appartment, which consistend of a special switch and a *very* loud windows machine that sat on our internet connection listening for hacker activity. The installation of the tap involved 7 FBI agents, none of which new nearly as much as my roomate about networking (that the broadcast ping couldn't get through their special switch with the word "tap" on it was a real mystery). Neadless to say, I didn't fool around with bittorrent or the like durring that time.
After a month or two, they caught the hacker (who was sweedish, apparently), and eventaully prosecuted him successfully.
Point is: sometimes it is useful to not reinstall immediately when hacked -- it can result in a good story :)
Disclaimer: I've actually written the security policy in place at my company, and I used to be the guy responsible for security before my last career move.
/sbin/kill could've been replaced by a trojan and not do what you expect it to do. Even /sbin/init is suspect. Your kernel, boot record, on some machines even your BIOS has possibly been manipulated.
My advise to sysadmins who notice a breach is this:
Take your hands off the damn keyboard. Don't do anything unless you are 150% certain that you can see all possible consequences of your action. Call the IRT if you have one.
If there's nobody to call and you have to act right now, pull the power plug on the machine, then call the experts. Don't power the machine up again under any circumstances. If you want to look at the harddrive, make a copy first and mount it read-only in a different machine.
Why? Because back in the days when I was, err... looking around inside machines not my own, one of the things everyone I knew did was put in some scripts, tools, something, that'd wipe the logs or even the machine if my shell gets killed or the machine shut down or rebooted.
TFA assumes that you learn of the incident long after it has happened. Many incidents in real life are being noticed while they are going on, no matter if it's a remote access or your machine running an FTP server that wasn't there last month. That FTP server is almost certainly patched, and one of the things it might do is destroy evidence if you kill it. There might be an invisible process watching it to wipe evidence if you kill -9 it. Heck,
Assorted stuff I do sometimes: Lemuria.org