Slashdot Mirror
What to Do When Your Security is Breached
ancientribe writes "When you've got a full-blown security breach on your hands, what do you do? If you've been smart, you'll already have a computer security incident response team — and a plan — in place. But many companies are too resource-strapped to have a full-blown, fully-tested incident response strategy. DarkReading has some tips on what to do — and what not to do."
Next time, run OpenBSD. If you don't, expect to be pwn3d.
When your security is breached by a handful of thugs you must immediately run out and attack a random neighbor's house.
many IT managers decide to purchase Microsoft so when something happens, well, "we couldn't go wrong with Microsoft" or "it's Microsoft, not us". Unfortunately, that's the extent of their plan, after pulling the network cable, i.e. cover their asses.
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
Complain! Call the help desk!
Grab your ankles, and kiss your ass goodbye!
That'll learn you not to use Microsoft again.
But since ours is a relatively small company, we went with the open-source Thai fighters.
Innovation makes enemies of all those who prospered under the old regime... -- Machiavelli
Run from side to side?
Kent Brockman: So, professor, would you say it's time for everyone to panic?
Professor: Yes I would, Kent.
I came to the datacenter drunk with a fake ID, don't you want to be just like me?
what to do if you burn your hand:
1. first, remove your hand from the burning stove.
2. use ice to cool your hand
3. seek medical attention.
wow. Thanks. I never would have figured any of that out on my own.
Silly trout -- security is for blowfish!
First thing to do is to pull the plug, and stop any further damage. After you're not connected to the Net, THEN you can figure out what happened and how to fix it
I don't respond to AC's.
I'd note that even if your company has a response plan, you may find it either completely useless or so general that it doesn't provide any help. Look at the article's point #1: it's almost nothing but "If $X, you may need $Y.". And it's far from complete. That's going to be a flaw in any security response plan: it's likely to not address the actual problem you face. Problems that you've thought of tend to get caught earlier before they turn into full-blown incidents, it's the ones nobody thought of that are most likely to bite you badly and it's exactly those that a plan won't cover. About the only part of the plan that'll be guaranteed to be useful is the part explaining what parts of the system are responsible for what and how to lock them down to preserve the evidence while you figure out where the breach is and what you need to do next. Beyond that you're into a twisty maze of little possibilities, all almost but not quite completely unlike each other, and what you need most isn't a plan but someone with enough Clue to analyze the situation and formulate a plan to fit it on the fly.
Switch to a paper only office, and an air-tube network.
Libertarian Leaning Political Discussion Forum.
All businesses should have contingency plans for all disasters.
For most disasters, whether it's an IT disaster, a natural disaster, a non-natural physical disaster like a fire, a real or frivolous patent lawsuit, employee or company malfeasance, or what not, you need a plan.
For "terminal" disasters, like a nuclear blast that kills all employees and destroys all company assets, folding up shop may be the right business plan. For small businesses, extreme disasters like car wreck that kills all the employees might also be terminal in a slightly less catastrophic way. In these cases, at least you can plan to sell your business or its assets to another entity, so your customers have continuity.
Basically, divide your disasters into categories, and plan and insure accordingly:
0) end of the world, big asteroid or global thermonuclear war
1) major catastrophe, we are dead, forget about the customer, nuclear detonation event
2) end of the company, save the customer, Enron
3) end of the management team, save the company, MCI
4) we can recover from this but it's gonna hurt a lot, Vonage(?)
5) it's a flesh wound, CEO dies of heart attack
6) mosquito bite, SCO sues IBM
7) what? something happened? I didn't even notice, {if I had an example it would be #6}
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
If you're working for a company too small for a "Security response team", and chances are, you are, then you've got to consider outsourcing. If a security breach happened, then obviously you don't have the expertise in house to handle security in house, and you're just putting out fires after they happen. It's time to start looking to outsource whatever it was that was broken. In this day and age, unless you're doing something very, very custom, there's really little value to having in house web serving, email, etc.
I don't respond to AC's.
You'll use this link. "Print buttons" are your friend, unless you really like 2 pages of content being spread over 10 pages.
The appropriate response is to shoot the lieutenant responsible for security. Then promote another ambitious, yet expendable underling to his/her place. Come on - this is Evil Overlord 101-level stuff.
'Loose' is when your pants are three sizes too big. 'Lose' is when you misuse 'loose'.
It's been a long time (thankfully) since I've had to deal with this. But I'd echo the article about disconnecting from the net to eliminate further attacks. Then I'd remove the drive and save it for forensics -- replacements are cheap (I'm assuming a small business doesn't have expensive RAID setups). Assume that everything has been compromised and restore from a backup prior to the intrusion (hopefully you can tell when that was).
Oh, and keep your clocks synchronized. This will help if you need to trace intrusions across systems.
Wanted: witty unique signature. Must be willing to relocate.
Right on.
http://www.maxineudall.com/2010/02/should-economists-be-sued-for-malpractice.html
I'm not sure if you meant the RJ45 or the AC plug.
In some cases, you may NOT want to pull the plug.
Sometimes proper forensic evaluation requires both plugs remain attached until the experts are done.
As the article said though, sometimes you have to balance continuing harm with the need to preserve the crime scene.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I've dealt with a couple security breaches in the past. It's never easy, and there's always that feeling of being violated as well. The important thing is to not lose your head about it, or you'll make mistakes that could lead to another or worse breach.
First, find out the extent of the breach. Analyze your log files. Find out what time it happened. Find out who was logged in at the time, and find out any log messages from any system services that can help you figure out what the problem was. If you can't figure out what the scope of the breach was with a high level of confidence, then you have to assume the worst: the entire network is compromised.
Second, salvage what you can. Again, be very careful about doing this. Hopefully you have a backup somewhere which would allow you to avoid or shorten this step as much as possible. In essence, do what you have to do to the compromised machine to avoid losing work, but always be concious of the fact that the machine is compromised, and may be transmitting or recording keylogs or other sensitive information. If possible, disconnect the compromised machines from the Internet and isolate it from the rest of your LAN.
Third, plan for the future. How would this breach be avoided in the future? Was it an OS problem? If so, then maybe you need to install OpenBSD instead. Was it a problem with a particular package you were using? Choose a different package. Can you configure your firewall or server to prevent or limit the abuse that caused the problem in the first place (e.g. fail2ban to deal with SSH phishing attacks) or install monitoring software to alert you of a problem (e.g. an IDS like Snort)? Do your users need further training? Does your password policy allow weak passwords? Etc.
Finally, take a deep breath. Unless you've been totally negligent in your job, there wasn't much you could do to prevent it. Don't worry about the fact that you don't have enough to go to the police; most Network Administrators don't have the hardware, training or certification to present evidence in a courtroom anyway. If you can go to the cops, then bully for you! Make that black-hat asshole pay!
"Please describe the scientific nature of the 'whammy'" - Agent Scully
Windows XP: What's security?
Windows Vista: This wouldn't happen to me anyway, I'm the Most Secure OS (tm)!
Mac OS X: I never get any viruses!
GNU/Linux: Me neither!
Windows Vista User Access Control: You are entering a conversation with flaming probability 89%. Cancel or Allow?
Windows Vista: [to Vista UAC] Allow. [to the others] That's because nobody uses you!
GNU/Linux: Oh yeah...
Mac OS X: That's because only elite people use Mac OS X. Because you're not worth them.
GNU/Linux: Wait! Windows Vista, you lie! Lot's of people from all around the world use me! In fact, they even improve me! That's because we believe that...
Mac OS X and Windows Vista: [at the same time] Shut up Linux.
Windows Vista: [to Mac OS X] But anyway, even if there were a "Security Breach", it's not like they'd be able to mess anything up!
Mac OS X: That's because it's impossible to do anything in Vista.
Windows Vista User Access Control: [to Vista] You are coming to a sad realization... Cancel or Allow?
NB: the views or opinions expressed by any of the characters do not necessarily resemble the views or opinions of the author.
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C1 bottles of beer on the wall. Take one down, pass it round... Oh, umm...
Maybe.
Let's assume the bad guys never stored any forensically useful stuff on disk in clear text. Peter Gutmann has a few things to say about recovering useful information from RAM chips.
The question for the real world is:
Is it worth going this far just to catch the bad guys?
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I got done reading this, and it's pretty dumb.
"If you're a big company, you already have a security team. If not, hire one." DOH!
That smacks me of the same kind of response from slashdot about legal advice... "Im being sued by the RIAA, should I ignore it?"
Still, why not gander around and see what the the real security experts and such say about such matters:
The Coroners Toolkit Tools for Unix
Nagios detection suite
Honeypots for 'sticking hackers'
And there's the wonderful tools in the Linux kernel for bridges and such that can be made to monitor data as if there was no computer there at all. Also, PF in FreeBSD can route and filter based on much more criteria than Linux netfilter can (like via OS).
You should have a secure layout of your network along with a respectable sensor network. The Sensornet should be separate from the general network.
If you already work in IT, these things should be obvious, as it is the similar measures required for data recovery on non-hack problems.
If you are 0wned, don't trust anything the box self-reports.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
"external consultants or forensics experts -- should be selected prior to an event, experts say."
What a shocker...
According to my first aid training, never ever use ice. It can cause further tissue damage. For small burns, run large amounts of cool but not cold water over the burn. Larger burns, soak a CLEAN towel in cool water.
From personal experience (unfortunately the personal experience came before the Red Cross training), running cold water over a burn causes excruciating pain about 30 seconds after the cold source is removed. My theory is that the cold constricts blood flow, and after you remove the cold source, the blood starts coming back through the damaged tissue area and oh my god does it hurt.
OpenBSD: [walks into room, looks around, walks out, shaking his head not understanding why everyone can't be as secure as he is]
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
I call Microsoft support.
the article is only 2, not 10 pages long to begin with.
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
It boggles me that so many people come up with so many "solutions" yet hardly anyone comes up with the really important step to take: you backup your data, wipe the HD clean and re-install your OS. No matter what you use; be it Linux, Solaris, BSD.
Pretend you are a Wal-Mart or IBM. Suppose Bentonville or Armonk gets wiped off the map by a terrorist bomb.
I hope both companies have some kind of continuity plan, even if it's just transferring their assets and customer lists to a competitor.
On the other hand, the Bentonville Bed And Breakfast will probably just fold up shop.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
5 - 6 page ones suck so we try to fix things with out tell PHB who will just make lock down thing that will get in the way of people doing there jobs.
I am curious to how many people actually go the next step to get the bad guy caught and how successful they are with it? It seems like its a tough battle to get the identity of the person behind an IP.
Ice will just make it worse.
Only thing worse than a hollow article is a wrong one.
The US free market: two halves of a government-granted duopoly are free to set the market price.
Just patch a socket. Problem solved. I learned that watching 24.
It was an open FTP server. Some kind soul put about 14Gb of movies on one of our servers, then we noticed the hole (mainly because of the space) and shut down access to that server.
So in our case the response was:
1. Stop access.
2. Buy beer and popcorn
3. Watch movies.
Once I was a four stone apology. Now I am two separate gorillas.
Just close your eyes, count to ten, then start shouting "Serenity now" over and over again until the problem passes you by. :)
Cry havoc And Let Slip The Dogs Of War.
I've considered it, but there's a lot of barriers. First, you need enough evidence for a subpoena. That means that the chain of custody has to be preserved, and the crime scene needs to be secured by the police. Usually that means giving the compromised machines, relevant logs from monitoring equipment, etc. over to Law Enforcement for an indeterminate amount of time. I know I can't live without my servers for that long.
You need to get the subpoena to identify the person behind the attack. That assumes that your evidence actually points to a specific suspect. Unless your attacker was a complete moron, or your network logs are incredibly voluminous, that's not very likely. Once the subpoena is served and you've got your suspect and laid charges, you need to present evidence. That requires an expert witness. If you're lucky, YOU are the expert witness, but there's training and certification involved in that process. Otherwise, you get to hire an expert witness, and that won't be cheap. Your opponent will probably hire an opposing expert, just to confuse everybody.
Overall, I'd say that chances of success are incredibly low. Legal fees will be very high, and you have to turn over a fair chunk of your network assets to Law Enforcement. Basically, if you aren't really, really sure that you've got your man, it's really not worth the time and effort to find out who it was. That effort is much better spent allowing you to sleep at night knowing that people aren't getting in, IMO.
"Please describe the scientific nature of the 'whammy'" - Agent Scully
and kiss your ass goodbye!
Lift off and nuke the site from orbit.
It's the only way to be sure.
panic!
Just post your IP addresses and remote access logons and I'll be glad to help with your break-in! I promise I'll take the data and put it somewhere safe -and offshore No payment up front, but trust me -I will be getting back to you. -I'm just sayin'
When in confusion
or in doubt
Run in circles
scream and shout.
And yeah, pull the ethernet cables out.
One-third content, two-thirds ads and links. Yeah, that's a good design.
1 part content, nothing extra
The most secure OS in the world wont protect you from a poorly-coded app. How many people are trying to crack your server at the OS level vs. the number of people looking for SQL injection vulnerabilities?
I don't know, their approach seems kinda... dangerous to me, but maybe that just shows that they're the big security gurus and I'm just a lowly coder. Maybe I can learn something from them. Or maybe they're talking out the ass, I dunno.
For starters the advice to wait until the whole team is assembled, including the accountants, lawyers, etc, then holding meetings to determine your strategy, etc, before even unplugging the damn thing... dunno, it seems to me bordering on criminal. Yes, you don't want to let one lone cowboy handle it from end to end, but a trained admin could at the very least be able to unplug the computer from the network and isolate the damage before it goes any worse. Or know enough to decide if it has to be unplugged. But if he thinks it is, it should be step #1 not IIRC step #4 after you're done holding your meetings and informing the employees and having PR draft the vaguely worded announcement that tries to make it sound unimportant to your customers.
Waiting for the designated accountant, and the designated lawyer, and the HR guy, and God knows who else to arrive at the middle of the night and hold their meeting while a breach is in progress and someone is downloading your productive database, seems to me dumb to the extreme. To reuse your example, it's like saying you should keep your hand in the stove until you talked to your lawyer and your doctor and a designated family member, make sure you have a strategy, and only then pull the hand out. By that time, it could be burned to a crisp.
I mean, by the elder Gods, especially when you include such non-techies... surely you've seen these guys when they have to give you a spec for a program. If you wait for them to hold a meeting on such technical issues as "are we in aggreement that we need to unplug the server?", at least one goes into responsibility avoidance mode and refuses to be remembered as the one who took any decision, at least one goes into alpha-dog-pissing-on-everything-to-mark-his-territ ory mode, etc. It's a meeting that could well take hours without going anywhere.
Frankly. I'd rather just trust the "cowboy" admin to know his job well enough, and know whether he needs to unplug the servers because of a serious breach, or just let it be if it's just a DDOS, while the non-techies deal with their own domain of competence. There is _nothing_ a non-techie can add that's meaningful to that kind of an inherently techie decision. Just like you don't have the admins tell the company lawyers what to do, have the decency to not have the admin hang around and wait for the lawyers to tell him what to do. It's not only a better use of the admins' time, it's also a better use of the lawyers' time, who could be doing something that's a better use of _their_ skills in that time.
I'll aggree, though, that the advice at step 1 seems to be dangerously content free. It's something which, although it may sound otherwise, actually noone ever actually did as such. Even if one "cowboy" admin did offer to contain the incident, it's not like someone let him deal with the _whole_ affair, including the HR, legal and financial aspects. Which is the domains they mention that you need on that team. More likely the "cowboy" just dealt with the servers, while the lawyer did his own job, the HR guy did his own, etc. I don't think (m)any people let the admin draft the press release too, for example. So the whole "don't let one 'cowboy' deal with it all" advice is basically like saying "don't try to fly on a broomstick off a bridge": you weren't actually planning to do that anyway, and it's not really giving you any insight you didn't already have.
Finally, I don't know, maybe I'm just paranoid by trade, but the whole thing looks more like PR and a bit of an IT-for-PHBs magazine than anything actually serious about security or IT. It reads like little more than an advertisment for the three companies they mention, with a bit of a scare theme to make you contact them ASAP, than anything else. I'm also a tad cir
A polar bear is a cartesian bear after a coordinate transform.
Sigh, Get the install CD's locate your last few data backups and plan on a very VERY long week + weekend. Restoring the data is fine, restoring the OS+apps is NOT but you have a good snapshot of the apps when you last installed them right?
When asked, make sure you mention how the CTO cancelled your project for security upgrades and audits and this would not have happened if security audits and upgrades were performed.
Key CTO's car just to make sure.
27B Stroke 6
I'm a stickler for paper work.
-Buddy of DoQ
MyPW just released a PAM module that allows you to use their password token on as many different Linux servers you install it on. It's pretty cool, I just installed it last week on 5 servers.
Curl up in the fetal position and wait for tomorrow.
Also works at performance reviews.
Burn the place to the ground, kill everyone, start again.
Now wash your hands.
1) If you have an IT team ready to go: Simply pick one, blame it on them, fire them, find some random script-kiddie hacker and blame them.
2) If you don't have an IT team ready to go: Blame users / customers since it clearly must be their fault in some way, shape, or form.
Remember kiddies, rules never apply if you're a corporation.
If you were offended by anything I said... No, I'm not sorry. Please lighten up.
Here's a great idea: impliment a fleet of Lunix servers maintainted by a fleet of consultants. That way, when (not IF) you get hacked, nobody will really know it, and nobody will actually be accountable! It worked for Munich, it can work for you! Go go infinite billable hours!
Lunix: the "cover your ass" alternative for spending other people's money!
Got r00t?
The sysadmins are smart. It's their managers who make those decisions. 99% of managers simply don't have the brains to see the need for a disaster recovery team.
The solution is simple. Plan, Plan, Plan, Test, Plan, Test, Plan, and Test.
Your Data has to live separately from your OS/Apps
The easiest way is to use a VMWare ESX server and keep daily/weekly backups of the VMs.
Your Data should live on a Large redundant Storage Array.
The ideal situation for every IT group is a scalable redundant SAN and a scalable Blade center running VMWARE bare metal.
The costs of such things don't seem worth it, but They really are.
They're using their grammar skills there.
Careful with the subpoena. There was a reporter sitting in the court trolling for news when we asked for a subpoena. The resulting news story was worse than the incident ever was. CIO is gone now.
the above comment would be correct, where it not for the fact that *BSD IS DYING!!!
1. Keep the suits and incompetent people the hell out!
Once a compromise happened, there is no time to listen to lawyers or marketing executives. If they have anything to say, they would write a document where they list all recommendations they can care about -- for example, how "This site is pwn3d" web page is supposed to look like, whether it is a good thing to send all users a letter "please cancel your credit card", or what information can be released to authorities. If they didn't do that already, let them write those things while sysadmins are working.
This, of course, means that if there is only one sysadmin competent enough to investigate and fix the problem, then he would have to work on it alone.
2. Shut it down and investigate changes made by the attackers.
Before doing any investigation or recovery, shut the compromised and potentially compromised devices down. No malicious code should remain running. Whatever services should remain, must run in the minimal mode on separate hardware. For example, keep email running on a newly installed box. All investigation should be done in a clean environment -- drives moved to dedicated "clean" machines, or original servers booted from clean images (CD, PXE, replacement drives) on a private subnet, not accessible to anyone but people involved in incident response. Make full images of compromised hosts' storage whenever possible.
Backups are your friend. IDS logs are, too, but make sure that your IDS isn't compromised, and actually recorded something meaningful.
3. Don't worry about the person who originated the the attack.
Find its results and, if possible, method. Likely there will be at least one person within the company (malicious or more likely negligent) and at least one outside. Screw them both, they don't have access to your network anymore because it's off.
4. Immediately restore known-clean backups, perform audit on potentially compromised data and update the systems.
Backup is "known-clean" if investigation shown that it is from a state before the attack and does not contain vulnerable versions of software or compromised authentication information that allowed attack to happen. Usually some data has to be restored from a compromised system because it's more recent than backup (or because you are an idiot and forgot to back it up). Audits are supposed to be painful. Once data is in place, update software and configuration. Erase all compromised authentication keys and tokens.
5. Document the process.
I mean, technical details.
6. Tell everyone that they are screwed.
Explain to every office drone that they are going to get new passwords. They won't like it, so keep your LART ready.
Oh, btw:
http://abelits.livejournal.com/30214.html
http://abelits.livejournal.com/30681.html
http://abelits.livejournal.com/30872.html
Contrary to the popular belief, there indeed is no God.
Enter the FBI, who showed up in my roomate's lab asking about his computer (amoung other things). Picture yourself a grad student answering his lab door to find men in suits (an uncommon experience) who say they're part of the FBI (also uncommon), and mean it (still less common). After some questions, it was hesitantly established that my roomate was not the hacker serving root kits from his home computer.
From there, the FBI (with our permission) bugged our appartment. They put a "tap" in our appartment, which consistend of a special switch and a *very* loud windows machine that sat on our internet connection listening for hacker activity. The installation of the tap involved 7 FBI agents, none of which new nearly as much as my roomate about networking (that the broadcast ping couldn't get through their special switch with the word "tap" on it was a real mystery). Neadless to say, I didn't fool around with bittorrent or the like durring that time.
After a month or two, they caught the hacker (who was sweedish, apparently), and eventaully prosecuted him successfully.
Point is: sometimes it is useful to not reinstall immediately when hacked -- it can result in a good story :)
My grandmother would just offer to patch my socks. I tried to convince her once I was wearing V5 SOCKS, but she called balderdash and kicked me from the channel!
...nuke it from orbit. It's the only way to be sure.
"What would MacGyver do?"
NetWare is proprietary (perhaps built on some FLOSS, but with proprietary software mixed in) so you can't completely know how well the implementation you're running follows a design you favor. No proprietary software's security can be verified as completely as FLOSS. It's never wise to conclude that any proprietary software is "secure by default".
Digital Citizen
1. I was assuming a serious breach. If it's a development server, frankly, it has no excuse to have any real data on it, or be accessible from the internet at all. So it would hardly qualify as a security breach, or be possible to breach.
2. Depending on the zone where a server is, leaving it happily running without isolating it, can be an invitation for the problem to magnify. E.g., for most companies there is more than one server in a DMZ: if one is pwned, it can be used as a proxy to attack the others. E.g., that server has access to a database. Don't be surprised if the same database server or cluster hosts more than one database. Just because the compromised server may host, say, some public informational pages about the company's products, don't assume it can't possibly also have access to the online shop database.
3. I still say that a qualified admin should be able to tell that kind of a difference. Unless the company's IT is a a chaotic (evil) disaster, the admin should already know which is the development server and which is the production server. I don't think he has to wait for the CEO to tell him which server is which, or that most CEOs will even know. That's not saying that the CEO is (necessarily) dumb, just that it's not his job. It's like waiting for the general to come tell you who's the designated marksman in each squad: it's just not his job to worry about details at that level.
4. Sad to say, getting a prosecution and a conviction is nowadays 99% pipe dream and fairy tale. Between containing some damage which you know _will_ expand if given enough time, and the off chance that you'll bring a hacker to justice, I dunno, I know what I'd choose. I'd love to see justice served, but at some point you have to be realistic and realize that there's no point in being the martyr (even as a whole company) for a lost cause.
A polar bear is a cartesian bear after a coordinate transform.
they always ask if more than $500,000 was lost, if not then no feds
--unless you are selling booze, drugs, or cigars online
or
serving kiddie porn (maybe in an encrypted file one of your "users" uploaded?)
I always keep an up to date copy of my resume.
Just kidding... mostly
This is great timing. Just this morning I had a client site visit to install our (financial) software. I couldn't get it to run and the reason turned out to be every single port (all 65k of them) was in use by a variant of Sasser. It took 2 hours to convince the client that it was even possible that they could get infected with a worm (this was a Windows server, direct atatched to the internet with no patching *at all* and AV software which had expired last year).
In the end the client still doesn't really believe me and thinks it's just "our buggy software" which must be the problem.
So what do you do when your security is breached, but no one (who can make a decision) cares or believes?
Create an account on this website ?
Don't you know it is now both immoral and criminal to think beyond the next quarterly report?
I'm not sure if you meant the RJ45 or the AC plug. In some cases, you may NOT want to pull the plug.
I've always understood that it's best to *physically* yank power, *then* network.
Rationale? Power down may corrupt the filesystem, but still preserves state (ie. log files, etc), while yanking network or performing a #shutdown -h now may trigger coverup or malicious scripts added by the intruder.
Next step is, of course, dd the disks to a DVD or something (evidence, analysis), format, then start to recover executables from clean install discs, config and data from backups.
Oh, and at work, I tail the security log files to an old dot matrix printer; it's pretty hard for a remote intruder to upset that. (Could also tail by one-way serial cable to a separate machine with no network connectivity.)
Call me paranoid, but discussing my logging and other security stuff freaks me out. That's why I've checked "Post Anonymously".
Disclaimer: I've actually written the security policy in place at my company, and I used to be the guy responsible for security before my last career move.
/sbin/kill could've been replaced by a trojan and not do what you expect it to do. Even /sbin/init is suspect. Your kernel, boot record, on some machines even your BIOS has possibly been manipulated.
My advise to sysadmins who notice a breach is this:
Take your hands off the damn keyboard. Don't do anything unless you are 150% certain that you can see all possible consequences of your action. Call the IRT if you have one.
If there's nobody to call and you have to act right now, pull the power plug on the machine, then call the experts. Don't power the machine up again under any circumstances. If you want to look at the harddrive, make a copy first and mount it read-only in a different machine.
Why? Because back in the days when I was, err... looking around inside machines not my own, one of the things everyone I knew did was put in some scripts, tools, something, that'd wipe the logs or even the machine if my shell gets killed or the machine shut down or rebooted.
TFA assumes that you learn of the incident long after it has happened. Many incidents in real life are being noticed while they are going on, no matter if it's a remote access or your machine running an FTP server that wasn't there last month. That FTP server is almost certainly patched, and one of the things it might do is destroy evidence if you kill it. There might be an invisible process watching it to wipe evidence if you kill -9 it. Heck,
Assorted stuff I do sometimes: Lemuria.org
Actually yes. (And you will excuse me for being anon for this). A while back I was working for a streaming media company. We had about 40 Win2k boxes (shut up; you couldn't stream WMA from *nix; and not a single one was hacked, god bless IISLockdown). As we were a small company I split my time between looking after the devs and looking after the network team. Unknown to me we had a couple of linux boxes which were no longer used; but were still active. The network person who was supposed to look after them hadn't documented them or updated them and they got owned. Badly.
The attackers started using them to share files (good call, as a streaming network we had fat bandwidth). When I discovered it (by looking at the weekly bandwidth stats and going "What the hell is that IP") they had started distributing child porn from them. So I rang the UK Computer Crimes Unit. They told me to shut down the boxes. So I did, and pulled them from the data centre; bringing them back to the office and waited for the police to turn up. And waited. And waited. These were big expensive boxes for a small company to have offline; £6k a pop. And the police never followed up. When the company ran out of cash 3 years later those boxes were still sitting in a corner waiting to be picked up by the police and examined.
When in doubt, or have a security breach, assault an other country!
Let's assess your response step by step.
1. Assemble an incident response team.
Gather the buddies round the terminal, see what we got here.
2. Assess the initial damage and the risk for more.
You measured the damage, all 14GB of it. In assessing the risk for more of this damage, you noted that no ftp write access had been tried in a while, concluding that the risk was relatively low.
3. Develop a notification plan.
You sent an email-to-all that there's going to be a movie night, cancel your dates, postpone dinner, it's going to be a long one!
4. Begin remediating the problem.
You closed off ftp access.
5. Document everything.
I guess watching the movies, I mean damage, would fall under the documentation stage.
6. Develop a strategy for stopping the next attack.
Contemplate re-opening the ftp server to encourage more damage.
Comment removed based on user account deletion
At the risk of tooting my own horn, I blogged about similar material about a week before the Dark Reading publication. My blog focused more on the PR foul-ups that companies tend to make and ways to prevent those foul-ups rather than the technical response. It was based off of a recent Google vulnerability that got publicly posted as ?revenge? by the vulnerability discover who was unhappy about having not gotten enough credit for previously reported Google vulnerabilities. Neil Smithline BEA WebLogic Security Architect
Neil Smithline http://www.neilsmithline.com
If you're not ultra concerned about the server, you can have some fun. One technique is replace the executables with similarly sized programs which do something slightly malicious, such as alter their internet settings to take them offline until they figure out how to fix it. Or corrupt a couple bytes in the file with a hex editor. Usually it seems like people just run irc bots on compromised systems. What they don't realize is that this gives you a method of determining channel passwords, bot passwords (which are probably their passwords on other compromised machines), and if you feel like it, you can use this information to take over their irc channel.
No, first you need to figure out a way to make the RIAA/MPAA think your attacker is hosting vast amounts of pirated music and movies. Let *them* get the subpoenas and harass your attacker with frivolous lawsuits...
Take, for example, any modern large financial services business: you get 20 GB of encrypted file transfers a day, which your sources delete immediately after sending. You process the data and then your 5000+ telephone'n'cubicle staff work the data from your 50 TB SAN-based data sink. Every day you retire 18 GB or so into archival media and do your backups from a snapshot (since your db runs 365/24 except for scheduled patching and the 5-minute sthudowns for snapping).
You figure out you are compromised due to IDS or firewall alarming. Initial analysis shows you've been infected for an indeterminate amount of time, because with modern staffing practices nobody can watch the IDS all the time. You don't know which of your backups are infected, and restoring completely from backup is easy for the desktops (you ghost multicast all 10,000 of them simultaneously, because you have a reasonably intelligently engineered infrastructure that allows that) but extremely difficult for the servers and data - because you have 20 GB incoming daily, and the further back you go the more you have to re-integrate intervening days' files, and the more of the constantly changes implemented by staff you will lose.
(Look, ma! I spelled "lose" correctly!)
Better option is stealthily tracing the intrusion to its source, breaking down the door of the criminal, and shoving lit bamboo slivers under his toenails until he agrees to help you remove his rootkits etc. Once the systems have been sanitized, kill him quickly and humanely; there's no need to be a dick about it.
The parent demonstrates a huge flaw in Anonymous Coward's logic the post before it:
Sometime what seems like "common sense" is just plain wrong and even damaging.
AC said in step 2 "use ice to cool your hand" and then added with sarcasm that they never would have figured that out on their own. However, if the burn was severe, that step could lead to the loss of skin (or limb!) Cooling a burn may seem logical, but its not the whole picture. In common cases, you probably do not need to worry about thermal shock, but it sometimes makes every difference. This is why we have professionals who know these things; that's why step 3 "seek medical attention" actually comes second.
Two examples. If you work outside in very cold whether, your exposed skin may freeze (i.e. frost bite). You will probably not notice the dying of the skin due the numbing the cold has on the nerves. However, go back into a warm area and, as the nerves return to normal, the pain will hit. The solution to save as much of the exposed skin/limb as possible is to return heat to it slowly-- place it in COLD water and let the water gradually come to room temperature. If you place those frozen fingers of yours until a hot tap, you will probably find 2 things: (1) you will suddenly be in extreme pain as your nervous system goes into overdrive; (2) your fingers will begin dying further and faster and probably have to be amputated as a result. Again, this is why you would consult a medical professional, in this case an ER or 911 (as I would consider this an emergency) and ask for advice rather before trying ANY fixes.
The other example (a non-emergency one). If you get blood on a garment, you may think that since hot water absorbs better, than you should put the garment in hot water. However, at temperatures just about the human body, blood "cooks". Put the garment in hot water and the blood will cook and probably never come out. Bleach? Bleach causes a chemical reaction that reduces blood to a green residue. Put the garment in bleach and you will end up with a green stain that will probably never come out. The solution here is to place the garment in cold or ice water and let the blood diffuse out. Then its safe to clean normally.
I'm sure there are hundreds or thousands of such gotchas out there, but this should prove a point: when something is really important, it is best not to assume things. Consult experts-- multiple if possible (as even experts make mistakes). Nothing covers bases like an open dialogue. Ultimately, consult because the price of being wrong may be too high.
Peace out.
--Dave
As the guy who wrote this story, just wanted to say thanks to all posters for some excellent discussion. Most of the criticism has been both valid and useful, and we'll try to keep some of these comments in mind for future stories. I also offer a special note of thanks to those who offered extra insight -- I'm the first to concede that a short story like this doesn't cover all the angles on a complex subject like this. Also a really big thanks to those who flamed the critics on the story's behalf.:) If you go through this entire thread, as I have, you'll find a fascinating array of opinions on what to do in the event of a breach, including some that are diametrically opposed. I think the spectrum of views on this proves that it's not all "common sense" stuff that everybody knows. There are some real questions on how to proceed after a breach is detected. I've done my best to summarize some of the comments and offer a few thoughts of my own in today's blog http://www.darkreading.com/blog.asp?blog_sectionid =327&WT.svl=blogger1_1. Hope we can continue the discussion.
mod troll