Slashdot Mirror
Cleaning Up the Mess After a Major Hack Attack
Hugh Pickens writes "Kevin Mandia has spent his entire career cleaning up problems much like the recent breach at Stratfor where Anonymous defaced Stratfor's Web site, published over 50,000 of its customers' credit card numbers online and have threatened to release a trove of 3.3 million e-mails, putting Stratfor is in the position of trying to recover from a potentially devastating attack without knowing whether the worst is over. Mandia, who has responded to breaches, extortion attacks and economic espionage campaigns at 22 companies in the Fortune 100 in the last two years and has told Congress that if an advanced attacker targets your company then a breach is inevitable (PDF), calls the first hour he spends with companies 'upchuck hour' as he asks for firewall logs, web logs, and emails to quickly determine the 'fingerprint' of the intrusion and its scope. The first thing a forensics team will do is try to get the hackers off the company's network, which entails simultaneously plugging any security holes, removing any back doors into the company's network that the intruders might have installed, and changing all the company's passwords. 'This is something most people fail at. It's like removing cancer. You have to remove it all at once. If you only remove the cancer in your leg, but you have it in your arm, you might as well have not had the operation on your leg.' In the case of Stratfor, hackers have taken to Twitter to announce that they plan to release more Stratfor data over the next several days, offering a ray of hope — experts say the most dangerous breaches are the quiet ones that leave no trace."
A bunch of people that had nothing to do with the breach will more than likely end up losing their jobs over it (often the same people that warn about these vulnerabilities beforehand), while the retards that caused the breach, either through their ineptitude or refusal to spend money on proper security, walk away unharmed.
Imagine that you have 1000 employees. Every workstation, every server, every switch, every usb-stick, every external drive could hold the seed to restoring hacker control on your network. You'd have to wipe all of them before allowing them to reconnect to the network.
Then, a week from now, someone asks IT for a file from the off-line backups, and your network is owned again.
I'm curious though. In the PDF Kevin Mandia states that 90% of private enterprises don't know their networks have been compromised until the government (DoD, etc) tell them. So, how does the government know that these companies are compromised ?
I mean, apart from seeing spammy emails coming out, or in the case of the spooks, them seeing information on another system somewhere that's obviously been "stolen" from a US bank or something, how would they know ?
What sort of things would have to happen for a company to get a "Hey, you have bad guys all over your network" visit from the government guys ?
I did work for a Fortune 100 company. We had Disaster Recovery Plans which involve that exact sort of thing. We rehearsed it once for an entire market we operated in, it took about 11 staff and 12 hours to do. We did it during the night during a weekend to reduce impact. Many of the systems were still operating during the rehearsal. We did phones, servers, workstations, restoration of images/backups, phones, network infrastructure, most HSMs (Excluding CAs) etc. As for viruses coming back from at-rest data backups, well, we virus scan them before it's used and nearly all of them are digitally signed (so tampering after the fact is harder) but I can't think of what else you can do. We can load our own heuristics and signatures onto our distributed IDSs though, so if we did find any type malware on our system, we can identify it and add it to the IDS and it would be picked up when/if it was on offline backups and when/if it's restored. The biggest weaknesses we identified was network bottlenecks, outdated documentation and outdated client software to handle the procedures.
Not a problem here. we simply re store the workstation boot image from the creation CD and run all the updates on it.
Thumb drives, not a problem, thumb drives dont work here.
as for switches, I can update ios on every switch in 60 seconds. not a hard thing to do.
as for the "backups" problem. I have yet to see a hacker that can infect a machine using an odf file, I'm not backing up ANY executables.
Honestly I can do a complete wipe and restore in under 5 days for a company that has 1000 employees and 20 servers. IF the IT department was set up and run by competent people.
If it's a typical cluster-turd... far far longer.
Do not look at laser with remaining good eye.
This also includes clean installs on employee portable systems (laptops, PDAs, tablets, phones) as well as anything they have at home that can connect to the corporate network.
Of course, this will never happen.
Then it's time to go through all backup media and sanitize it, since of course a potential future restore could re-initiate the breach.
Of course, this will never happen.
Meanwhile, forensic work needs to be done to figure out what the vector(s) was/were for this incident. It's not enough to just identify and deal with those, however; they need to be studied in context in order to achieve an understanding of what additional, latent vectors exist that could be used.
Of course, this will never happen.
And then it's time for a very pointed session with a copy of Marcus Ranum's "Six Dumbest Ideas in Computer Security", because chances are pretty high that this organization used all six.
Of course, this -- especially this -- will never happen.
Clean installs on everything, new passwords, and don't trust anything executable that has been on the compromised machines anywhere near the time it was hacked. It's not a huge deal here anyway - because this lot have a high profile everyone forgets how small they really are. Your local newspaper probably has a bigger operation and a hell of a lot more subscribers.
I seriously doubt your local newspaper has more money involved - or any local newspaper. Maybe some of the national broadsheets - but that's a moot point.
Cleanups aren't complicated - but fixes are - they just sound simple. And most commonly people seem to believe they are the same thing - I contend that they're not.
In my experience these things happen again and again to the same companies (though the majority put a lot of effort into keeping it secret). Not the same dog each time, but definitely the same leg action.
I've done a bit of due diligence on companies, listened in on workers at lunch, chatted to ex-staff, and hired investigators - and I've found few that are as clean as presented - it's like buying a pub where the bartenders or staff don't dip into the till, or regulars (and staff) have never dealt in drugs (rare as hen's teeth).
I'm not talking about defending against attackers - and I don't dispute that a determined, well resourced, intelligent attack will always succeed if time permits (it's like robbing armoured cash vans really - or so I've heard). I'm talking about the things that make it easy for attackers - I believe that if you raise the bar enough - all the hurdlers don't get better - just a few of them (and when you're robbed you're robbed, so number of occurrences is important)
What interests me is why there's always talk of plugging gaps and fixing procedures - but never any mention of fixing the primary problem. The primary problem being institutional psychology. Like storing your beer on the nature strip it having it stolen (surprise - people want your beer). Then "cleaning up" by making sure all liquor is secured inside the premises, and "fixing" the problem by telling people to store their beer in the fridge and lecturing them on physical security. It overlooks the possibility that only an untrustworthy idiot would put beer on the nature strip in the first place - and even if they don't put it on the nature strip again they will probably lose a house key, or leave a window open.
*1 I don't believe lazy, stupid staff change if you send them to motivation and inspiration seminars either, certainly I've seen no evidence to support it.
I'm working on a theory that dumb travels downward - I call it "The Argument from Moron Motion"
> Imagine that you have 1000 employees. Every workstation, every server, every switch, every usb-stick, every external drive could hold the seed to restoring hacker control on your network. You'd have to wipe all of them before allowing them to reconnect to the network.
I wish people would remember this when they claim company's estimates of damage from a cracked system are excessive. You can bring an entire company to a standstill for an extended period of time by requiring (unless as a customer you're just fine with them taking risks with your data?) multiple critical systems to be isolated and rebuilt from scratch at the same time, even if there's no clear damage done, because you have no other way of verifying they're clear.
In a high security environment, destroying the physical machines to be sure (tampered firmware, stuff hidden in bad blocks on the hard drive, or who knows what else) is probably a sensible move.
Uhm no, mere vandals need to be cherished and promoted; those who work for the Chinese govt won't tell you something is amiss.
It is the companies' fault for not following basic security practices, especially if what they take taxpayers' money for is "intelligence".
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
Like Kevin Mandia, I too clean up these messes professionally. Cleaning these things up starts with the data gathering and analysis, virus scans, offline analysis - and more that are not mentioned.
The MOST important thing that ANY admin should know is that the true professional hackers do not use rootkits. They will use exploits to gain their foothold, but rather than install a rootkit, they will install remote network admin utilities, such as Dameware NT utilities (old), or more recently I've seen LabTech Software.
This software is great for Managed Service Providers - it also is a dream come true for cyber-criminals as it provides a backdoor into networks using signed code that will not appear on any antivirus, anti-malware or anti-rootkit scan. It can sit dormant for years, get backed up, and restored. Even if you do run anti-virus scans on your backups prior to restoring them - as one commenter stated above - it would be of no use.
So, when I am gathering the data dump, what I do is look for ALL network management tools, and I have created scripts that search for these.
*****
Google this: C:\WINDOWS\LTSVC\LTSVC.exe Hijackthis
You will find examples of people who have run Hijackthis on their computer and posted the log online - the common complaint is that they keep getting reinfected and cannot figure out how. They've run {insert virus tools here} a number of times and cannot figure it out. They usually resort to reinstalling the OS.
*****
Anyhow - gathering up all the logs from every device on the network, linking how they went from machine-to-machine, enumerating lists of installed software on each machine, and also performing offline analysis of drives, searching for any file/directory modifications based upon time stamp. It is FAR more involved, but it is the only way to enumerate the intrusion.
Removal must be done all at once. Either cut the network access of all the devices, then remove, or write a custom removal script and schedule it as a task to have everything be done at precisely the same moment.
I then have custom IDS signatures that look for any unauthorized Remote Management & Monitoring software.
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
experts say the most dangerous breaches are the quiet ones that leave no trace.
You would not have known.
In fact, security experts would like that to be your last thought before you go to sleep at night, and your first thought when you wake up, and uppermost in your mind when they pad your bill with zeroes.
Honestly I can do a complete wipe and restore in under 5 days for a company that has 1000 employees and 20 servers.
That's not too bad. But of course any machine that's not been wiped and restored can not be allowed on the network. And for the employees that means up to five days of not being able to do much. That's a long time to wait.
What you're doing, although I don't think you intended to, is making excuses as to why those six mistakes are necessary. This is a fatal error. By justifying them, you ignore the consequences -- which are that you've just about guaranteed that you will be hacked the first time someone with sufficient expertise and resources decides to target you.
The trick is to recognize that you cannot make these mistakes. Period. No matter who you have to run over, who you have to piss off, who you have to overrule, who you have to upset, no matter what. You have to be, and yes I am, an arrogant bastard. Because the moment you compromise, you're doomed. We've seen it over and over and over and over again, we're seeing it again today, we'll see it again tomorrow. Every single data breach incident I've ever read about included at least one of those six mistakes, and most of them included several. Yet incompetent, weak-willed IT people insist on making them because "we've always done it this way" or "that can't work!" or "but it would break..." or for a thousand other reasons...none of which matter. (What good is having a spiffy computing environment if it's not secure?)
The problem isn't that we don't know what to do. We do. The problem is lack of will to do it.