Slashdot Mirror
We've Been Hacked... or Have We?
hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"
If you don't know what was changed then you need to rebuild any machines suspected of being compromised from scratch and restore the data from a clean backup. Unless you're very sure your file checksum database is accurate (you run tripwire or aide hopefully) you will need to rebuild. Don't screw around and contribute to the global insecurity of the Internet.. especially with people's credit card data at stake!!! WTF is your company thinking keeping that on a public network?
Windows: use Moosoft's The Cleaner (http://www.moosoft.com/products/cleaner/download/ ) - it can detect trojans and things that virus scanners don't pick up...
Linux: nmap the box from a trusted PC on the same network, and then build a copy of netstat on the trusted PC for the server in question... copy the binary over, and run "netstat -pultw" as root... it'll list all ports that are listening for connections, and, the processes that are opening them (by PID, and usually by name). The reason for the clean copy is that a lot of root kits replace netstat on infection...
Hope this helps...
-PhaseBurn Welcome to Linux country. On quiet nights, you can hear windows reboot.
First thing you are going to want before you can go to the boss and shove it up his nose is proof. Put a machine on a monitor port of the switch right beside the egress point of the firewall and sniff away. You know what sort of traffic should be going out so you can filter a lot of that out. You also know that given you're hosting sites, most of your connections are going to be originated from the outside, coming in. So filter those out and look for connections from the inside going out.
Ultimately you're going to need to build up some pretty decent filters or you'll just overwhelm the machine doing the sniffing, but if you know what traffic should look like going across your firewall, you can look for the anomalies quite easily.
Probably the easiest way for you to do this would be to use something like Snort with some pretty decent rules. The downside to IDS is that it takes a lot of effort to get the rules setup properly for your network, but that could easily be done in your spare time, off the clock as it were.
Virus scanners aren't always the best solution to finding back doors and such. On Windows there sure to be malware out there that just hasn't been noticed yet because it doesn't do anything overtly "virus-like". Trojans and malware like NetBus or BackOrifice (to use two very well known examples) are easily altered to hide from many AV apps (just that no one bothers anymore).
Because of this, any machine you think is compromised should be rebuilt and patched up from scratch. Once you've got it rebuilt, and before you put it back on the network, use Tripwire (or even a shell script) to take a baseline MD5 hash of every single windows and application file on your harddrive. This has two advantages. 1) You build up a list of known good files on your machine. 2) It becomes easy to spot new/added files and investigate them. You can do this on any platform, *nix or Windows.
To get the evidence you want though, you're going to need to use tcpdump/snort/ethereal (or any other sniffer you are comfortable with) to prove there is traffic going across your [firewall|egress routers] that shouldn't be.
There are stories just now breaking into the general consciousness of IIS servers that have been compromised for months and feeding executable code tacked on the end of http calls to IE and spreading malicious code and feeding something (reports and opinions vary) to servers in Russia (also with varying opinions as to the owners - organized crime seems to be the evil de jour).
These compromised IIS servers often have the server attached to the explorer.exe process and are therefore not detectable by virus scans. Using netstat or filemon you can find the open ports. The only solution is a bare-metal rebuild.
Have fun if that is what you are dealing with.
As an aside, if this company is unconcerned about the compromise of credit card information you might want to find a new place to hang your hat. The civil and criminal liabilities are pretty steep for the compromise of financial transaction information (if you are in the U.S.) and they extend to individuals inside the company, not just the board and officers.
Those were the days. Adminstrator discovers *one* hacker, catches him, then has enough time to write a book about it before worrying about the next one.
If tits were wings it'd be flying around.
...post the IP address here, and I'm quite certain your worst fears will be so perfectly confirmed that your boss will have no choice but to admit you were right!
--
Don't like it? Respond with words, not karma.
Listen to your boss and do what you do best, web programming. But convince him to get an outside security consultant to scan your network for problems and then listen to their advice.
My company has an outside security company run quarterly checks against our network, and they sometimes catch stuff that I miss. Just don't let them talk you into buying a over-priced checkpoint firewall when all you need is a Linux box and Iptables.
--Ajay
This isn't as much "normalization" as it is "don't take so many drugs when you're designing tables."
You're already doing your job, and not being listened to. Since I'm not a sysadmin, I've got no direct advice for you regarding the tracking of such activity. However, it seems to me that this is the smaller of two problems. The first is being able to do your job from a technical perspective. The second (and it seems, more immediate) problem is being able to do your job from a political perspective.
Your boss has already watched his public facing site(s) and servers go down due to his failure to listen to you. Now it sounds like he's about to make the same fatal mistake. This, of course, places you in the lovely position of having to remind him that he's about to make another major tactical error... but you also have to do so in as subtle manner as possible, so it doesn't sound like a recrimination. All I can suggest is to try to make it blatantly obvious to him without coming out and saying it, thus giving him the opportunity to "discover" his error and correct it on his own.
One other suggestion: document, document, DOCUMENT! Make sure that you can prove later (should it be necessary) that you did everything you could. This is another area of vital importance for your job security that also must be done very carefully. Simply CCing the higher-ups will likely piss off your boss (and possibly the folks you're CCing as well), and may look like unnecessary whistle-blowing or complaining. Do it as unobtrusively as possible, but make sure that you're covered, in case there's any question later.
One possible solution to both problems is to communicate all of your concerns in an e-mail. Write it during your lunch hour, so he can't get upset at you for "wasting" more company time on it. Make it clear exactly why you think there is an issue, and mention (if you can do so gently and without provoking him) the past incident. Remember that if you want someone to do something, you shouldn't tell them why *you* want them to do it. They don't care about what you want. Tell them why *they* want to do it. Best-case scenario, he listens. Worst-case scenario, he gets a little more annoyed at you, but you've got your documentation.
I don't envy your position, and wish you luck.
Get your resume up to date, and get moving. This outfit clearly has no real concern for security, but since you've started poking around, asking questions, and raising hell, you are the messenger that gets shot when the company really does start to take security seriously, if they ever do.
On the other hand, if they continue as they are, they may not survive, and you are screwed again.
Since you're asking "how do I disinfect these boxes" (essentially), you make it pretty clear you don't already know how. You could download Nessus and chkrootkit, and maybe get lucky - e.g. the cracker who's built a nest inside your server doesn't detect your attempts to detect him.
The worst case scenario here is that you detect a problem, attempt to fix it yourself, and trigger Something Bad[tm] in the process: the cracker retaliates, or you break a working app because you upgraded something out-of-sync with glibc (or whatever), or you otherwise become the catalyst for noticeable downtime that will piss off your boss and get you fired, or worse - they turn you into a scapegoat (see the Intel case against that security chap.)
Just make sure you cover your ass. You've notified your boss, copy those emails to a nice safe place (headers and all), and don't do anything stupid.
Best scenario is to build a fresh box, backup the old box's data, restore it to the new box (clean! no code! only data!) Don't bother trying to salvage a compromised O/S installation. Too many things to miss. And, when you're building fresh, don't ssh via one of the infected boxes! Don't inadvertently give *any* info to the crackers that you're setting up a new machine. Better yet, build it with the ethernet cable unplugged, if possible. Do it from CDs.
Big Daddy, Johnny, Burp, Aunt Zelda, Scott, Slurp, Big Momma
All you have to do is stand straight and announce that we've been 'hacked'. If they ask to what extent, how bad etc, just say we've completely been hacked. Its all gone.
You'll be given all the time and budget to fix it. FUD doesnt always require proof, unless someone calls in some consultant.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
Another good rootkit checker, which seems to have a more active development cycle, is Rootkit Hunter. Here's a Newsforge article on it, with a few more details.
A few other comments:
Virus scanners won't help on jot against a custom hack (as Valve found out, for instance). They can be helpful, but don't put full reliance on them.
Running an Intrustion Detection/Prevention System such as Snort, Samhain, Prelude, etc. will help you manage the monitoring side of things; more than a few machines becomes a pain without additional help. Also take a look centralising all your logs on a syslogng server or something similar, if you don't already (note that there are various solutions out there to get Windows boxes to log to a syslog server).
A honeypot may distract the hacker from your production servers for long enough for you to identify that there's a problem.
Also take a look at "HoneyTokens": specifically created database records that trigger alarms if they're accessed - usually high profile fictious targets that would make excellent trophy hacks - there's more info on this over at SecurityFocus.
If you suspect that a machine has been compromised, as other have said, the ONLY WAY TO BE SURE is to rebuild the box from scratch. While this may be a real pain, hopefully it'll help you get the procedures in place to make this as painless as possible, so it's not all bad.
Perform security audits/pentests every now and again. Tools like Nessus help: here's a good series on using Nessus (part 2, part 3).
Get familiar with security tools such as the top 75 recommendations at Insecure.org (home of Nmap).
Remember that security is a PROCESS, so be thorough; get an entire plan together and cover all the bases that you can, taking special care to identify and cover the weak points. Your company's security is only as good as its weakest link; for instance, priviledge escalation of weak user account passwords is a good one.
Read SecurityFocus, PacketStorm, CERT and the like, and try to get involved in their communities; they can be invaluable! They're also got a lot of good tutorials, such as how to lock down Apache, IIS; securing PHP, ASP; etc.