Slashdot Mirror
We've Been Hacked... or Have We?
hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"
If you don't know what was changed then you need to rebuild any machines suspected of being compromised from scratch and restore the data from a clean backup. Unless you're very sure your file checksum database is accurate (you run tripwire or aide hopefully) you will need to rebuild. Don't screw around and contribute to the global insecurity of the Internet.. especially with people's credit card data at stake!!! WTF is your company thinking keeping that on a public network?
Clifford Stoll, Cuckoos Egg.
a cking_a_Spy_Through_the_Maze_of_Computer_Espionage _0743411463.html
Lesson to learn is secure that stuff, what they don't know will hurt them.
http://www.programming-reviews.com/Cuckoos_Egg_Tr
Listen to your boss and do what you do best, web programming. But convince him to get an outside security consultant to scan your network for problems and then listen to their advice.
My company has an outside security company run quarterly checks against our network, and they sometimes catch stuff that I miss. Just don't let them talk you into buying a over-priced checkpoint firewall when all you need is a Linux box and Iptables.
--Ajay
This isn't as much "normalization" as it is "don't take so many drugs when you're designing tables."
You're already doing your job, and not being listened to. Since I'm not a sysadmin, I've got no direct advice for you regarding the tracking of such activity. However, it seems to me that this is the smaller of two problems. The first is being able to do your job from a technical perspective. The second (and it seems, more immediate) problem is being able to do your job from a political perspective.
Your boss has already watched his public facing site(s) and servers go down due to his failure to listen to you. Now it sounds like he's about to make the same fatal mistake. This, of course, places you in the lovely position of having to remind him that he's about to make another major tactical error... but you also have to do so in as subtle manner as possible, so it doesn't sound like a recrimination. All I can suggest is to try to make it blatantly obvious to him without coming out and saying it, thus giving him the opportunity to "discover" his error and correct it on his own.
One other suggestion: document, document, DOCUMENT! Make sure that you can prove later (should it be necessary) that you did everything you could. This is another area of vital importance for your job security that also must be done very carefully. Simply CCing the higher-ups will likely piss off your boss (and possibly the folks you're CCing as well), and may look like unnecessary whistle-blowing or complaining. Do it as unobtrusively as possible, but make sure that you're covered, in case there's any question later.
One possible solution to both problems is to communicate all of your concerns in an e-mail. Write it during your lunch hour, so he can't get upset at you for "wasting" more company time on it. Make it clear exactly why you think there is an issue, and mention (if you can do so gently and without provoking him) the past incident. Remember that if you want someone to do something, you shouldn't tell them why *you* want them to do it. They don't care about what you want. Tell them why *they* want to do it. Best-case scenario, he listens. Worst-case scenario, he gets a little more annoyed at you, but you've got your documentation.
I don't envy your position, and wish you luck.
Get your resume up to date, and get moving. This outfit clearly has no real concern for security, but since you've started poking around, asking questions, and raising hell, you are the messenger that gets shot when the company really does start to take security seriously, if they ever do.
On the other hand, if they continue as they are, they may not survive, and you are screwed again.
Anyways, the subject says it all. Its very possible that you have been comprimised if your kernel is panicing. I think its time to boot from a safe copy of your distro (or some other custom distro...etc) and confirm the checksums of everything and do a good scan of the filesystems.
Also, one good thing to do is place a clean box in between the comprimised server and its internet connect and run tcpdump/tethereal on the brided connection. The first thing you need to do is be able to identify _all_ traffic going in and out of the box. If you dont know what something is, research it. You never know what you might stumble upon.
Good luck.
All you have to do is stand straight and announce that we've been 'hacked'. If they ask to what extent, how bad etc, just say we've completely been hacked. Its all gone.
You'll be given all the time and budget to fix it. FUD doesnt always require proof, unless someone calls in some consultant.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
He's paying you. So you're sort of obligated to tolerate him. Occasionally my boss buys unusual things but never like that. The best you can do is to educate him and do a good job despite his mistakes.
Maybe ask for an IT budget whereby you and the others get a fixed amount of cash to spend on hardware, since he's obviously paranoid about spending (forgetting that he pays you tons more to deal with the crap he buys) and wants to avoid getting something more expensive than it needs to be, and you want to avoid spending on things that don't meet your needs. Even a measly $500 can go a long way when you, the expert, has to do the shopping.
I've seen situations like this in the past many times, especially in smaller companies where the boss started it and built it up by himself and feels the need to micro-manage.
What I said to one guy like that is "Sir, we respect you and will do whatever you want us to do here, because you're the boss and it's your company, but you hired us to take the load off you so you'd be able to do less work and make more money. Trust us to do a good job and we will. We might not do everything exactly the same as you would, but we know the end result will still make you happy."
Of course, it takes more than that to change a personality flaw that massive, but it's a good start.
I'm not normally an irrational zealous dickhead, but I figure "When in Rome..."