Slashdot Mirror
We've Been Hacked... or Have We?
hidden_fire asks: "I recently got a job as a Web Programmer at a web company that hosts many sites. The company had many badly firewalled Windows and Linux servers without any security patches, and a shared administrator password. I warned them that they needed to improve their security, but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised, and the Sasser Worm took us offline. Now, I've been allowed to rebuild the compromised box and tighten our firewalling, but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, a Linux kernel lockup, strange ports being open, and performance issues. I think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security. Has anybody else been in this situation? How can I detect these guys if their tools don't show in virus scans?"
If you don't know what was changed then you need to rebuild any machines suspected of being compromised from scratch and restore the data from a clean backup. Unless you're very sure your file checksum database is accurate (you run tripwire or aide hopefully) you will need to rebuild. Don't screw around and contribute to the global insecurity of the Internet.. especially with people's credit card data at stake!!! WTF is your company thinking keeping that on a public network?
Clifford Stoll, Cuckoos Egg.
a cking_a_Spy_Through_the_Maze_of_Computer_Espionage _0743411463.html
Lesson to learn is secure that stuff, what they don't know will hurt them.
http://www.programming-reviews.com/Cuckoos_Egg_Tr
Windows: use Moosoft's The Cleaner (http://www.moosoft.com/products/cleaner/download/ ) - it can detect trojans and things that virus scanners don't pick up...
Linux: nmap the box from a trusted PC on the same network, and then build a copy of netstat on the trusted PC for the server in question... copy the binary over, and run "netstat -pultw" as root... it'll list all ports that are listening for connections, and, the processes that are opening them (by PID, and usually by name). The reason for the clean copy is that a lot of root kits replace netstat on infection...
Hope this helps...
-PhaseBurn Welcome to Linux country. On quiet nights, you can hear windows reboot.
Here's a couple of things you could do:
Download and build chkrootkit. This will detect a lot (most?) stealthed kits on Linux systems, and it is always my first port of call when I'm invited in to clean up after a breakin.
Plug in a hub (so all traffic can be seen by multiple machines - a switch ain't as good, unless it has a monitoring port) in front of the machine(s) and run tcpdump or ethereal on another system to watch traffic from the machine. This will let you watch exactly what traffic is happening on those weird ports, or watch outbound SMTP traffic for spammer activity.
We don't put Windows-based systems on the internet, partly for security reasons, and partly because we don't have any Windows specialists, so I can't help for on-the-box detection there, although I would expect a commercial virus scanner should find everything.
I was looking into ways of approaching this type of problem, and it seems that a good firewall monitoring solution could help. One I came across is made by OnGarde. They run a service that allows deep inspection of traffic, and they support alot of devices.
First thing you are going to want before you can go to the boss and shove it up his nose is proof. Put a machine on a monitor port of the switch right beside the egress point of the firewall and sniff away. You know what sort of traffic should be going out so you can filter a lot of that out. You also know that given you're hosting sites, most of your connections are going to be originated from the outside, coming in. So filter those out and look for connections from the inside going out.
Ultimately you're going to need to build up some pretty decent filters or you'll just overwhelm the machine doing the sniffing, but if you know what traffic should look like going across your firewall, you can look for the anomalies quite easily.
Probably the easiest way for you to do this would be to use something like Snort with some pretty decent rules. The downside to IDS is that it takes a lot of effort to get the rules setup properly for your network, but that could easily be done in your spare time, off the clock as it were.
Virus scanners aren't always the best solution to finding back doors and such. On Windows there sure to be malware out there that just hasn't been noticed yet because it doesn't do anything overtly "virus-like". Trojans and malware like NetBus or BackOrifice (to use two very well known examples) are easily altered to hide from many AV apps (just that no one bothers anymore).
Because of this, any machine you think is compromised should be rebuilt and patched up from scratch. Once you've got it rebuilt, and before you put it back on the network, use Tripwire (or even a shell script) to take a baseline MD5 hash of every single windows and application file on your harddrive. This has two advantages. 1) You build up a list of known good files on your machine. 2) It becomes easy to spot new/added files and investigate them. You can do this on any platform, *nix or Windows.
To get the evidence you want though, you're going to need to use tcpdump/snort/ethereal (or any other sniffer you are comfortable with) to prove there is traffic going across your [firewall|egress routers] that shouldn't be.
but our other servers show many signs of possibly being compromised including unexplained outgoing traffic, ... strange ports being open...
Perhaps I'm being naive, but what's preventing you from using 'netstat', 'nmap -sV', and plain old 'netcat' to figure out what the strange ports are doing?
There are stories just now breaking into the general consciousness of IIS servers that have been compromised for months and feeding executable code tacked on the end of http calls to IE and spreading malicious code and feeding something (reports and opinions vary) to servers in Russia (also with varying opinions as to the owners - organized crime seems to be the evil de jour).
These compromised IIS servers often have the server attached to the explorer.exe process and are therefore not detectable by virus scans. Using netstat or filemon you can find the open ports. The only solution is a bare-metal rebuild.
Have fun if that is what you are dealing with.
As an aside, if this company is unconcerned about the compromise of credit card information you might want to find a new place to hang your hat. The civil and criminal liabilities are pretty steep for the compromise of financial transaction information (if you are in the U.S.) and they extend to individuals inside the company, not just the board and officers.
...post the IP address here, and I'm quite certain your worst fears will be so perfectly confirmed that your boss will have no choice but to admit you were right!
--
Don't like it? Respond with words, not karma.
Listen to your boss and do what you do best, web programming. But convince him to get an outside security consultant to scan your network for problems and then listen to their advice.
My company has an outside security company run quarterly checks against our network, and they sometimes catch stuff that I miss. Just don't let them talk you into buying a over-priced checkpoint firewall when all you need is a Linux box and Iptables.
--Ajay
In case you didn't know, that's not a Voltaire quote. It's a description of Voltaire's beliefs, as written by Evelyn Beatrice Hall in "Friends of Voltaire".
LOAD "SIG",8,1
This isn't as much "normalization" as it is "don't take so many drugs when you're designing tables."
You're already doing your job, and not being listened to. Since I'm not a sysadmin, I've got no direct advice for you regarding the tracking of such activity. However, it seems to me that this is the smaller of two problems. The first is being able to do your job from a technical perspective. The second (and it seems, more immediate) problem is being able to do your job from a political perspective.
Your boss has already watched his public facing site(s) and servers go down due to his failure to listen to you. Now it sounds like he's about to make the same fatal mistake. This, of course, places you in the lovely position of having to remind him that he's about to make another major tactical error... but you also have to do so in as subtle manner as possible, so it doesn't sound like a recrimination. All I can suggest is to try to make it blatantly obvious to him without coming out and saying it, thus giving him the opportunity to "discover" his error and correct it on his own.
One other suggestion: document, document, DOCUMENT! Make sure that you can prove later (should it be necessary) that you did everything you could. This is another area of vital importance for your job security that also must be done very carefully. Simply CCing the higher-ups will likely piss off your boss (and possibly the folks you're CCing as well), and may look like unnecessary whistle-blowing or complaining. Do it as unobtrusively as possible, but make sure that you're covered, in case there's any question later.
One possible solution to both problems is to communicate all of your concerns in an e-mail. Write it during your lunch hour, so he can't get upset at you for "wasting" more company time on it. Make it clear exactly why you think there is an issue, and mention (if you can do so gently and without provoking him) the past incident. Remember that if you want someone to do something, you shouldn't tell them why *you* want them to do it. They don't care about what you want. Tell them why *they* want to do it. Best-case scenario, he listens. Worst-case scenario, he gets a little more annoyed at you, but you've got your documentation.
I don't envy your position, and wish you luck.
Get your resume up to date, and get moving. This outfit clearly has no real concern for security, but since you've started poking around, asking questions, and raising hell, you are the messenger that gets shot when the company really does start to take security seriously, if they ever do.
On the other hand, if they continue as they are, they may not survive, and you are screwed again.
Since you're asking "how do I disinfect these boxes" (essentially), you make it pretty clear you don't already know how. You could download Nessus and chkrootkit, and maybe get lucky - e.g. the cracker who's built a nest inside your server doesn't detect your attempts to detect him.
The worst case scenario here is that you detect a problem, attempt to fix it yourself, and trigger Something Bad[tm] in the process: the cracker retaliates, or you break a working app because you upgraded something out-of-sync with glibc (or whatever), or you otherwise become the catalyst for noticeable downtime that will piss off your boss and get you fired, or worse - they turn you into a scapegoat (see the Intel case against that security chap.)
Just make sure you cover your ass. You've notified your boss, copy those emails to a nice safe place (headers and all), and don't do anything stupid.
Best scenario is to build a fresh box, backup the old box's data, restore it to the new box (clean! no code! only data!) Don't bother trying to salvage a compromised O/S installation. Too many things to miss. And, when you're building fresh, don't ssh via one of the infected boxes! Don't inadvertently give *any* info to the crackers that you're setting up a new machine. Better yet, build it with the ethernet cable unplugged, if possible. Do it from CDs.
Big Daddy, Johnny, Burp, Aunt Zelda, Scott, Slurp, Big Momma
Anyways, the subject says it all. Its very possible that you have been comprimised if your kernel is panicing. I think its time to boot from a safe copy of your distro (or some other custom distro...etc) and confirm the checksums of everything and do a good scan of the filesystems.
Also, one good thing to do is place a clean box in between the comprimised server and its internet connect and run tcpdump/tethereal on the brided connection. The first thing you need to do is be able to identify _all_ traffic going in and out of the box. If you dont know what something is, research it. You never know what you might stumble upon.
Good luck.
Imagine you are a chef. Maybe not the world's greatest but you can sear a mean steak. You've been out of work for a while and you find a nice-looking restaurant, do the interview, and you're hired.
Once you step into the kitchen, you want to throw up. Cockaroaches everywhere. Nobody washes anything. They scrape off the filth before using the pots and then toss them in a pile. When the wind is right, you smell rotting food.
You point this out and the boss says, sorry you're a chef, not the cleaning crew, mind your business.
You have two choices:
#1: do what the boss says (and DOCUMENT all conversations because at this kind of outfit, you will be the first to get blamed when the restaurant gets sued by a sick and angry customer).
#2: quit, find another job, even if you have to take a pay cut to find one. Find a place that harmonizes with YOUR idea of a good place to work.
I would take #2 every time. Your example is not the kind of place I'd like to work, either as a programmer or a sysadmin. EVERYBODY should be acutely aware of security issues, and willing to hear anything that improves security. It's like not having insurance, or bathing in sewage with open wounds. You'll eventually regret it.
All you have to do is stand straight and announce that we've been 'hacked'. If they ask to what extent, how bad etc, just say we've completely been hacked. Its all gone.
You'll be given all the time and budget to fix it. FUD doesnt always require proof, unless someone calls in some consultant.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
think we are possibly providing hosting for undetectable spammers but the boss thinks I'm paranoid, and says that I need to be working on paying work, not security.
You've got a money hemorrage on your servers. Don't bother trying to talk sense to your boss, just strengthen up your arms and be read to jump ship and swim when the next boat comes along. Start shopping your resume now, and remember: when everyone is out to get you, paranoia is just good sense.
Honestly letting a credit card server be comprimised and not doing anything about it until it becomes urgent. Then ignoring more possible exploits? If I was a customer I would be might pissed. Buy something from your company and then go get you a lawyer.
...but was ignored until a hacker kindly emailed them proof that their credit-card server was compromised.....and says that I need to be working on paying work, not security.....
Tell him that keeping your servers secure IS paying work. As a customer, if I know that a company is not keeping my credit cards, etc. secure, I will not do business with them. Ask him what would happen if there was a big 'scandal' (for lack of better terms), and thousands of customers credit was exploited. How much of an opportunity would you have to sell your so called 'paying' work then?
At my job, I am one of two web developers. Besides us, there are the two owners and our systems admin. The owners want to become a viable commercial hosting service with secure storefronts, etc. Fine says me.
The problem lies in that one of the two owners (The husband) is a pig-headed idiot. Recently he asked us to implement a RAID solution for the webserver (notice the lack of an 's' at the end of webserver). Not a problem says sysadmin and myself, we come up with the plan, and present it to him, it involves RAID-5, blah, blah, blah, all the standard normal stuff that people do...He quickly scoffs at the idea, hands us a OLD P.O.S. with a couple 10'ish gig drives and says make it out of this and use RAID-1, and promises to order some large drives for the machine, so we can implement his RAID-1 solution but insists that nobody in their right mind would ever use RAID-5...
We of course are like what the hell? You want something that is enterprise level, and expect us to make it out of this P.O.S.???
We resign ourselves to doing the best we can with it and get a crummy webserver up and running with Slackware.. It is not the fastest machine, but it works for now. We currently have no RAID, becuase the large drives he said he would order have not been ordered for 2 months.
A couple weeks ago, we are talking about a file server for our internal software, etc. and he loves the idea..GREAT!!! So we spec out a modest system that will fulfill our needs and he says, oh I have a perfectly good fileserver at home that you guys can use to make it, and the COOL thing about it is that it runs on something similar to (but not) this. We research his little linux memory card thing and yes, it is cool, however it is not capable of doing what we need to have it do, and from what we can find out about it online it is not capable of performing one of the tasks without substantial work being done, the least of which is compiling and installing netatalk (which is no big deal) for some machines that cannot connect via Samba (MacOS 9 that would require DAVE that he is not willing to purchase).
So we add in the 2 120-Gig Drives to the PII/166 with 64M RAM that he gave us to make a file server from and find out that
So we switch out to some other P.O.S. motherboard he has lying around and find that it has some popped caps, so it won't work either.
We eventually come up with a working P.O.S. motherboard, put some WRONG entries in the BIOS to make it recognize the 120Gig Drives and install a 4.xG drive to use as the system drive running samba and netatalk. All is looking well...
So we get FreeBSD installed today and are in the process of setting up the Xserver so he can have his GUI, since he doesn't know jack about the command line and then we are going to implement his RAID-1 that he loves so well on this machine also. We are stoked to say the least that after all that hassle we have a working system and FreeBSD sees the entirity of the 120G drives...
So he comes into work today and sees XWindows (twm) up and running and asks what we are doing, and we tell him, making the file server like we had talked about. He asks what it is running and we tell him FreeBSD 5.2.1 and we are finishing the config, then implementing the RAID and it will be ready to go. This is where it gets good.
He flies off the handle and says, did I tell you to use FreeBSD?? We are like...ummmmmmm...we talked about that memory card thing and explained to you why it wouldn't work and therefore have implemented a solution that fits our needs like we talked about. His response was "I told you
you need to nuke and Pave the OS's sumultainiously if possible, and restore data from a trusted backup if you have it, otherwise notify customers that THEY need to restore from THEIR backup (they do backup their data right?).
Snowden and Manning are heroes.
Forget running ethereal or other packet sniffers, they're far too fine-grained for what you're trying to do: prove they're being abused.
Connect a small box running your favorite *nix running ntop to the service port (or whatever they call it, I'm half asleep) of the switch/router to which the box(es) in question are connected.
That's it.
Ntop will give you very nice graphs to print out for the guy who needs a clue, showing not only the IPs involved, but the ports, percentages of traffic per protocal/port/whatever.
If they're being used for SPAM, for example, you'll see tons of outgoing port-25 traffic.
Just make sure that's not what the box is supposed to be doing!
If this isn't enough for him, and you're sure this is something against the company's policy, bring it to his boss' attention immediately (calmly) with a full explanation of why you are doing so.
If you delay this action, you risk having your boss "poison" his perception of the situation, and end up thinking that you're a troublemaker (that is, if the politics are as bad as I'm guessing they are there).
So either fight it all the way, or drop it... or be prepared to find a new job (these situations always suck).
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
Security is paying work, unless your company has a reputation without worth. You can get that sentiment out of any of the good security books out there, but saddly management can only see the spending cycle, not the averted crash-burn-patch-fix cycle. Remind him that if we were all "paranoid" to begin with, we wouldn't have had a hacker tell us to patch our stuff. What if the hacker didn't like you so much?
Security really isn't a game or a buzz word. Folks have to own that concept, not just parrot it.
Sam
I was in a very similar situation awhile back when the call center I worked for had some major internal and external network security issues. When I initially stumbled apon this I emailed the IT dept explaining the issue and I was quickily brushed off and told it wasn't an issue. Not being satisfied with that I dug a little deeper and was fired for it. They ended up making wild accusations regarding my intentions etc etc.
So my best advise to you is that if it's not in your job description, leave it the fuck alone. You don't want to end up being the scape goat should something happen or should you step on someones toes in the process of honestly trying to help. Besides; if they want network security, they need to open a position and pay for it.
tell him about rbl's and how his company would loose the ability to reliably send email if your netblock ended up on some of them. also, install and run ethereal and/or ntop and see what is really going on on your network
But seriously, get a Snort box installed and be more active(*) in your intrusion detection. Surely your boss can't object if you slap down a printed snort long on his desk, and show him proof of intrusion.
dave
(*) I absolutely refuse to use the word 'proactive'. I'm not playing buzzword bingo here.
It's the only way to be sure.
http://neokosmos.blogsome.com
I've seen all the usual goodies mentioned (Ethereal, tcpdump, nmap, nessus) but I found when it comes to detecting a virus infection I find Etherape impossible to beat.
The main reason for that is that etherape actually maps out live traffic patterns (and volume) on a network. I personally am quite surprised it's not a default part of more security bootdisks. You'll spot an infection straightaway as they tend to 'broadcast' on the network.
As for protection, host checksumming has one disadvantage: it's AFTER the fact...
Good luck - you'll need it. You may want to remind your boss that being negligent with CC data might lose you your merchant status, and losing customer date could result in lawsuits from clients or, in EU countries, a harsh fine from data protection regulators. IANAL but frequently involved in cleanign up situation you just described...
Insert
I've seen situations like this in the past many times, especially in smaller companies where the boss started it and built it up by himself and feels the need to micro-manage.
What I said to one guy like that is "Sir, we respect you and will do whatever you want us to do here, because you're the boss and it's your company, but you hired us to take the load off you so you'd be able to do less work and make more money. Trust us to do a good job and we will. We might not do everything exactly the same as you would, but we know the end result will still make you happy."
Of course, it takes more than that to change a personality flaw that massive, but it's a good start.
I'm not normally an irrational zealous dickhead, but I figure "When in Rome..."
Your boss seems to be interested in the bottom line $$$. So thats the answer to the problem. It seems (not being a lawyer) that your company would be responsible for securing the credit card data. A large financial company recently had to disclose to its customers that it (or rather the offsite storage company) had lost one of its backup containing customer information.
Even if they don't take the CC data, its costing your company bandwidth (you can't serve up your data if your pipe is full serving the spammers and viruses), disk space (with viruses), CPU time from running those "extra" programs, and downtime either from being hacked by someone who is tired of your contributions to the network, your provider taking you offline because your causing them problems from the GIGO, or the worm itself killing the system.
It will also cost your company lost revenue due to the loss of business from your customers moving else where due to the services issues resulting from the above.
Even if he does want you to work on something that will "profit" the company, how much will it cost vrs save the company from bringing in a consultant to evaluate the systems, practics and policies? I bet its cheaper than going out of business.
lol, heyyy share your server with us! i want some 0days hahahaha