Slashdot Mirror
Forensics On a Cracked Linux Server
This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.
Why Slashdot would such obvious anti-Linux FUD is beyond me. Maybe the M$ advertising dollars are turning their heads.
The bottom line is that a LINUX SERVER CAN'T BE CRACKED.
Maybe this admin got his login info phished by Nigerian scammers, I don't know. The guy probably is wondering why his Ebay account has a bunch of negative feedback and his MySpace is all jacked up and hasn't put 2 and 2 together with that time he responsed to that clever email asking for the triple whammy of MySpace/Ebay/root on your servers so that you could clear the money transfer.
That or he didn't have his updates turned on and had an outdated BIND. And its not like BIND means Linux is unsecure.
Even not that the idea that Linux is crackable is laughable and not worht front page at digg let alone slashdot. You don;t see Technorait or Bruce Perens' site posting garbage like this ever so why slashdot editors can't see thru it i dont kno.
Where did the word forensics come from? This is the completely wrong approach if working forensically. Can slashdot please use not use sensational titles! "Analysis of a cracked box" maybe more appropriate.
* An exploit unknown to the public.
* A user accessing this server from an already compromised host. The attacker could then sniff the the password. It's a very good question, because if the guy was keeping his server up-to-date, then these two are the most likely scenarios.
On tools...it's important to note that in forensics on a Linux box, your friends are ethereal (for watching packets on open connections), netstat (to see what's listening), and strace (shows you what UNIX API calls a running process makes, which gives you very good idea about what's going on.)
Other tools: nmap may be useful for seeing what's going on with 62.101.251.166 and 83.18.74.235. The service detection options, in particular. Always do this on a sandboxed host. Something running in a VM might be useful in this regard.
Anyway, nice article. This is almost exactly how I proceeded when one of my own servers was hacked a few years ago.
My blog
Looks as if there was another way to crash his server...
sPh
Forensics has to be useful in court. This is not - it's tainted evidence. Now if they took the original disk out, copied it with DD or similar to a file and mounted it as loopback and worked on that, then that's a first start to a forensic analysis.
I have rkhunter on all of my machines, sends a nice email letting me know of any changes in system files.
We had a cracked linux server at work one time and I took it upon myself to find out who did it. Long story short: some server monkey decided it would be a fun idea to ride his bike around inside the data center and smashed into one of the racks.
The server seems to be having ... problems with the load (maybe it got cracked again?), here's Google's cache:
: blog.gnist.org/article.php%3Fstory%3DHollidayCrack ing+http://blog.gnist.org/article.php%3Fstory%3DHo llidayCracking&hl=en&ct=clnk&cd=1&gl=us&client=fir efox-a
http://64.233.183.104/search?q=cache:TyrHbOqUhLgJ
Quality, performance, value; you get only two, and you don't always get to pick.
Bruce Schneier posted this a few days back. Consensus is that it's not that good an analysis, but that the attacker was even worse. Some discussion also of whether it is better to take the machine offline immediately (and risk alerting the attacker that he has been rumbled) or to begin your analysis with the machine still live and operational. I for one side with the 'shut that thing down NOW' faction.
Real Daleks don't climb stairs - they level the building.
Oh, I see, it's a clever DOS attack:
1. Infect Linux server of some guy with a blog.
2. Guy blogs about how he dealt with said infection.
3. Blog posting gets linked to on Slashdot.
4. Millions of computers attempt to access the blog, hence bringing down the server.
Don't you see? We've a socially engineered botnet!
(And please, for the love of all that is sacred and funny, don't reply to this and add steps for "???" and "Profit". It's just tired and completely not funny. And the clever little variation on that theme you're thinking about posting right now isn't funny either.)
The shell is a working Bourne shell
I knew it! Jason Bourne was involved in this!
I think it's probably the fact that the owner of this system had the root password set to "GOD" as all good sysadmins do. The hacker's extensive experience hacking the Gibson made getting into this system a cakewalk.
Clearly, we as sysadmins should rethink the long-standing policy of setting all root passwords to either love, secret, sex, or god. Perhaps we should at least add another password to the list, like "unhackable" or something truly secure like that.
http://mirrordot.org/stories/36c2ae6afc0420241dbcc 88a56046a6c/index.html
Raise your hand if you typed "ls -h" on your box just to make sure it still works right.
Does rtkhunter send you a email when the cracker changes /usr/bin/rtkhunter so that it won't email you the attacker's changes?
If you think that rtkhunter will protect you from a Linux kernel module rootkit your completely delusional. NOTHING will _reliably_ locate a LKM rootkit. That's the point of it.
Think about it. Rtkhunter relies on the ability of the kernel to accurately indicate files sizes, file names, and running proccesses as well as a bunch of other little detail things that normal rootkit makers tend to get wrong. When that kernel is subverted and controlled by it's new owner to give rtkunter, as well as other processes (such as your bash shell) false information about the system then those things are completely worthless.
It's the same as virus scanning on Linux (or any other system). Once the attacker gets root access then they have access to the kernel. Once they have access to the kernel they can use the kernel against you to hide what they are doing. Since userspace runs on top of the kernel then any sort of activity can be hidden by making the kernel lie to anything running in userspace.
This includes logging daemons, rootkit detection software, administrators, virus detection, rpm checksums, or anything else that people use to give themselves a FALSE sense of security.
There are two ways to reliably detect a rooted machine.
The first way is to use a network-based Intrusion Detection System (IDS). One of the best ones is commercially supported open source application called Snort. These guys can be hooked up to networks in a passive and completely undetectable way and are used to monitor traffic. They will alert administrators to any unusual network activity.
Network based IDS can be fooled, but as a administrator your at least operating on the same playing feild since your own software isn't used against you.
The second, and more reliable way, is to use a checksum-style IDS. MD5deep, AIDE, or Tripwire are 3 very good examples of this.
However how people use these things are completely worthless. If you keep the checksums and run the checksum software on the same machine as the one your trying to detect, then it's not good. Since they rely on the kernel any kernel-level rootkit can defeat them and the attacker can edit and substitute incorrect checksums.
In order for stuff like AIDE to be usefull it needs to be ran from read-only media and from a different operating system then the one your checking. (for example booted up in a knoppix cdrom, or a removable disk in a dedicated unconnected-to-any-network 'Tripwire' machine)
Both forms of IDS are very expensive and difficult to correctly use. Virtual machines make this stuff somewhat easier, but it's still much better to have dedicated machines for these things.
rtkhunter is nice if it's job is to make you feel good. If it's job is to make sure your machine is secure then it's shit. (no offense to the rtkhunter authors, I am sure they understand it's role and effectiveness.. to bad their users don't tend to) It's only good for kiddies that don't know better and if your being owned by kiddies then you have bigger problems.
Does Ubuntu install selinux and a policy in a default installation, or is it necessary to add it later?
/tmp, lesson learned. But it was interesting to see selinux do its job and I'd be curious if it was utilized in this instance.
I've only performed one Ubuntu install and most of my experience is with Red Hat and Fedora linux distros. Fedora installs selinux with a targeted policy enforcing by default which I think is a good thing. I had an experimental Fedora web server with PHPbb installed which was comprimised via the PHPbb application but looking through the log files it appeared that selinux had thwarted attempts to root the box or setup a zombie to connect to an irc server.
Other than the mistake of an outdated PHPbb application I also made the mistake of allowing execution of code in
All of these will help only if it is cracked by amateur sr1pt k1dd10tz like in this case. If it is cracked properly you will not see anything or spook off the intruder. He will either go underground or destroy the box with all of your data (not that you should try to use it as it may have been altered).
I have seen a number of rootkits for Linux as far back as 97-98 which were considerably more advanced. It was a bit of an arms race between the admins (including me) and the guys who were breaking in. By the end the best rootkits could:
1. Load a whole hidden fs with tools into a ramdisk or hidden area on the filesystem not visible using normal tools.
2. Hide all sockets, processes and files belonging to the rootkit completely. You simply could no longer see them using netstat, ps and other similar tools.
3. Monitor network driver state for the promisc flag and "scrub" backdoor traffic out of it so it is no longer visible using tcpdump and ethereal.
4. Adjust memory totals and df so that you do not see them. This was also the only way we found to catch it. Try to allocate 95% of the remaining free memory and see the system oops magestically.
5. Doctor logs so that you could not notice anything.
6. The rootkit itself handled all connections via something that looked like ssh. I never managed to figure out how it loaded. One of the executables in the system loaded at startup was backdoored. Probably sendmail or one of the other daemons it could not do without.
7. The rootkit managed to masq changed files completely. Tripwire and md5sums were reporting all OK while executables were being changed.
That was a the tech level in 97. I would expect 10 years later a good rootkit to be even better. Looking at the blog post I can only laugh.
If you suspect a system is cracked:
1. Take it offline and take the disks out. Analyse the system completely offline looking at the disk from another system mounted as ro (on SCSI discs use the RO jumper). Never ever even try to start it. Nowdays knoppix is a great help. Most importantly - do not fsck systems before mounting as the rootkit may hide in orphaned areas which fsck will fix.
2. If you are monitoring traffic, monitor it on a switch span port or create yourself a simple multiple interface box which serves as a firewalling bridge (so you can hijack the more interesting bits and alter them). Lex Book PCs are a good choice as they can run either Linux or BSD and are as portable as a laptop. A recent Via with 2 Ethernet ports is also a good choice as it can handle up to 1GB of traffic across as a bridge.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
The 220,000 or so members of the Slashdot Members Who Post Authoritative Statements On The Inner Workings Of Microsoft To Support Their Arguments warmly welcomes you to the club.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
Security is very important to me, I can't be screwing around with something that can be so easily cracked.
If you suspect a system is cracked:
1. Take it offline and take the disks out.
And I've been told don't use the 'shutodwn' command--instead, pull the power plug out of the wall. A rootkit could include a cleanup routine that gets run at shutdown time.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
I work in a large, low-end datacenter. Almost all the servers there are rented buy non-technical people, who for some reason feel qualified to run web hosting businesses. There are so many exploits going on there at any given time, we can't really do anything about it--especially as theoretically the customer is responsible. So when they call in because their server is running slow, we usually find a php hijack happening, tell them their server has been compromised, and suggest that they do something about it.
It's pretty appalling. We would need an army of sysadmins--an army which is currently employed already--to really do something about it. Most of what we see are primitive script kiddie hacks, but guess what--that's good enough, and rarely are the perpetrators hunted down.
Who knows what the more sophisticated hackers are up to!
expandfairuse.org