Slashdot Mirror
Forensics On a Cracked Linux Server
This blog entry is the step-by-step process that one administrator followed to figure out what was going on with a cracked Linux server. It's quite interesting to me, since I have had the exact same problem (a misbehaving ls -h command) on a development server quite a while back. As it turns out, my server was cracked, maybe with the same tool, and this analysis is much more thorough than the one I was able to do at the time. If you've ever wondered how to diagnose a Linux server that has been hijacked, this short article is a good starting point.
Forensics has to be useful in court. This is not - it's tainted evidence. Now if they took the original disk out, copied it with DD or similar to a file and mounted it as loopback and worked on that, then that's a first start to a forensic analysis.
I have rkhunter on all of my machines, sends a nice email letting me know of any changes in system files.
http://www.sans.org/reading_room/whitepapers/honor s/1491.php?portal=ca149743c753474b2eff202cff878a48
The server seems to be having ... problems with the load (maybe it got cracked again?), here's Google's cache:
: blog.gnist.org/article.php%3Fstory%3DHollidayCrack ing+http://blog.gnist.org/article.php%3Fstory%3DHo llidayCracking&hl=en&ct=clnk&cd=1&gl=us&client=fir efox-a
http://64.233.183.104/search?q=cache:TyrHbOqUhLgJ
Quality, performance, value; you get only two, and you don't always get to pick.
Bruce Schneier posted this a few days back. Consensus is that it's not that good an analysis, but that the attacker was even worse. Some discussion also of whether it is better to take the machine offline immediately (and risk alerting the attacker that he has been rumbled) or to begin your analysis with the machine still live and operational. I for one side with the 'shut that thing down NOW' faction.
Real Daleks don't climb stairs - they level the building.
http://mirrordot.org/stories/36c2ae6afc0420241dbcc 88a56046a6c/index.html
All of these will help only if it is cracked by amateur sr1pt k1dd10tz like in this case. If it is cracked properly you will not see anything or spook off the intruder. He will either go underground or destroy the box with all of your data (not that you should try to use it as it may have been altered).
I have seen a number of rootkits for Linux as far back as 97-98 which were considerably more advanced. It was a bit of an arms race between the admins (including me) and the guys who were breaking in. By the end the best rootkits could:
1. Load a whole hidden fs with tools into a ramdisk or hidden area on the filesystem not visible using normal tools.
2. Hide all sockets, processes and files belonging to the rootkit completely. You simply could no longer see them using netstat, ps and other similar tools.
3. Monitor network driver state for the promisc flag and "scrub" backdoor traffic out of it so it is no longer visible using tcpdump and ethereal.
4. Adjust memory totals and df so that you do not see them. This was also the only way we found to catch it. Try to allocate 95% of the remaining free memory and see the system oops magestically.
5. Doctor logs so that you could not notice anything.
6. The rootkit itself handled all connections via something that looked like ssh. I never managed to figure out how it loaded. One of the executables in the system loaded at startup was backdoored. Probably sendmail or one of the other daemons it could not do without.
7. The rootkit managed to masq changed files completely. Tripwire and md5sums were reporting all OK while executables were being changed.
That was a the tech level in 97. I would expect 10 years later a good rootkit to be even better. Looking at the blog post I can only laugh.
If you suspect a system is cracked:
1. Take it offline and take the disks out. Analyse the system completely offline looking at the disk from another system mounted as ro (on SCSI discs use the RO jumper). Never ever even try to start it. Nowdays knoppix is a great help. Most importantly - do not fsck systems before mounting as the rootkit may hide in orphaned areas which fsck will fix.
2. If you are monitoring traffic, monitor it on a switch span port or create yourself a simple multiple interface box which serves as a firewalling bridge (so you can hijack the more interesting bits and alter them). Lex Book PCs are a good choice as they can run either Linux or BSD and are as portable as a laptop. A recent Via with 2 Ethernet ports is also a good choice as it can handle up to 1GB of traffic across as a bridge.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
If you suspect a system is cracked:
1. Take it offline and take the disks out.
And I've been told don't use the 'shutodwn' command--instead, pull the power plug out of the wall. A rootkit could include a cleanup routine that gets run at shutdown time.
Dear Slashdot: next time you want to mess with the site, add a rich-text editor for comments.
Correct. Always pull the plug out of the wall the moment you suspect that something is wrong. This is what I meant when I said - take it offline (my fault, should have written it better). If it is compromised the data on it is worthless anyway and you need to go back to backups so the loss of data from pulling the plug is trully in the "who cares" area.
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
Ubuntu, as of the latest version (Feisty Fawn), does not install SELinux. If you want that functionality, you'll have to install it yourself. I think this is because SELinux policies can be difficult for beginning users to navigate. Also, when SELinux thwarts execution of some file, there is often no explicit message stating that the file was blocked by SELinux, please change your configuration. In all too many cases, the user is left on their own to figure out why their file isn't executing.
We all know what to do, but we don't know how to get re-elected once we have done it