Slashdot Mirror
Worst Security Clean-Up You've Performed?
nakhla writes "Last night, I was tasked (by my wife) to help fix her friend's computer. It is a Windows XP home system which has been running slowly, almost to the point of un-usability (like *that's* never happened before). It turns out that hundreds of random processes had filled up its meager 256 MB of RAM. The cause? Nearly 7,500 viruses and worms that had infected the system. That number doesn't even include the hundreds of spyware and adware programs that had installed themselves, as well. Although the box is now behind a firewall, that wasn't always the case. This was, by far, the most infected system I'd ever seen, but I'm sure it can't be the worst ever. What was the worst security cleanup you ever had to perform?"
Worst cleanup by far was on a corporate Windows server in 2000 or 2001. The system did not have any anti-virus, and doubled as a SQL Server and File server. A couple viruses got on the drive and started trashing files. Unfortunately, they had been on there for months before anyone noticed, so backups were basically useless. We had to go file by file to retrieve important data, and then have users manually validate exported/imported SQL Server data. Uggghhhh. It took us months before everything was sorted out, but it was an easy sell to get the client onto Oracle and a HP-UX system soon after.
Crack - Free with every butt and set of boobs
Geek Squad. One customer had 35,000 pieces of spyware and over 3000 instances of some 30 or 40 viruses on her computer, some of which required some alternative methods to remove since they were locked when in safe mode and encrypted so you couldn't scan with a boot CD. After 4 scans taking about 6 hours I managed to get the spyware gone, and also inbetween had made note of viruses I needed to manually purge. Cleaned it up nice; meanwhile my supervisor was telling me to call the customer and tell them we needed to just reinstall Windows.
My aunt got AOL with anti-spyware and firewall and security. Eventually she had 35 different viruses, managed to remove all but 28 unique signatures (this was before I developed my brute-force removal method). Chucked a ton of spyware too.
While at WhiteWolf Security, we had a little game going; eventually our opponents got pissed at us for unrelated reasons and decided to physically break into WhiteWolf at 4am. They shorted CMOS pins and used boot CDs to evade password lock-outs, adding extra administrative accounts and rootkits that continuously gave them remote log-ins. We couldn't feasibly assess the damage and determine all the changes; I filed an incident report with cost of infinite and put the machine in the evidence locker for forensics to deal with. We got third place too.
Support my political activism on Patreon.
Had a 65yr old woman who's grandkids used the computer... I doubt she ever did. Windows 98 SE, ran Spybot on it and I just about died, over 34,000 items marked as spyware. So I closed the app and ran a virus sweep with AVG and found over 2000 trojans (only like 11 different viruses with variants but multiple installations).
... 300 items marked as spyware from the restore disk, and 3 viruses on the restore disk.
I realized at that point that it wasnt worth cleaning it up, so I reinstalled with her manufacturers restore disk and rescanned it
I did the old woman a favor and installed my old unused retail copy of Win98 on the box.
Thats why you should never buy a computer from Rent-A-Center... *shudder*
I consider myself a computer-saavy Linux and Windows systems administrator.
But, I must ask, how on earth do you guys perform these kinds of clean-ups?
Most spyware that I have seen in the last months are rootkits. They hide underneath the kernel, are impossible to delete and "reinject" themselves upon reboot. I've even seen spyware which injects malicious code and/or replaces the main Windows binaries (explorer.exe, taskmgr.exe, cmd.exe, notepad.exe, etc.) How would you deal with these buggers?
When I come to a spywared computer, I start by running Spybot, AdAware and then AVG AntiVirus (to check for viruses/trojans). I would say that this technique is successful about 50% of the time. If it's not, I consider the situation disastrous and ask the person to do backups and go for a reformat.
I've even touched computer which froze upon startup (Windows boots up and everything freezes up). What would you do in these cases? I boot a livecd to do backups of a drive before the reformat.
So once again, Slashdotters, how do you guys get rid of these nasty rootkit and evolved spywares which can hide very well without reformatting?
The hip way to get your IP. No ads, ever.
I 'inherited' a SPARCserver running SunOS 4.1. Yeah, you can secure SunOS 4.1 (kinda). But the guy who was in charge of the UNIX machines for the past few years, hadn't. This was in 1996 or so and commercial ISPs were relatively new and nobody had really ever considered security.
When I took over the machine I started lobbying the boss to let me do some security work on it and he'd never let me do it. We gave used FULL SHELL ACCESS. Compilers included. Oh and SunOS didn't even have shadow passwords by default!
Anyway, a few months into that someone changed the MOTD to some racist statement. That's when the boss finally let me do stuff.
But he wouldn't let me reinstall the thing. OR take shell-access away.
It was a constant battle. Every day I'd show up and look for what they did TODAY, and fix it. just try to stay ahead of them, and they tried to stay ahead of me...
Sometimes I'd stay up at night and ttysnoop on them talking to their other friends on IRC. Then I'd sigsev their IRC client, and watch them log back on and complain about how the sysadmin can't even keep IRC from segfaulting randomly. Then I'd take over their terminal and start saying crap about the other people he was talking to, until his friends kickbanned the hell out of him. Haha.
I eventually managed to let the boss allow me to replace the shell with a restricted shell (ok, a shell replacement I wrote in perl - it was easier than reading the manpage for rksh).
So basically the point was to make it not worth their while to break into my server.
Eventually this kid started DOSing us. We had a small 64K line to the 'real' internet, and he was on a DS3 in some university in Sweden. Our uplink (UUnet) said they couldn't do anything. Yay. So one day my boss (not the big boss) goes "hey, didn't you say they brag about this stuff on IRC?" I said "Yeah" and he goes "Teach me how to use IRC!!!"
The guy figured out IRC, found some 'hacker' channels, and FOUND THE GUY who was bragging about DOSing us. Started talking to him, getting kinda friendly. Guy starts blackmailing us - said that unless we give him a machine with his own harddrive (he demanded at least 4 gigs) or he'd DOS us again. So we gave it to him to see what he'd do. he filled it up with warez (gah) fairly fast, and then had to download it all with a 28.8K modem...
so my boss goes "Hey...why don't you come in and bring a harddrive and we'll copy it for you?"
And the guy did it. He came into our office. Where I had an IndyCam setup for him. And where we had a PI waiting outside to follow him home. And of course he brought his harddrive which we copied everything off, including his master host/password list.
The kid was 15, so we couldn't sue him or anything. But we did get a LOT of info about him. My boss basically went through all the guy's hosts and nuked them or, if they seemed legit, changed his passwords and Emailed the admins. And some of these were machines belonging to some pretty big cracker/hacker/whatever rings. We nuked those, too.
I like to think that was a pretty good security clean-up. We got rid of a LOT of bad-guy hangouts at that point.
Oh, and I was no longer with that company, but when that kid turned 18, they got him thrown in jail. That was fun, too.
In the land of the blind, the one-eyed man is kinky.
This is on-topic, but not the answer everyone else is giving...
... surprise, he got a virus using IM, which spammed everyone in his address book. I notified everyone in his address book, cleaned up a few infections, and have never had a problem since.
My last encounter with a virus was when my brother (who had been abroad) came home, and a few days later I got an email from him with an executable in it. I downloaded the executable and found
Seriously. I didn't even have the free version of Ad-Aware installed until late 2004, and when I ran it I had lots of tracking cookies... that was all.
I do heavy development in Visual Studio, but only for consulting work. The rest I do in linux. I've never had a problem. I admin lots of systems, and I've seen rootkits on Solaris, but I've been lucky so far with all the linux servers I've looked at.
It's possible some of my mistakes weren't discovered until much later and no one bothered to tell me. But my own workstation has never been exploited. Sorry, hate to disappoint everyone, but I have nothing to tell.
Flash back to around five years ago.
I was a junior admin at my current job and at the time, we ran Exchange 5.5 on WinNT4.
One day, the Exchange server stopped responding. Our senior network admin was not in - in fact nobody was there that day, except for little old me - so I meandered into the server room to check it out.
Now, Windows NT4, while it had the potential to be fairly stable, was not exactly known for it's rock solid reliability, so I wasn't to alarmed when the server stopped responding. I logged onto the machine, and checked the services. Some of them were stopped. I tried to start them and got some cryptic error message. I also noticed that launching other executables, like notepad gave similar cryptic errors.
I did what every semi-incompetent Windows admin would do in that situation; I rebooted the server. The server came up, and I got the dreaded "One or more services failed to start up..." message. I logged on and noticed that the same exchange server services that were not started before the reboot were still not running.
Not good.
So I tried to launch a few other programs and some of them failed too. BY this time, I was suspecting a virus. The server was rather sluggish for having no major services running and the task manager has lots of weird things jumping around in the process list.
I was able to open up the local virus scan app and start a scan and soon I got my answer. Klez.
A hour or so of research and dozens of reboots later, the server was finally free of the Klez virus. Unfortunately due to the fact that Klez was a file infector and the cleaning process didn't always leave infected executables in a usable state, Exchange, and many parts of Windows were still very broken.
Oh. Did I mention this was our first in site Exchange server...and our PDC?
In order to try and get Windows back to working order, I reinstalled Windows NT service pack 4. To my delight, this actually fixed Windows! I was ecstatic. So the next order of business was to get Exchange back up. I tried installing the latest Exchange service pack, but that didn't work. I was not an Exchange expert by any means, so I wasn't quite sure WTF to do at this point. I could just say fuck it, and wait until the next morning for the senior network admin to come in, or stick with it. I decided to do something that I was sure would hose the system - stick the Exchange CD in and reinstall Exchange over the broken copy. Since the system was already hosed, I figured I couldn't make it any worse. I figured this would wipe out any custom settings, so I made backups (and backups of those backups) of all of the Exchange information stores before starting.
To my delight, reinstalling Exchange, and the service packs actually worked! The Exchange system was back up!
It was now about ten O'Clock and I had triumphantly recovered the company jewels. But I was not done.
Somehow a few of the other servers had also become infected with the Virus. Cleaning these up was a but easier, and the virus never actually got executed on those machines. I spent another hour or so, scanning and cleaning the other servers that had infected files.
It was about midnight by the time I was done.
Now, you might be wondering. How the heck did this ever happen? Klez was primarily an email virus that relied on social engineering or extremely weak share permissions to spread.
Here's how:
Our senior network admin had an "interesting" way of administering exchange accounts. He would install the entire Microsoft Office Suite on the Exchange server, and after creating a new user account, he would log onto the Exchange server as his domain admin account, and set up the account in Outlook to "test it". If you have half a brain cell, you can see now how the Exchange server got infected.
As for the other servers that got infected, our senior network admin just LOVED to have network drives mapped at all times (just in case?). He had THE logon script from hell, and Klez, also having the ability to spread via file shares, infected every server he was mapped to when he logged onto the Exchange server.
That's my story.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
A bunch of my computers once got infected with Chernobyl, and it proceeded to trash the BIOS on two or three machines. I was pretty pissed about potentially having to replace these motherboards, so I said screw that and got an EPROM writer. With the latest version of the BIOS from the manufacturer, I flashed me up a few EPROMS and plugged 'em in. Suckers booted right up, and since the only way to erase them was with UV light, they were completely immune to BIOS attacking viruses thereafter.
I get on the horn the folks in the IT department. "Yo, d00dz, we finally got pwn3d."
"Not our problem."
"No, really. The reason the box is so slow is because we've run an open relay for (censored) months, and this dude from a (censored).aol.com dialup has finally decided to exploit it. Shut him down".
So they do.
An hour or two later, the guy dials back in to his .aol.com dialup IP, and pwns us again.
So I get on the horn again.
And when the same dude I reported the problem to not six hours earlier comes to my own cube, and I show him the output of "ps" with his own two eyes, he denies that there's been any compromise.
So I escalate to my manager who's not there.
And in the absence of my manager, to her manager.
Who asks me what kind of spam is going out. And I reply, having seen the megabytes of world-readable spam in the output queue, that it appears that "Hot Vegas Sluts Want To Suck Your Cock", Sir, and that anyone reading *clickity-click* these headers will be able to determine that the spam's coming from our netblock, he sorta went blank.
I pointed out (actually, I lied, because our sysadmin had seen the evidence with his own two eyes) that our sysadmin he'd left for the day and had no way to know that the system was pwn3d. And suggested that it was therefore up to the PHB-type that in the absence of sysadmin expertise, that it was up to him to make the call as to whether to page our sysadmin.
Couple of hours after having left the office for the night, the spam stopped and the open relay got closed.
Copule of hours after coming back in the next day, the guy who owned the root password suggested, by means of forged email to the entire company, that someone in the company oughta either do his own job or find another.
(Props to the admin for being discreet about it. Seriously.)
But since the #0 commandment of any user of a system is to "never piss off the dude with root", I realized my days were numbered. Never mind that I'd actually lied to cover up my sysadmin's incompetence, and in doing so, saved his ass , but since my sysadmin (obviously, since he didn't need to know I'd lied to cover his ass :) wasn't in a position to see it that way, I was back to following rule #0.
I never had the heart to tell him how hard I covered for him. He probably still thinks I ratted him out.
Wait a minute. The worst security cleanup I've performed? That was the best security cleanup I've performed.
Because I took his advice. I quit the company a few months later, and am now around a million bucks richer, a good chunk of which came out in the form of stock options that I'd never have been issued had I not left the company and landed at a startup that made good.
So - if my former BOFH ever reads this - thanks. If I'd just kept my head down and done my own job, I'd still be working at the same place you stayed. But because I took your advice and stopped doing my job in order to "get another", I'm not only happy - but able - to buy you, and all your staff, as many beers as you like whenever I swing back into town.
The funny part of the story is all the admin would have had to do was wink at me while pulling one plug outa the box and lie to his boss, claiming whatever he wanted for the 20 minutes of downtime it woulda taken him to compile a current version of Sendmail, and I woulda backed him up on it. "Yeah, I saw the mail server go down around the same time. Mail was down for a bit, but it's a good thing XXXXX was on the job. He saw it before I did."
But it didn't work out
I don't clean up virused windows machines. I consider them to be pre-virused from the start. Anyway, they can only infect other windows machines, so what's the harm ? I use them until they get too slow to use and then re-install, when I use them.
/whatever/file" on each bad file. It took hours in spite of perl scripting a lot of it.
I've delt with some nasty cases on linux though. Be forewarned, a lot of the twitchy sys admin types who believe in the "proper" way of doing things are going to be driven crazy by what follows.
Story 1: A visitor to my house needed to use ftp (ftp something TO me, for obscure reasons I have forgotten), and I temporarily turned on the ftp server on my Redhat 6.1 box on my cable modem. Later I noticed the machine running slow and a stuck process with a disguised name; grepping strings on the executable showed it to be an IRC server with built in commands that would DDOS people. Examination of logs showed I was cracked within three hours of turning that ftp server on. I was running tripwire, so I had a daily email showing what files had changed, but I had not been updating tripwire much, so I had to dig through lengthly lists to find out what new files had arrived and remove them. The computer that hacked mine was another RH 6.1 on a DSL in California, that was serving up web pages of pictures of salvage autos from a junkyard, all in spanish. I did not bother to contact them.
Story 2: About three years later, when RH 6.1 was pretty old, I was working for a guy who had a few remote RH 6.1 servers at his customer's sites around the country. They never connected to the internet, we dialed into them on the modem, thus no security worries, right ? Well, we had to make them dial out to an ISP and email us the IP address, because they changed their phone system and we temporarily couldn't dial into the remote machine, and that got cracked within a few hours. Examination of a few clues, which I have forgotten, lead me to conclude it had an Aurora root kit on it, which is a kernel module that the kernel reads in on bootup, that then filters all your ls and lsof and other commands to stop you from finding it or removing it. The solution I came up with was to go to an identical machine and compile an identical kernel, except with all modules built in and the ability to load modules turned off. The decision was made to make them mail us the harddrive back and we mailed them a replacement before I got to try it.
Story 3: a Debian server a different, later employer used was the NATing gateway, mail server, file server, essentially everything for a very small office. The boss-man either connected to it from an invested public terminal at a university, or it was brute-force ssh'd, not sure. It was compromised, and not noticed for months because the guy never did anything (this was confirmed by going back through backups and checking for when the key files appeared). I noticed it when I discovered I couldn't update something because someone had used chattr to make the file immutable, and of course that file was a trojan (it took me a while to figure that out). I booted up with a live CD to make sure no aurora type root kit was intefering with my access, and searched the entire disk for every immutable file (using lsattr and grep), and then hand-replaced the binaries used by apt-get and dpkg and friends, and then chrooted to the disk and did "apt-get --reinstall install packagename" for every compromised binary. I got the package name from "dpkg -S
I discovered a "hidden" directory (named with a single space character) that had tools to make random searches on yahoo and scrape the resulting pages for email addresses, and the spam had links to a fake bank login page, and the stuff to host that page was also there. As far as I could tell it was never unpacked and run. It was in a tar.gz with a script to unpack it and set it all up automatically.
He was running a package of two or three cobbled together sniffers and a compromised ssh
The headmasters wife of the school where my wife works gave me her laptop to look at whilst we were at a party at their place once. The schools IT guy wouldnt touch it. It was windows XP but it took something like 10 minutes to boot and she said it was "reeeally reeeally slew" (she is French).
Found out the disk had 5k of space left on it. Checked and there was no antivirus, firewall or antimalware installed and it had been directly connected to a broadband line with a adsl modem for the last 3 months. And the cursors were animated dinosaurs.
Once I had cleared off some space I installed AVG and Ad-Aware. They both went through the roof. One of the many many viruses was inflating every file on the drive that was around 150k to around 300k which partially explained the lack of disk space. Eventually I couldnt do any more and it was still crap. I suggested wiping it. "Oh you cant do that... I dont keep any backups and the Outlook Express has all the details of our side business in it"
I ended up passing the mess onto my brother who has a nice sideline. He actually said it was the second worse pc he has ever sorted out. The worst was a guy who downloaded from Kazaa constantly as well. After 3 days he fixed it though. He ended up using 3 different virus scanners to get everything.
When I gave it back to her I explained that someone was probably using her laptop to send out loads of spam and host kiddie porn on. She went out and bought Norton that very day. Lets hope she keeps it up to date.
another Roadkill on the Information Superhighway
Well, once, a little more than a year ago, I paid a visit to some friends and the afternoon progressed as usual, I eventually found myself in front of their computer. Because they had some trouble with their broadband access, it seemed.
As I soon found out the broadband company had cut them off, since the computer was a breeding ground for virus and spam of all sorts. Why did they have so much problem, you ask? This is what I found.
No hardware firewall, one computer directly accessing the internet on a (albeit slow) broadband connection, no software firewall, no anti-virus program, no ad ware-removal program, outlook express and (actually!) a really old version of Firefox (0.3 I believe), all of it running on an unpatched version of Windows 98A.
It took me some time to clean that one out.
But it did impress me somewhat that the broadband company (Telia, Sweden) actually demanded proof that they had installed both anti virus and a firewall before they reactivated the connection. That is surprisingly good ethics for such a company, although it might be considered pure survival tactics, as the internet climate are today.
I've learned all I know about politics from
I actually had a favorite mail trojan at one point. I can't remember what it was called, and it expired itself a couple of years ago. It was distributed via mail, picking out everone in their address book. The fun thing about it was that it would pick out a random file from the victims computer, preferrably some sort of document, but it didn't seem to fussy, attach a copy of itself to the beginning of the file and send it on. Made a quick script which chopped off the virus whenever I received a mail, and then saved the actual file somewhere so I could take a look. It was like a little surprise in the mailbox every day. Some of my favorite ones were:
.pst file from their MS outlook. Lots of mail, nothing interesting, but the program sent the file without the user noticing it.
* An excel spreadsheet showing the expenses for a french shoe manufacturer
* Someone's thesis on the spawning habits of canadian salmon (quite well written too, best of luck with the masters)
* A strange photograph of a person driving a car with a giant carrot for a passenger
* Someones 10Mb
* No porn whatsover, dissapointing
* An no password files, which I guess would have been a good primary target for the trojan.
Quality trojan, they don't write them like that anymore.
Task Mangler
Had these folks not too long ago that were getting phone calls and actual snail mail from their ISP telling them to take their computer off line and have it repaired. The ISP actualy did cut them off, because their machine was saturating the line all the time as a spambot and as a server for other bot infections.
The machine was about a year old (and out of warranty, of course) - a 2.6 gig cpu with a gig of ram. It took almost 35 minutes to go from power off to the desktop. They had an antivirus that came with the machine, but the "free 90 day subscription" to it had run out long and they weren't aware of it, since that was one of the first things the malware went after. Their 16 year old son who loved to surf porn all the time didn't help matters. A machine like that really isn't worth the time to hunt and peck for individual pieces of malware and should wiped clean and started fresh, however the godawful shit that was on it even hosed the recovery partition. And since actual install media isn't included with a $MAJORMANUFACTURER machine, they would have had to shell out for a retail copy of their previous OS.
Since these folks were obviously pretty cluless about computers, I fired them up a knoppix CD to see how they took to it. They honestly had zero problems navigating the KDE desktop and were able to do everything they wanted with the computer, except obviously to save stuff.
They now have a shiny Debian Etch based KDE desktop that they're enjoying, virus, malware, and calls from the ISP free.
That was one of the worst I've ever seen.
...Rob
The American Dream isn't an SUV and a house in the suburbs; it's Don't Tread On Me.
I once helped out a lady with Win98 who called me after she received an $800 long-distance phone bill. She wa a dial-up ISP user and caught SOMETHING that was dialing Sweden in the middle of the night to do God-only-knows what.
Not the "worst" infection I've ever cleaned up, but certainly the weirdest!
signat-url: http://www2.potsdam.edu/dctm/prescor/signat-url.h