Slashdot Mirror
Worst Security Clean-Up You've Performed?
nakhla writes "Last night, I was tasked (by my wife) to help fix her friend's computer. It is a Windows XP home system which has been running slowly, almost to the point of un-usability (like *that's* never happened before). It turns out that hundreds of random processes had filled up its meager 256 MB of RAM. The cause? Nearly 7,500 viruses and worms that had infected the system. That number doesn't even include the hundreds of spyware and adware programs that had installed themselves, as well. Although the box is now behind a firewall, that wasn't always the case. This was, by far, the most infected system I'd ever seen, but I'm sure it can't be the worst ever. What was the worst security cleanup you ever had to perform?"
Once, I saw a computer infected with Windows ME.
Demented But Determined.
With that many viruses, is it even possible to "clean" it?
Hell, i do a reinstall if I get even 1 bad virus..
MABASPLOOM!
Geek Squad. One customer had 35,000 pieces of spyware and over 3000 instances of some 30 or 40 viruses on her computer, some of which required some alternative methods to remove since they were locked when in safe mode and encrypted so you couldn't scan with a boot CD. After 4 scans taking about 6 hours I managed to get the spyware gone, and also inbetween had made note of viruses I needed to manually purge. Cleaned it up nice; meanwhile my supervisor was telling me to call the customer and tell them we needed to just reinstall Windows.
My aunt got AOL with anti-spyware and firewall and security. Eventually she had 35 different viruses, managed to remove all but 28 unique signatures (this was before I developed my brute-force removal method). Chucked a ton of spyware too.
While at WhiteWolf Security, we had a little game going; eventually our opponents got pissed at us for unrelated reasons and decided to physically break into WhiteWolf at 4am. They shorted CMOS pins and used boot CDs to evade password lock-outs, adding extra administrative accounts and rootkits that continuously gave them remote log-ins. We couldn't feasibly assess the damage and determine all the changes; I filed an incident report with cost of infinite and put the machine in the evidence locker for forensics to deal with. We got third place too.
Support my political activism on Patreon.
Had a 65yr old woman who's grandkids used the computer... I doubt she ever did. Windows 98 SE, ran Spybot on it and I just about died, over 34,000 items marked as spyware. So I closed the app and ran a virus sweep with AVG and found over 2000 trojans (only like 11 different viruses with variants but multiple installations).
... 300 items marked as spyware from the restore disk, and 3 viruses on the restore disk.
I realized at that point that it wasnt worth cleaning it up, so I reinstalled with her manufacturers restore disk and rescanned it
I did the old woman a favor and installed my old unused retail copy of Win98 on the box.
Thats why you should never buy a computer from Rent-A-Center... *shudder*
I used to keep the case off of my computer, to help keep it cool. That is, until a friend crashed in my study after a big night out and somehow managed to throw up inside it. Needless to say I have a whole new setup now.
Buh!
My uncle's computer had a meager 128 megabytes of RAM, running XP, with two teenagers using it.
... just that they are the culprits for at least 2 computers I cleaned so far.
It was a mess a real mess.
5 minutes starting XP, 2 minutes seeing the window of Internet Explorer appear. 10-15 minutes to be able to download Spybot and AVG. 3 hours running spybot (you read me right).
The hard drive stayed constantly ON during all that time. Then I said Screw That, and I reinstalled.
My conclusions after 3 hours:
- The first and biggest threat all the newbie users have on their computer are OUTDATED norton utilities giveaways they got with their machine. They THINK they are protected, but they closed the "renew" window so often they forgot it's there. Either the software is FREE AND CONTINUOUS, or it's not there, capiche? Avg is excellent, there are many other free ones too... just find one and be happy. Not something that's NOT free.
- The second biggest threat are Norton Security centers, again outdated, again with useless popups. Again with people finding it nagging and deactivating it, making certain not only the Windows Firewall is properly deactivated by Norton's presence, but that their system is totally uselessly unprotected. Very great, coming from a security company. Again, there are many FREE (beer) softwares that do spyware detection and stuff, and Windows Firewall, in all its eloquence, is still better than a kick in the butt, at least compared to the useless deactivated softwares I found.
Not that I hate norton, that is
Then, even if you got years of pro experience in computers, people trust only one person, and if it's not you, you're d00med. I have been explaining to them their meager 128 megs of memory was not enough.... to no avail, they wanted to change computers, almost bought a new one, then another member of my family told them the exact same thing I did, now they have 512 megs and it's screaming. "told you so" was the only answer I could say. Oh well.
I consider myself a computer-saavy Linux and Windows systems administrator.
But, I must ask, how on earth do you guys perform these kinds of clean-ups?
Most spyware that I have seen in the last months are rootkits. They hide underneath the kernel, are impossible to delete and "reinject" themselves upon reboot. I've even seen spyware which injects malicious code and/or replaces the main Windows binaries (explorer.exe, taskmgr.exe, cmd.exe, notepad.exe, etc.) How would you deal with these buggers?
When I come to a spywared computer, I start by running Spybot, AdAware and then AVG AntiVirus (to check for viruses/trojans). I would say that this technique is successful about 50% of the time. If it's not, I consider the situation disastrous and ask the person to do backups and go for a reformat.
I've even touched computer which froze upon startup (Windows boots up and everything freezes up). What would you do in these cases? I boot a livecd to do backups of a drive before the reformat.
So once again, Slashdotters, how do you guys get rid of these nasty rootkit and evolved spywares which can hide very well without reformatting?
The hip way to get your IP. No ads, ever.
I 'inherited' a SPARCserver running SunOS 4.1. Yeah, you can secure SunOS 4.1 (kinda). But the guy who was in charge of the UNIX machines for the past few years, hadn't. This was in 1996 or so and commercial ISPs were relatively new and nobody had really ever considered security.
When I took over the machine I started lobbying the boss to let me do some security work on it and he'd never let me do it. We gave used FULL SHELL ACCESS. Compilers included. Oh and SunOS didn't even have shadow passwords by default!
Anyway, a few months into that someone changed the MOTD to some racist statement. That's when the boss finally let me do stuff.
But he wouldn't let me reinstall the thing. OR take shell-access away.
It was a constant battle. Every day I'd show up and look for what they did TODAY, and fix it. just try to stay ahead of them, and they tried to stay ahead of me...
Sometimes I'd stay up at night and ttysnoop on them talking to their other friends on IRC. Then I'd sigsev their IRC client, and watch them log back on and complain about how the sysadmin can't even keep IRC from segfaulting randomly. Then I'd take over their terminal and start saying crap about the other people he was talking to, until his friends kickbanned the hell out of him. Haha.
I eventually managed to let the boss allow me to replace the shell with a restricted shell (ok, a shell replacement I wrote in perl - it was easier than reading the manpage for rksh).
So basically the point was to make it not worth their while to break into my server.
Eventually this kid started DOSing us. We had a small 64K line to the 'real' internet, and he was on a DS3 in some university in Sweden. Our uplink (UUnet) said they couldn't do anything. Yay. So one day my boss (not the big boss) goes "hey, didn't you say they brag about this stuff on IRC?" I said "Yeah" and he goes "Teach me how to use IRC!!!"
The guy figured out IRC, found some 'hacker' channels, and FOUND THE GUY who was bragging about DOSing us. Started talking to him, getting kinda friendly. Guy starts blackmailing us - said that unless we give him a machine with his own harddrive (he demanded at least 4 gigs) or he'd DOS us again. So we gave it to him to see what he'd do. he filled it up with warez (gah) fairly fast, and then had to download it all with a 28.8K modem...
so my boss goes "Hey...why don't you come in and bring a harddrive and we'll copy it for you?"
And the guy did it. He came into our office. Where I had an IndyCam setup for him. And where we had a PI waiting outside to follow him home. And of course he brought his harddrive which we copied everything off, including his master host/password list.
The kid was 15, so we couldn't sue him or anything. But we did get a LOT of info about him. My boss basically went through all the guy's hosts and nuked them or, if they seemed legit, changed his passwords and Emailed the admins. And some of these were machines belonging to some pretty big cracker/hacker/whatever rings. We nuked those, too.
I like to think that was a pretty good security clean-up. We got rid of a LOT of bad-guy hangouts at that point.
Oh, and I was no longer with that company, but when that kid turned 18, they got him thrown in jail. That was fun, too.
In the land of the blind, the one-eyed man is kinky.
I once tried to uninstall Real Player, but I was not successful so I guess it does not count.
Ronald said nothing. He flung himself from the room, flung himself upon his horse, and rode madly off in all directions.
Flash back to around five years ago.
I was a junior admin at my current job and at the time, we ran Exchange 5.5 on WinNT4.
One day, the Exchange server stopped responding. Our senior network admin was not in - in fact nobody was there that day, except for little old me - so I meandered into the server room to check it out.
Now, Windows NT4, while it had the potential to be fairly stable, was not exactly known for it's rock solid reliability, so I wasn't to alarmed when the server stopped responding. I logged onto the machine, and checked the services. Some of them were stopped. I tried to start them and got some cryptic error message. I also noticed that launching other executables, like notepad gave similar cryptic errors.
I did what every semi-incompetent Windows admin would do in that situation; I rebooted the server. The server came up, and I got the dreaded "One or more services failed to start up..." message. I logged on and noticed that the same exchange server services that were not started before the reboot were still not running.
Not good.
So I tried to launch a few other programs and some of them failed too. BY this time, I was suspecting a virus. The server was rather sluggish for having no major services running and the task manager has lots of weird things jumping around in the process list.
I was able to open up the local virus scan app and start a scan and soon I got my answer. Klez.
A hour or so of research and dozens of reboots later, the server was finally free of the Klez virus. Unfortunately due to the fact that Klez was a file infector and the cleaning process didn't always leave infected executables in a usable state, Exchange, and many parts of Windows were still very broken.
Oh. Did I mention this was our first in site Exchange server...and our PDC?
In order to try and get Windows back to working order, I reinstalled Windows NT service pack 4. To my delight, this actually fixed Windows! I was ecstatic. So the next order of business was to get Exchange back up. I tried installing the latest Exchange service pack, but that didn't work. I was not an Exchange expert by any means, so I wasn't quite sure WTF to do at this point. I could just say fuck it, and wait until the next morning for the senior network admin to come in, or stick with it. I decided to do something that I was sure would hose the system - stick the Exchange CD in and reinstall Exchange over the broken copy. Since the system was already hosed, I figured I couldn't make it any worse. I figured this would wipe out any custom settings, so I made backups (and backups of those backups) of all of the Exchange information stores before starting.
To my delight, reinstalling Exchange, and the service packs actually worked! The Exchange system was back up!
It was now about ten O'Clock and I had triumphantly recovered the company jewels. But I was not done.
Somehow a few of the other servers had also become infected with the Virus. Cleaning these up was a but easier, and the virus never actually got executed on those machines. I spent another hour or so, scanning and cleaning the other servers that had infected files.
It was about midnight by the time I was done.
Now, you might be wondering. How the heck did this ever happen? Klez was primarily an email virus that relied on social engineering or extremely weak share permissions to spread.
Here's how:
Our senior network admin had an "interesting" way of administering exchange accounts. He would install the entire Microsoft Office Suite on the Exchange server, and after creating a new user account, he would log onto the Exchange server as his domain admin account, and set up the account in Outlook to "test it". If you have half a brain cell, you can see now how the Exchange server got infected.
As for the other servers that got infected, our senior network admin just LOVED to have network drives mapped at all times (just in case?). He had THE logon script from hell, and Klez, also having the ability to spread via file shares, infected every server he was mapped to when he logged onto the Exchange server.
That's my story.
I don't always use unix-like operating systems; but when I do, I prefer FreeBSD.
This is from 2005! Her computer was a PII 75 running Windows 95. The basic problem is that it had been overrun by viruses. A one hour fix if I had taken out her hard drive, plunked it into my repair PC and done a virus scan... but she refused to allow her machine out of her house for fear that I would steal it. Rather than entrust her $50 PC to me, she instead payed me $280 in house-call fees while I sat there for 8 hours with my arms crossed, watching AVG do it's stuff.
I don't clean up virused windows machines. I consider them to be pre-virused from the start. Anyway, they can only infect other windows machines, so what's the harm ? I use them until they get too slow to use and then re-install, when I use them.
/whatever/file" on each bad file. It took hours in spite of perl scripting a lot of it.
I've delt with some nasty cases on linux though. Be forewarned, a lot of the twitchy sys admin types who believe in the "proper" way of doing things are going to be driven crazy by what follows.
Story 1: A visitor to my house needed to use ftp (ftp something TO me, for obscure reasons I have forgotten), and I temporarily turned on the ftp server on my Redhat 6.1 box on my cable modem. Later I noticed the machine running slow and a stuck process with a disguised name; grepping strings on the executable showed it to be an IRC server with built in commands that would DDOS people. Examination of logs showed I was cracked within three hours of turning that ftp server on. I was running tripwire, so I had a daily email showing what files had changed, but I had not been updating tripwire much, so I had to dig through lengthly lists to find out what new files had arrived and remove them. The computer that hacked mine was another RH 6.1 on a DSL in California, that was serving up web pages of pictures of salvage autos from a junkyard, all in spanish. I did not bother to contact them.
Story 2: About three years later, when RH 6.1 was pretty old, I was working for a guy who had a few remote RH 6.1 servers at his customer's sites around the country. They never connected to the internet, we dialed into them on the modem, thus no security worries, right ? Well, we had to make them dial out to an ISP and email us the IP address, because they changed their phone system and we temporarily couldn't dial into the remote machine, and that got cracked within a few hours. Examination of a few clues, which I have forgotten, lead me to conclude it had an Aurora root kit on it, which is a kernel module that the kernel reads in on bootup, that then filters all your ls and lsof and other commands to stop you from finding it or removing it. The solution I came up with was to go to an identical machine and compile an identical kernel, except with all modules built in and the ability to load modules turned off. The decision was made to make them mail us the harddrive back and we mailed them a replacement before I got to try it.
Story 3: a Debian server a different, later employer used was the NATing gateway, mail server, file server, essentially everything for a very small office. The boss-man either connected to it from an invested public terminal at a university, or it was brute-force ssh'd, not sure. It was compromised, and not noticed for months because the guy never did anything (this was confirmed by going back through backups and checking for when the key files appeared). I noticed it when I discovered I couldn't update something because someone had used chattr to make the file immutable, and of course that file was a trojan (it took me a while to figure that out). I booted up with a live CD to make sure no aurora type root kit was intefering with my access, and searched the entire disk for every immutable file (using lsattr and grep), and then hand-replaced the binaries used by apt-get and dpkg and friends, and then chrooted to the disk and did "apt-get --reinstall install packagename" for every compromised binary. I got the package name from "dpkg -S
I discovered a "hidden" directory (named with a single space character) that had tools to make random searches on yahoo and scrape the resulting pages for email addresses, and the spam had links to a fake bank login page, and the stuff to host that page was also there. As far as I could tell it was never unpacked and run. It was in a tar.gz with a script to unpack it and set it all up automatically.
He was running a package of two or three cobbled together sniffers and a compromised ssh
Congress. Got that bitch all cleaned up. Sure took a while, though. You wouldn't _believe_ the shit that was going on in there!
Well, once, a little more than a year ago, I paid a visit to some friends and the afternoon progressed as usual, I eventually found myself in front of their computer. Because they had some trouble with their broadband access, it seemed.
As I soon found out the broadband company had cut them off, since the computer was a breeding ground for virus and spam of all sorts. Why did they have so much problem, you ask? This is what I found.
No hardware firewall, one computer directly accessing the internet on a (albeit slow) broadband connection, no software firewall, no anti-virus program, no ad ware-removal program, outlook express and (actually!) a really old version of Firefox (0.3 I believe), all of it running on an unpatched version of Windows 98A.
It took me some time to clean that one out.
But it did impress me somewhat that the broadband company (Telia, Sweden) actually demanded proof that they had installed both anti virus and a firewall before they reactivated the connection. That is surprisingly good ethics for such a company, although it might be considered pure survival tactics, as the internet climate are today.
I've learned all I know about politics from
At the university I once had the job to produce 100 copies of a circa 100 page application document for a very important government funded research project.
I had a high-performance copier, to which I fed the original pages, cranked the lever to 100 copies and kept shoveling paper into it until it finished.
Only then I realized that I misunderstood the sort/collate switch and ended up with 10,000 sorted pages, meaning that 100 pages #1 were followed by 100 pages #2 etc.
I was out of fresh paper for a retry, too.
After some decent swearing and a couple of cigarettes, I arranged the tables of a seminar room around myself, then spent the whole night making 100 stacks of paper one by one.
When it was over, the skin on my fingers was so dry that it cracked and started bleeding. Not to speak of the over-exercised muscles in my hands...
Say out loud: I'm an Aspie and I'm somewhat proud, I guess. Uh. Can I write an email in all caps instead? Hm...
About twenty years ago an exterminator was spraying my apartment complex and asked if I had seen any bugs. I replied, "Only in the computer." Sadly, he actually sprayed inside the computer and killed it. I've since learned to curb the computer humor with non-technical people.
I actually had a favorite mail trojan at one point. I can't remember what it was called, and it expired itself a couple of years ago. It was distributed via mail, picking out everone in their address book. The fun thing about it was that it would pick out a random file from the victims computer, preferrably some sort of document, but it didn't seem to fussy, attach a copy of itself to the beginning of the file and send it on. Made a quick script which chopped off the virus whenever I received a mail, and then saved the actual file somewhere so I could take a look. It was like a little surprise in the mailbox every day. Some of my favorite ones were:
.pst file from their MS outlook. Lots of mail, nothing interesting, but the program sent the file without the user noticing it.
* An excel spreadsheet showing the expenses for a french shoe manufacturer
* Someone's thesis on the spawning habits of canadian salmon (quite well written too, best of luck with the masters)
* A strange photograph of a person driving a car with a giant carrot for a passenger
* Someones 10Mb
* No porn whatsover, dissapointing
* An no password files, which I guess would have been a good primary target for the trojan.
Quality trojan, they don't write them like that anymore.
Task Mangler