Slashdot Mirror
Time Bomb May Have Destroyed 800 Norfolk City PCs' Data
krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"
It's Naw-Fuck.
And it's nowhere near as embarrassing as how we pronounce Buena Vista.
Is it just my observation, or are there way too many stupid people in the world?
I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches...
Hardly. It's just something that messed with the Win32 folder. This could be fixed by a few temps over the weekend if the city government was half-competent.
I am the richest astronaut ever to win the superbowl.
if they were running backups, they wouldn't be scratching their heads and behaving completely ignorant of what exactly it was or when it was put in. They obviously lost everything, which I'm sorry but I find some darwinism/justice in that. If you don't even have a backup to look at to see what it was sitting on the hard drive waiting to blow up, you're just beyond help. Maybe better luck next time.
But too many out there simply must learn their lessons the hard way. That will never change.
I work for the Department of Redundancy Department.
At first glance that blows my mind. That's absolutely huge. Then I check my linux box and /usr/lib64 is 1.7 GB.
As to why they couldn't just boot to linux or a recovery CD and salvage the data....
"Eve of Destruction", it's not just for old hippies anymore...
So the data is wiped because the System32 folder is fucked up? Uh-huh... guess they have to throw out all those computers and order new ones. Looks like the data's gone forever.
mmmm...forbidden donut
We've instituted offsite backups, both over the tubes and physically taking images of our servers (all virtualized of course) offsite to a bank safety deposit box. If, for whatever reason, the whole damned building explodes tomorrow, we've got the data sitting on servers in two other geographically distant locations. But if we can't get to those, we have the VM images, so as long as we can get our hands on a server capable of running Linux KVM, we could be up and running in short order (I estimate 3-4 hours, including host OS installation).
The days when a physical or digital attack can fuck the whole organization are gone. There are enough traditional and newer backup schemes out there that even long downtimes aren't necessary.
The world's burning. Moped Jesus spotted on I50. Details at 11.
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.
Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.
You got it. it's also a great example of how incompetent most City's IT staff are, Hey municipalities... you get what you pay for. How's those $25,000 a year IT staff working out for ya?
Do not look at laser with remaining good eye.
You cant take any details from any news articles at face value.
Deleted
Twenty bucks says that they never figure out what happened.
From working in the backup industry for years, I'm sure they have backups, the problem is that they never tried to verify or restore them. but is there really isn't any data there, compression is great when you just "tar cv * > /dev/null" ...
Heck one time I had a guy who was getting Parity Errors decide that the best way to solve them was to just shut off Parity Checking... Ignorance is bliss I suppose.
Seriously I can't count the number of times I tried to help someone restore their backups after a critical loss that turned out to never have actually verified that they worked in the first place. Just as bad as when I worked in a photo shop and someone said they couldn't get their film out... put the camera in the light locked compartment, stuck my hands in, just to find that he had taken 36 'priceless vacation pictures' on the back of the camera body instead of film.
-=JML=-
If lil' ol' me can spend a few hundred dollars on enough hard drives stuffed into external enclosures the have two complete backups of all ~1.5TB of data in my system, surely a municipal government can spend a few thousand dollars to do it too.
What the hell, who runs systems that important without backups? Management teams named Shirley?
Living With a Nerd
When you sort the bits first compressed backups are really small.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Even if you're a complete dolt and don't lose all of that, you can still recover data with some sophisticated technology. The hard drive might claim its empty but the bits are likely still in their last position. (Ever noticed how clearing the partitions off of your hard drive is instantaneous?)
This is why professionals can still recover a large chunk of data from a hard drive even if you used a drillbit to punch a hole in it. .
* Check every few seconds to see if network goes down
* Write a bogus entry in the log files that points to some oddball behavior, like a disk-read error or something
* If network is down freeze screen so it looks like computer just locked up
* Ignore all input
* Wipe key parts of disk so forensic recovery is impossible or at least very difficult
* Wipe key parts of memory so forensic recovery is impossible or at least very difficult
* Wipe key parts of cache so forensic recovery is impossible or at least very difficult
* Force or fake a BSOD screen so a casual user will think his computer crashed and blame any resulting data loss on the crash
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Um, they're called 'roaming profiles' and have been around for some time. You can store users' profiles anywhere you want...different drive, or even a remote server.
I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
Re-worked summary of TFA:
- All that has been damaged is the System32 folder of user machines.
- 'Destroyed' I imagine is an IT staff trying to dumb down his language to his perception of the level of the reporter's IT knowledge
- Their IT may have done quite well, the only 'damage' is to PCs that were shut down in the 1 hour window between the attack starting and IT containing it
- Employees were supposed to save to the network. The only issue stated is that some staff were breaking the rules and saved things to their own PC.
All they need to do with the affected machines is to boot from a Windows or Linux CD, copy the files to memory stick and throw their standard "new install" image on. No data loss. No network down time. All they're looking at is some hassle for the ~ 18% of users affected and a very busy IT department. Provided the affected users have other machines to work on (or however not losing much productivity) they're not far off having the best scenario any It department can realistically hope for (well, I'd like to say it's reasonable to hope for not having pissed off employees). Sure, no doubt a dozen IT managers can post their "perfect" system, and another dozen IT managers can show how they could destroy it.
But whoever hated them enough to install the timebomb would obviously have sabotaged the backups. Maybe that was what the delay was all about.
http://michaelsmith.id.au
It's not, except for the insane or people who aren't able or willing to use a reasonable imaging and app distribution system.
It appears that people who didn't RTFA or who work at tiny tiny sites are criticizing these guys without knowing what the hell they're talking about.
No one does workstation backups because it's costly, risky, inefficient, and generally doesn't work. The only way to make it work is to say "put all the documents you need to backup here" and here is better off being a network drive anyway.
VMware Data Recovery is a piece of shit that rarely works the way you want it. Try reading the forums sometime to see how much grief it gives others.