Slashdot Mirror
Time Bomb May Have Destroyed 800 Norfolk City PCs' Data
krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"
It's Naw-Fuck.
And it's nowhere near as embarrassing as how we pronounce Buena Vista.
Is it just my observation, or are there way too many stupid people in the world?
for making bomb threats.
Whatever it was, it essentially destroyed these machines.
Unless this time bomb was something from the 90's like Win32.CIH and nuked the bios, I doubt that the computers are "essentially destroyed"
You guys have backups, right?
... this is the internet... Isn't the apostrophe in the title supposed to be further to the left? :|
I had to read it twice to confirm it was used correctly.
I live in VA Beach, which is the next city down the road (I live a few blocks from exist 20 264, and down-town Norfolk is exit 13ish), and I work in a security-related position, so we tend to keep up on news like this, but this is the first I'm hearing of it, though it looks to have gone down last week (apparently the boot.ini files were modified between 16:30 and 17:30 on 9 February, and only the computers which rebooted during that time period were affected).
It doesn't sound like the attack was particularly complex or anything, so maybe that's why it isn't exactly "newsworthy" (I also don't watch local TV news, so I don't know if they mentioned it), but still, sucks for them. I hope they have good backup policies.
I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches...
... someone has a case of the 2 A.M. Worm.
"How the hell did it get in here?"
> We don't know how it got into our system... We speculate...
As long as we're speculating, may I nominate last week's "Operation Cyber Storm" (http://www.dhs.gov/xnews/releases/press_release_0853.shtm).
At first glance that blows my mind. That's absolutely huge. Then I check my linux box and /usr/lib64 is 1.7 GB.
As to why they couldn't just boot to linux or a recovery CD and salvage the data....
"Eve of Destruction", it's not just for old hippies anymore...
So the data is wiped because the System32 folder is fucked up? Uh-huh... guess they have to throw out all those computers and order new ones. Looks like the data's gone forever.
mmmm...forbidden donut
Could be...
I think Time Bomb is the best of all the No-Heroics superheros. http://www.youtube.com/watch?v=JLaXUTdybjc ....
Oh wait, you were talking about that
Flexible bare-metal recovery for Linux/UNIX
So you can just restore from backup right?
Right?
Please don't look at me that way! You do have backups, don't you?
I myself have a RAID 5 and two large external hard drives. Once a week I swap the external on my desk with the one I keep in a bank safe deposit.
But it took the loss of the third drive of my career before I got Religion myself.
IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.
Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.
I have seen time bombs left behind by two types of people when being called in as a consultant to deal with the aftermath:
1: The disgruntled employee. He leaves a hidden file that if not touched in 2-3 weeks will start wreaking havoc. I've even seen modified binaries of tar and such that encrypt the files, so even backups are trashed.
2: Someone wanting to frame another person. I've seen this done by clients of other consultants who do not want to pay the consulting fee. So they put a logic bomb in. The admin that left gets blamed and faces jail time. In this scenario, it is a word against word issue almost always, and juries tend to believe business owners far more than the admin who got railroaded.
You just restore the image from a ghost backup without worrying about the data because the data is stored (by policy) on the servers. What? A user ignored that policy? Tough luck for him.
Best Slashdot Co
You cant take any details from any news articles at face value.
Deleted
How many machines can you reimage in a day? Even if you only do one at a time, I imagine you could do 4 or 5 in a working day. If you have an entire office full, ready connected up to the network, you just have to pop in a CD (if you even need one) start the PC and move on. A couple of dozen people could do that lot in a weekends worth of overtime.
Most of the time I spend on rolling out a new PC is delivery, connection and admin. Where's the problem here?
I'll see your Constitution and raise you a Queen.
Twenty bucks says that they never figure out what happened.
And then he says to me You Gotta Make them Pay, Baby!
And I said Yeah Baby, I'm the Mad Code that Codes at Midnight!
And he says You're bad, baby! And I said Yeah!!!
If lil' ol' me can spend a few hundred dollars on enough hard drives stuffed into external enclosures the have two complete backups of all ~1.5TB of data in my system, surely a municipal government can spend a few thousand dollars to do it too.
What the hell, who runs systems that important without backups? Management teams named Shirley?
Living With a Nerd
There was a reason that somebody did this. It was somebody that knew this environment.
Either follow the money or follow the motive. They WILL find the perp.
We don't drink!
We don't cuss!
Norfolk! Norfolk! Norfolk!
i ~ Celebrating Science, Cyberspace, Speculation
"destroyed data on nearly 800 computers citywide".
By corrupting the Windows System32 folder install they lost their own files? Did the malware delete some key file that prevents Window's from hosing the disk and crushing the MFT and/or MBR? I doubt it. The OS installs may be unrecoverable, but the article / spokes people seem to jump the gun by stating such generalizations like "destroyed data" and "essentially destroyed these machines". I imagine that actual "data" of importance is still recoverable via external means, and that a quick reformat will make the machine quite OK again.
Maybe this is good incentive for them to install Linux, now that they have a ~800 machine testbed to work with.
'We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.' RPF
Just blame Terry Childs. It was a backdoor into a citywide system. Clearly he's responsible. Doubtlessly the D.A. is already concocting a theory involving him having visited the city during a conference, and installing a modem into the network. By phoning a specific number and entering a sequence of numbers from his prison phone, he's brought the network to its knees.
"you know why? Because we got the bomb, thats why" -Dennis Leary
If you're not a developer, and you don't install the "-dev" or "-devel" versions of the library packages, you won't have the dwarf2 source code debugging symbols, which can be quite verbose.
Search the Intertubes for "Windblows" and "Windoze" and enjoy the laughs.
Parroty Interactive had Microshaft Windblows 98.
When you sort the bits first compressed backups are really small.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
The mark of true intelligence is the ability to learn from the mistakes of others. Sad how few people seem to posses it.
* Check every few seconds to see if network goes down
* Write a bogus entry in the log files that points to some oddball behavior, like a disk-read error or something
* If network is down freeze screen so it looks like computer just locked up
* Ignore all input
* Wipe key parts of disk so forensic recovery is impossible or at least very difficult
* Wipe key parts of memory so forensic recovery is impossible or at least very difficult
* Wipe key parts of cache so forensic recovery is impossible or at least very difficult
* Force or fake a BSOD screen so a casual user will think his computer crashed and blame any resulting data loss on the crash
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Anyone else find it odd that the first thing the IT techs there "quickly isolated and rebuilt the offending print server."?
Sounds to me like they know who it may be and were covering up for a friend.
This sounds like a job for the Innocence Project.
It also sounds like something police and courts need to be made more aware of.
This sounds like the tech industry's equivalent of a divorcing parent accusing the other of child abuse - where the type of abuse doesn't leave scars and the child is too young to give credible evidence, it's 50/50 or less whether justice will prevail.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
That's a hard lesson to learn.
Now I understand why the desktops weren't backed up properly - their drive content was considered disposable. This is not unreasonable given the above statement.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
A similar thing happened where I work (uni campus), although due to config errors, not a timebomb.
400 machines got imaged and we're scrambling to collect drives, install new ones, reimage and then run recovery on the old orig drives.
Microsoft really needs to add the ability to set user profiles on a different partition, as you can w/ UNIX.
No sig for you!!
I think that Arizona, with its odd mix of Indian, Spanish, English and who-knows-what takes the cake with odd spellings and pronunciations.
Ft. Huachuca (Wa-chu-ka)
Mogollon Rim (Mo-gee-yawn)
Tempe (Tem-pee)
Canyon de Chelly (dee-shay)
On the other hand, I spent some time in Pueblo, Colorado where about 1/4 of those born there pronounced it Pee-eb-lo.
Great civilizations have lived and died on false theories. Don't mess up mine with a few facts.
FTA: "... the city found that the system serving as the distribution point ... was a print server. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server."
Ok, if I have a single workstation with "AntiVirus 2009", I will probably nuke it without a second thought. If one of my servers has been commandeered to serve as the command and control channel for a worm that just ate 800 of my PCs, I SURE AS HELL AM GOING TO GET A dd OR OTHER FORENSICALLY SOUND IMAGE OF THE MACHINE BEFORE I WIPE IT!!!!!!!!! For crying out loud, they contacted the FBI, but they just destroyed what could have been the single most important piece of evidence! Do they have a Best Buy in Norfolk? For $100 they could have brought the machine up on a clean hard disk and set the existing one aside for forensic examination without wasting the time of taking an image of the drive.
Also, they have no idea how the attack occurred, but they are sure it didn't come from the internet. Any evidence to back that up? It's one thing to say it probably didn't come from the internet because our logs show no traffic to support that possibility. It's ridiculous to make that same statement based on a gut feel.
If this article is accurate, these guys are playing amateur hour IT security. Their first action should have been to contact a qualified incident handler.
is destroyed the right word?
Just saying... I live nearby and it is well accepted the Norfolk, VA has an incredible amount of corruption. My first thought on reading this as a local was, "wow, that's a pretty creative way to cover your tracks..."
Growing up in Ohio, some of the pronunciations for local places are horrible.
The first are mostly just anglicizations. Not awful, but sometimes quaint, odd, and hickish. There are a lot more that I'm forgetting.
Lima - "LYE-muh".
Ravenna - "Ruh-VEN-nuh"
Medina - "Meh-DYE-nuh"
Berlin - "BER-lin' "
Milan - "MYE-lin'
Vienna - "VYE-en-nah"
Bellefontaine - "Bell Fountin' " Ack.
Then they just get really bad and annoying.
Nevada - "Nuh-VAY-duh". Really. And most locals pronounce the state Nuh-vah-da or Nuh-vad-ah, so what gives?
Mantua - "MAN-uh-way." The Italians are laughing and Shakespeare must be turning in his grave.
Versailles - "Vur-SAILS" Ugh.
A preposition is a terrible thing to end a sentence with.
The only Netware that is not a Netmare.
I fully believe a 12+ year uptime.
Bet it's still running strong.
2 was good once it was setup. Genning sys was a netmare however.
John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
Sarah Palin will not have sex with you.
Awwww....
-Todd
It would be interesting to see the evolution of the department's budget in the last few years.
It wouldn't surprise me that they got ride if their best assets and replaced them with cheaper guys with less than the required experience just to "reduce operating cost". They may have saved a few dollars during a year or two doing that but the shit had to spread one day...
When I set up my backup system, one of my concerns was that my loved ones might need to restore my backups, should I get hit by a truck someday.
I realized that if my filesystem is a rat's nest when I back it up, then my backups will be rat's nests as well, as would any restored data. So I have spent several months scrupulously organizing all of my filesystems on all of my computers.
That simplifies my backups, and would make it much easier to find what one needed when doing a restore.
My external drives both have ext3 filesystems. But they each have a small FAT32 partition containing installers for Mac OS X and Windows ext3 filesystem drivers. There are three such drivers available for Windows; I included all three.
My backups are not quite yet where they need to be, but they're getting close. I have automated continuous backup of my Subversion repository for instance. Whenever I do a checkin of my own personal code, a post-commit hook backs up my entire repository with the hot-backup.py script, then within three hours a cron job replicates the repository backup to my external drive.
I also need to write up detailed instructions that my survivors could follow, then mail them off to all of my relatives.
Recovering my data would be complex enough that these instructions would have to come right out and tell my mother to get my sister to do it for her. Happily my sister is hip to The Penguin.
Request your free CD of my piano music.
Re-worked summary of TFA:
- All that has been damaged is the System32 folder of user machines.
- 'Destroyed' I imagine is an IT staff trying to dumb down his language to his perception of the level of the reporter's IT knowledge
- Their IT may have done quite well, the only 'damage' is to PCs that were shut down in the 1 hour window between the attack starting and IT containing it
- Employees were supposed to save to the network. The only issue stated is that some staff were breaking the rules and saved things to their own PC.
All they need to do with the affected machines is to boot from a Windows or Linux CD, copy the files to memory stick and throw their standard "new install" image on. No data loss. No network down time. All they're looking at is some hassle for the ~ 18% of users affected and a very busy IT department. Provided the affected users have other machines to work on (or however not losing much productivity) they're not far off having the best scenario any It department can realistically hope for (well, I'd like to say it's reasonable to hope for not having pissed off employees). Sure, no doubt a dozen IT managers can post their "perfect" system, and another dozen IT managers can show how they could destroy it.
"There's too much money to be made by having a bot-army" bla bla bla.
I don't believe everyone is money driven. One day you'll meet a non-money driver dark-side cracker who shall write a mega bot. A huge one. And it's going to wipe clean tens of millions of Windows PCs.
Sadly it's going to be a very good day for Microsoft because a lot of persons are going to go buy a new PC but there's going to be some chaos with all that lost data.
It's not going to be 800. It's going to be the equivalent of "rm -rf *" on an entire botnet.
Shall be fun. Will happen.
That's what happens when you don't know what is going on under the hood.
GO FOSS!!!
If all this did was modify or delete the system32 directory, then the data is still going to be on the drives and should be easily recoverable - so I wouldn't refer to those machines as "essentially destroyed," - I wouldn't even refer to the data as "destroyed."
All it means is the machine wont boot normally. I know for most users that renders the machine temporarily useless, but even a low level IT tech should be able to recover data or get the system booting again, there are about 5 different ways to do it.
That doesn't change the fact that someone unleashed this on a civic network, but it bothers me when electronic attacks are described in a way that makes them sound much worse than they actually are - because we already have the government looking to use "cyber security" as the next big issue which they'll surely try to tackle via censorship, privacy violations, internet filtering, and wrongheaded laws.
According to the local 6 pm news, all fixed and back online, data intact. Evidently, the affected machines were on a shared network, NOT just the City's.
Some one said it came from the print sever so any will to bet in came form some hole in HP or some other vendor software?
Came through the rift in Cardiff and drifted all the way to Norfolk.....
Paula Bean was working there? May be?
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
There was the guy who locked out administrator privileges in the San Francisco computer system. He recently went on trial in December, but I have not heard a verdict.
If you need a lair, hold the world's data hostage for One... Hundred... BILLION DOLLARS!
yes reformatting the machine there total useless now. total loss call the insurance company.
We don't drink!
We don't smoke!
Norfolk! Norfolk!
""Bricked" means non-responsively broke."
No, "bricked", means unfixably broken.
If you use a word incorrectly, don't get all snotty when people point it out. Just pick the right word. "The machines are unbootble" or "The file system was damaged rendering the computer inoperable".
You're a technical person, use the correct technical language. Don't expect us to have to wade through your verbal sloppiness.
http://blogs.pcmag.com/securitywatch/2010/02/rootkit_authors_issue_patch_fo.php
Tuesday February 16, 2010
Rootkit Authors Issue Patch For Critical Bug
Categories:
Malware, Security Software, Software Patches, Top Threat, Windows 7, Windows Vista, Windows XP
Tags:
bsod, ms10-015, patch, rootkit, TDL3, TDSS
According to security vendor Prevx, the authors of the rootkit which was the cause of a large number of unbootable systems which applied the MS10-015 patch issued last week have issued a patch to fix the incompatibility.
The authors of the rootkit, which Prevx names TDL3/TDSS, have been active for months updating it to evade detection and to defend itself against removal. It has many other names from other vendors. The error which caused the BSOD was due to the rootkit hard-coding the address of a particular Windows routine, and this address was moved by MS10-015. The rootkit authors had a fix out before too long, but that wasn't enough to save large numbers of users who couldn't boot their systems.
LET THIS BE A LESSON TO ALL YOU SYSTEM ADMINISTRATORS!
whom I have heard saying (repeatedly) - "it is not a critical server, it is only a print server... we can wait to patch it later."
From just the article, I have a pretty good guess as to what or how it happened... or how I could replicate such an event with two commands, and little or no evidence left behind.
A disgruntled citizen comes in to use a public access terminal placed there for citizens to look up public records, and PRINT THEM OUT. This public terminal is locked down - sure, it is also on its own private VLAN, lest anyone plug into the network with their own laptop... heck, lets go one further and say they even bound the mac address to the switch port to make sure that any other network device plugged in wouldn't work (unless they spoofed the mac address).
So, our Disgruntled Citizen Hacker (DCH) takes a bootable USB thumb drive/boot CD and inserts it into the computer and reboots it to Backtrack4 or some other utility - or they simply plug into the network using their own laptop...
Once booted from his device, DCH launches an ancient exploit against the print server that "doesn't contain any sensitive data" according to the SYSADMIN "and can be rebuilt within hours if it ever got infected." - except that DCH isn't all about stealing data, he's all about getting revenge against the cop that gave him that speeding ticket - and HE'S GONNA SHOW YOU!
Once his script kiddie exploit has him sitting at the c:\ prompt, he does a "NET VIEW" and sees that the print server is on the domain, and can see the entire network from its secondary interface that connects it to the internal network. This system administrator has even copied the SYSINTERNALS suite of tools to the hard drive (he even added them to the PATH! -OR- he copies the SYSINTERNALS suite from his boot device) and with one command, DCH gets to work. "PSEXEC \\* DEL c:\boot.ini" and hits enter, the command starts cycling through all the computers on the network -but he screwed up... it is taking much too long to connect to each computer - only to screw up the boot.ini file? Naw, thats too easy to recover from.
CTRL+C
-DCH's Adrenaline is now pumping-
PSEXEC -d \\* DEL *.* /F /Q /S
This time, it runs in disconnected mode.
"Ah yes, much faster." DCH says to himself - except he screwed up again, he forgot to put the "C:\" in front of the *.*, so it is (Q)uietly, yet (F)orcefully deleting all the files listed under the %SystemRoot%\System32 folder and (S)ub-folders (including those files marked as read only), instead of the entire C: drive. Major adrenaline sets in - he's not gonna cancel it this time. He's already committed, it's too late now. That and he's lost his nerve and is visibly shaking as he's feeling the rush.
He retrieves his boot device, reboots the computer, and quietly walks away, trying oh-so-hard to not raise any suspicions as he quietly walks back to his car. "Take THAT..Your Honor." he mumbles to himself as he jams the key into his Honda Civic, it fires up with a roar as the ported exhaust reverberates throughout the parking garage. He revs the engine and squeals the tires as he leaves the ramp - radio blaring.
One hour and 800 computers later the print server is taken offline -and promptly rebuilt- exactly according to the disaster recovery plan. Doesn't matter - even if they did forensically analyze it, the only evidence they'll find is a single error (among thousands of errors) in the event log that was caused by the exploit, of itself signifying nothing conclusive. The admins never did set up event log correlation, so once the server was rebuilt, all bet were off. So, our DCH walks away, scot free.
But wait! Did he really?
Check the courthouse cameras. On Tuesday, Feb. 9, sitting down at 4:07pm you'll see the DCH take his seat at the public terminal. He looks around and cannot believe that the stupid IT depar
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
The teams name was NcGill.
They called themselves Lill.
But everyone knew them as Nancy.
No brain, no pain.