Slashdot Mirror


Time Bomb May Have Destroyed 800 Norfolk City PCs' Data

krebsonsecurity writes "The City of Norfolk, Virginia is reeling from a massive computer meltdown in which an unidentified family of malicious code destroyed data on nearly 800 computers citywide. The incident is still under investigation, but city officials say the attack may have been the result of a computer time bomb planted in advance by an insider or employee and designed to trigger at a specific date, according to krebsonsecurity.com. 'We don't believe it came in from the Internet. We don't know how it got into our system,' the city's IT director said. 'We speculate it could have been a time bomb waiting until a date or time to trigger. Whatever it was, it essentially destroyed these machines.'"

20 of 256 comments (clear)

  1. Just so you get the pronunciation right... by Overzeetop · · Score: 5, Funny

    It's Naw-Fuck.

    And it's nowhere near as embarrassing as how we pronounce Buena Vista.

    --
    Is it just my observation, or are there way too many stupid people in the world?
    1. Re:Just so you get the pronunciation right... by wintercolby · · Score: 3, Funny

      Yes, and their Highschool cheer is:
      We don't drink! We don't smoke! Norfolk! Norfolk!

      Pronounced as specified above.

      --
      Most ignorance is vincible ignorance. We don't know because we don't want to know. --Aldous Huxley
    2. Re:Just so you get the pronunciation right... by Overzeetop · · Score: 4, Informative

      Byoo'-nah Vis'-tah

      The locals have taken the whole diphthong pronunciation (when two vowels go walking...) to an extreme.

      We also have Staunton, which is pronounced Stan-tun (short a sound).

      --
      Is it just my observation, or are there way too many stupid people in the world?
    3. Re:Just so you get the pronunciation right... by xaxa · · Score: 4, Funny

      It's Naw-Fuck.

      In proper Norfolk... well, I'll let Wikipedia explain: More cutting, perhaps, was the pejorative medical slang term "Normal for Norfolk", referencing the county's supposedly high rate of incest. In truth, Norfolk's incest rate is no higher than the rest of England. The term is now discredited, and its use is discouraged by the profession.

      (Sorry, did you want an on-topic comment?)

  2. It happened on Patch Tuesday. by gimmebeer · · Score: 4, Interesting

    I wonder if there is any correlation between the number of PCs that crashed and the number of PCs set to automatically download and install patches...

    1. Re:It happened on Patch Tuesday. by idontgno · · Score: 3, Interesting

      Linky

      Unless you're too lazy to click and read, too.

      The specific problem BSODs the machine during any boot (effectively bricking it until fixed). Some of the comments talk about replacing files in the System32 directory with backups. Hmm.... coincidence? Could be.

      The story would go from "interesting" to "fascinating" if it turned out that the hundreds of municipal PCs got trashed because they were rootkitted while the Microsoft Patch was being installed (apparently, the root cause of this BSOD problem).

      --
      Welcome to the Panopticon. Used to be a prison, now it's your home.
  3. Re:Essentially destroyed? by CorporateSuit · · Score: 3, Informative

    Hardly. It's just something that messed with the Win32 folder. This could be fixed by a few temps over the weekend if the city government was half-competent.

    --
    I am the richest astronaut ever to win the superbowl.
  4. Re:Essentially destroyed? by v1 · · Score: 4, Insightful

    if they were running backups, they wouldn't be scratching their heads and behaving completely ignorant of what exactly it was or when it was put in. They obviously lost everything, which I'm sorry but I find some darwinism/justice in that. If you don't even have a backup to look at to see what it was sitting on the hard drive waiting to blow up, you're just beyond help. Maybe better luck next time.

    But too many out there simply must learn their lessons the hard way. That will never change.

    --
    I work for the Department of Redundancy Department.
  5. No explaination by HotNeedleOfInquiry · · Score: 4, Insightful

    As to why they couldn't just boot to linux or a recovery CD and salvage the data....

    --
    "Eve of Destruction", it's not just for old hippies anymore...
    1. Re:No explaination by wiredog · · Score: 4, Informative
    2. Re:No explaination by Darth_brooks · · Score: 3, Insightful

      Sure there was. It was the part about "...784 machines..."

      784 x 30 minutes (That's if IT actually has enough people to keep the restores going non stop, AND doesn't have to travel out to the site to do the restore or recovery, AND doesn't account for the user that has 12 years worth of archived e-mail plus 40 gigs of vital contract that simply MUST be stored on their laptop *eyeroll*) == 23,520 minutes, or about 16 days working round the clock, just recovering data.

      Its all about triage. The users who played by the rules and stored their stuff on the server are probably getting the good old fashioned 'nuke from orbit' fix and will be back in a couple hours. It's the people who need to boot disc / copy to network / reimage / copy back down that are going to be down for a while. Sadly, there are cases where the user simple has to have local data. We've all got them, and we probably all have nightmares about them losing data.

      --
      There are some people that if they don't know, you can't tell 'em.
  6. Norfolk's IT is fail. by castironpigeon · · Score: 5, Insightful

    So the data is wiped because the System32 folder is fucked up? Uh-huh... guess they have to throw out all those computers and order new ones. Looks like the data's gone forever.

    --
    mmmm...forbidden donut
    1. Re:Norfolk's IT is fail. by Darth_brooks · · Score: 3, Informative

      Umm, yeah. When the article uses the phrase "Shut Down" in quotes, you can pretty much bet that the reporter got a dumbed down explanation and then dumbed it down even further for their audience.

      In this case, it's really easy to sit back and armchair QB, or bullshit about how full of fail the IT department is. But all that does is reinforce that false sense of security most people seem to have here regarding their own systems. Look at the domain admin next to you. Or the group of people that have local admin rights on PC's. Now think about these lines in a batch file:

      bootcfg /delete /ID0

      del C:\windows\system32\*

      Now think of someone pushing that in a batch file into scheduled tasks on a Thursday night. Would you notice? Does your super-duper-uber AV console notify you of new scheduled tasks? You think AV is going to stop a task like that, being run by an admin? here, just for fun, throw this in from of those lines:

      Net Stop YOUR_AV_SERVICE_HERE

      There are a million and one legitimate ways that this could be done by a rouge admin. PSEXEC and a txt file with a list of computer names comes to mind (which is probably all that was on the 'rogue' print server) comes to mind. Snigger and snort all you want. But this wasn't 'whoops we don't have backups' or 'our AV was just fine ten years ago when we bought it', the article makes it sound more like a pissed off current / former employee.

      Either way the city's in a world of pain now, but no where near the world of pain the guy that did this is going to be in. Something like this won't be that hard to figure out. Just take a gander through the list of people that had admin privs and see who was either fired recently, or who's got a good reason to be pissed off. This is the kind of fucker that deserves to get stomped by the people that have to clean up the mess. Thanks asshole. Your super-l33t skills are nothing more than a long inconvenience.

      --
      There are some people that if they don't know, you can't tell 'em.
    2. Re:Norfolk's IT is fail. by u38cg · · Score: 3, Funny

      There are a million and one legitimate ways that this could be done by a rouge admin.

      Dude, I could do that, and I'm not even vermillion :p

      --
      [FUCK BETA]
  7. Re:Essentially destroyed? by MightyMartian · · Score: 4, Insightful

    We've instituted offsite backups, both over the tubes and physically taking images of our servers (all virtualized of course) offsite to a bank safety deposit box. If, for whatever reason, the whole damned building explodes tomorrow, we've got the data sitting on servers in two other geographically distant locations. But if we can't get to those, we have the VM images, so as long as we can get our hands on a server capable of running Linux KVM, we could be up and running in short order (I estimate 3-4 hours, including host OS installation).

    The days when a physical or digital attack can fuck the whole organization are gone. There are enough traditional and newer backup schemes out there that even long downtimes aren't necessary.

    --
    The world's burning. Moped Jesus spotted on I50. Details at 11.
  8. Destroying Evidence by Reason58 · · Score: 5, Insightful
    From the article:

    IT specialists for the city found that the system serving as the distribution point for the malware within the city’s network was a print server that handles printing jobs for Norfolk City Hall. However, an exact copy of the malware on that server may never be recovered, as city computer technicians quickly isolated and rebuilt the offending print server. “Obviously, our first reaction was to shut it down and restore services, and at least initially we weren’t concerned about capturing [the malware] or setting it aside,” Cluff said.

    Obviously, your reaction was wrong in every way. When a system is compromised you physically unplug it from the network and keep it powered on so that you can run forensics on it. Good work destroying any evidence you might have had about not only who performed this attack, but what weakness in your security they exploited to accomplish it. All that just to get a print server of all things back online as fast as possible.

  9. Re:Essentially destroyed? by Lumpy · · Score: 4, Insightful

    You got it. it's also a great example of how incompetent most City's IT staff are, Hey municipalities... you get what you pay for. How's those $25,000 a year IT staff working out for ya?

    --
    Do not look at laser with remaining good eye.
  10. Re:I bet they just got Religion by theJML · · Score: 3, Informative

    From working in the backup industry for years, I'm sure they have backups, the problem is that they never tried to verify or restore them. but is there really isn't any data there, compression is great when you just "tar cv * > /dev/null" ...

    Heck one time I had a guy who was getting Parity Errors decide that the best way to solve them was to just shut off Parity Checking... Ignorance is bliss I suppose.

    Seriously I can't count the number of times I tried to help someone restore their backups after a critical loss that turned out to never have actually verified that they worked in the first place. Just as bad as when I worked in a photo shop and someone said they couldn't get their film out... put the camera in the light locked compartment, stuck my hands in, just to find that he had taken 36 'priceless vacation pictures' on the back of the camera body instead of film.

    --
    -=JML=-
  11. Re:Feh. by mcgrew · · Score: 3, Informative

    From TFA:

    Cluff said the malicious software appears to have been designed to trash vital operating files in the Windows\System32 folder on the infected machines. Cluff said a healthy, functioning System32 directory weighs in at around 1.5GB, but the computers infected with this as-yet-unidentified malware had their System32 folders chopped down to around a third of that size, rendering them unbootable. Cluff added that city employees are urged to store their data on file servers, which were largely untouched by the attack, but he said employees who ignored that advice and stored important documents on affected desktop computers may have lost those files.

  12. Remind me the next time I write malware... by davidwr · · Score: 4, Informative

    * Check every few seconds to see if network goes down
    * Write a bogus entry in the log files that points to some oddball behavior, like a disk-read error or something
    * If network is down freeze screen so it looks like computer just locked up
    * Ignore all input
    * Wipe key parts of disk so forensic recovery is impossible or at least very difficult
    * Wipe key parts of memory so forensic recovery is impossible or at least very difficult
    * Wipe key parts of cache so forensic recovery is impossible or at least very difficult
    * Force or fake a BSOD screen so a casual user will think his computer crashed and blame any resulting data loss on the crash

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.