Slashdot Mirror


Those 100,000 Lost Air Force Files Have Been Found Again (govexec.com)

The Air Force now says it will be able to recover those 100,000 investigation files dating back to 2004, after "aggressively leveraging all vendor and department capabilities." An anonymous reader quotes a report from Government Executive about the mysteriously corrupted database: In a short, four-sentence statement released midday on Wednesday, service officials said the Air Force continues to investigate the embarrassing incident in which the files and their backups were corrupted. "Through extensive data recovery efforts over the weekend and this week, the Air Force has been able to regain access to the data in the Air Force Inspector General Automated Case Tracking System..." the statement reads. Earlier on Wednesday, the Air Force chief of staff said that the effort to recover the files involved Lockheed Martin and Oracle, the two defense contractors that run the database, plus Air Force cyber and defense cyber crime personnel.
The Chief of Staff hopes "there won't be a long-term impact, other than making sure we understand exactly what happened, how it happened and how we keep it from ever happening again." The Air Force is conducting an independent review, while Lockheed Martin is now also performing a separate internal review.

21 of 36 comments (clear)

  1. So I'm not the only one by Hognoxious · · Score: 5, Insightful

    So I'm not the only one who has a shoebox full of floppies and no idea at all what's on them.

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    1. Re:So I'm not the only one by ortholattice · · Score: 2

      I have a shelf full of DEC TK50 tape cartridges, and I do know what's on them: all the programs, many in C, that I wrote before the mid-90s. Some of which I was rather proud of, with clever algorithms (IMNSHO) and meticulously documented.. The VAX gave up the ghost with no funds to repair/replace, and the backup tapes were all that was left.. While I don't have any realistic hope of recovering them without spending a lot of money, I keep them around in case my lottery ticket comes through.

  2. What a scandal by Anonymous Coward · · Score: 1

    What a horrible scandal; they didn't actually lose the data, they just took a government contractor time frame to recover a corrupted database. I'm guessing the worst part of LockMart's inability to figure out which form to use to report the problem.

  3. This Sounds Familiar by Anonymous Coward · · Score: 5, Interesting

    This sounds like an incident at my workplace. HR had an ancient SCO UnixWare server that we (IT) never knew about. Damn thing crashed, 2 failed hard drives on a RAID 1 array. Turns out they knew for years that one drive had failed AND there were memory errors, but they never bothered to tell us or anyone about it to fix the issues. They had a tape drive backup, but that also failed with hardware problems.

    Naturally, once the server wouldn't boot, they went into full panic mode and are trying to recover the data from a failed RAID and tape backups dating back to the 1960's (don't ask me how they plan to do that, I have no idea).

    Now they are "investigating the incident" and "working to prevent future failures like this in the future". It's the same at every workplace.

    1. Re:This Sounds Familiar by l0n3s0m3phr34k · · Score: 1

      LOL, indeed. Your story reminds me of when I finally got to go into the "secure bunker" area of the old SABRE cold-war underground data center. Some ancient SCO box, making a horrible whining noise...a stack of other boxes of the same model next to it, we had already scavenged them for all their parts and "rebuilt" this box before. No more replacement parts. We didn't even have a backup either, and this box was considered somewhat important for all the scripts it ran that monitored other systems...just not "important" enough to allocate time for the Front End team to even migrate over to a VM. Since our client is so hardcore ITIL, we couldn't just do a VM on our own without proper change control procedures, a dozen sign-offs, etc.

      The bunker area itself was pretty cool though; two stories underground, foot-thick old-school blast door, etc. Took me two years of asking just to finally get in and see it...real piece of history there!

    2. Re:This Sounds Familiar by TheCarp · · Score: 1

      > This sounds like an incident at my workplace.

      I was thinking the same thing about a different part....corrupted backups. Old job had a mail server update that broke the backups in a subtle way: it corrupted the mailbox names such that the path wouldn't match when the restore looked for it.

      As support waved a dead chicken over it, I learned a few commands that turned out to be how to dump data streams off the tape from the command line, and was able to write a perl script to extract mailboxes that we needed, but, we were not exactly able to put the kind of pressure on vendors that gets them to go any further than "Sorry your backups are corrupt"; we were on our own to solve it after that.

      --
      "I opened my eyes, and everything went dark again"
  4. At this point , I just need to ask: by Izhido · · Score: 3, Interesting

    What the hell does "leverage" mean?

    I'm no longer able to pretend to know what these nonsense buzzwords mean.

    1. Re:At this point , I just need to ask: by zenlessyank · · Score: 1

      It is the age of levers. Or. It is when someone has a gun to your head and uses the fact of leverage over your life to make you do something you don't want to do. I means someone uses an imaginary power over something to get something done. That is my GED version of the definition.

    2. Re:At this point , I just need to ask: by SeaFox · · Score: 2

      What the hell does "leverage" mean?

      In this case it means they actually tried to recover the files, instead of just claiming they couldn't because they're lazy, or because they would rather the files be lost anyway.

      You see this happen in private industry too, when files needed for legal proceedings are "lost" and the judge isn't going to let that get by. Miraculously, "unrecoverable files" are suddenly retrieved.

    3. Re:At this point , I just need to ask: by Impy+the+Impiuos+Imp · · Score: 1

      What the hell does "leverage" mean?

      I'm no longer able to pretend to know what these nonsense buzzwords mean.

      What the hell does "leverage" mean?

      I'm no longer able to pretend to know what these nonsense buzzwords mean.

      I'm sorry they removed the study of simple machines in 6th grade in favor of a unit on how not to offend on twitter. It is a wedge issue, to be sure, when it was wheeled out without much discussion by people with undue pully on the situation. I'm inclined to think they've put the screws to future generations.

      --
      (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    4. Re:At this point , I just need to ask: by MountainLogic · · Score: 1

      They finally paid Cozy Bear to give them copy of the files

    5. Re:At this point , I just need to ask: by Toad-san · · Score: 1

      Accompanied by some serious threats to never give the careless bastiges another contract.

  5. International Cooperation by NMBob · · Score: 4, Funny

    They must have asked for a copy the Chinese and Russians had stolen.

  6. Lost, found again? by Yvan256 · · Score: 1

    This reminds me of the Vogons. They are one of the most unpleasant races in the Galaxy. Not actually evil, but bad-tempered, bureaucratic, officious and callous. They wouldn't even lift a finger to save their own grandmothers from the Ravenous Bugblatter Beast of Traal without orders – signed in triplicate, sent in, sent back, queried, lost, found, subjected to public inquiry, lost again, and finally buried in soft peat for three months and recycled as firelighters. The best way to get a drink out of a Vogon is to stick your finger down his throat, and the best way to irritate him is to feed his grandmother to the Ravenous Bugblatter Beast of Traal. On no account should you allow a Vogon to read poetry at you.

  7. Chain of Custody? by Irate+Engineer · · Score: 1

    If these are active AF investigations, do they have a documented chain of custody for these files if they are evidence (emails, etc.)?

    Even under the UCMJ, I think that the validity of any evidence files on these servers could be questioned.

    --

    Left MS Windows for Linux Mint and never looked back!

    Vote for Bernie in 2016!

    1. Re:Chain of Custody? by ColdWetDog · · Score: 1

      "Hey guys! Look at this! Push the button and the whole thing boots up again!"

      "We fixed it!"

      --
      Faster! Faster! Faster would be better!
  8. Re:Once again. . . by gtall · · Score: 1

    "private industry doing it better than government" The thing about private industry is they get to decide who will be their clients. Government has to take everyone as client. Ever listen to CSPAN's call in show? Grandma the gun-nut is sure the Jews are behind Islamic terrorism and taking her tax money...and she regularly calls the IRS to explain to them the depth of her problems.

  9. Re:Dude! by eric_harris_76 · · Score: 1

    So, what's the deal? Do they just do this for grins, or do they expect something in return -- like retain a copy of the contents?

    There's got to be someone who -- for a price -- will take the contents of those tapes and transfer them to another medium. Maybe some kind-hearted and mostly trustworthy OpenVMS Sys Admin.

    Which medium will eventually become obsolete, I expect. Lather, rinse, repeat.

    --
    There's no time like the present. Well, the past used to be.
  10. Re:Once again. . . by Reziac · · Score: 1

    From: opportunity@turbans.iq
    Subject: They're at it again
    Disposition: spambinned

    rinse and repeat thousands of times...

    --
    ~REZ~ #43301. Who'd fake being me anyway?
  11. Re:At this point, I just need to ask: by wwphx · · Score: 1

    I have in front of me of my recently arrived DVD complete Yes Minister, and I am reminded of an exchange between Sir Humphrey and Jim Hacker. To paraphrase:

    Sir Humphrey: There are two responses. "It is under consideration", which means we've lost it.
    Hacker: And the other?
    Humphrey: "It is under active consideration", which means we're looking for it.

    --
    When you sympathize with stupidity, you start thinking like an idiot.
  12. Ransomware by peawormsworth · · Score: 1

    ...plus Air Force cyber and defence cyber crime personnel

    I suspect ransomware was at play... and then they just paid it.

    why else bring in the cyber crime personnel... who will just say "pay the ransom"