Slashdot Mirror


Those 100,000 Lost Air Force Files Have Been Found Again (govexec.com)

The Air Force now says it will be able to recover those 100,000 investigation files dating back to 2004, after "aggressively leveraging all vendor and department capabilities." An anonymous reader quotes a report from Government Executive about the mysteriously corrupted database: In a short, four-sentence statement released midday on Wednesday, service officials said the Air Force continues to investigate the embarrassing incident in which the files and their backups were corrupted. "Through extensive data recovery efforts over the weekend and this week, the Air Force has been able to regain access to the data in the Air Force Inspector General Automated Case Tracking System..." the statement reads. Earlier on Wednesday, the Air Force chief of staff said that the effort to recover the files involved Lockheed Martin and Oracle, the two defense contractors that run the database, plus Air Force cyber and defense cyber crime personnel.
The Chief of Staff hopes "there won't be a long-term impact, other than making sure we understand exactly what happened, how it happened and how we keep it from ever happening again." The Air Force is conducting an independent review, while Lockheed Martin is now also performing a separate internal review.

36 comments

  1. So I'm not the only one by Hognoxious · · Score: 5, Insightful

    So I'm not the only one who has a shoebox full of floppies and no idea at all what's on them.

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    1. Re:So I'm not the only one by ortholattice · · Score: 2

      I have a shelf full of DEC TK50 tape cartridges, and I do know what's on them: all the programs, many in C, that I wrote before the mid-90s. Some of which I was rather proud of, with clever algorithms (IMNSHO) and meticulously documented.. The VAX gave up the ghost with no funds to repair/replace, and the backup tapes were all that was left.. While I don't have any realistic hope of recovering them without spending a lot of money, I keep them around in case my lottery ticket comes through.

    2. Re:So I'm not the only one by Anonymous Coward · · Score: 0

      Goto comp.os.vms and tell them your problem AND your location within your country. There might be someone nearby who can help you.

  2. What a scandal by Anonymous Coward · · Score: 1

    What a horrible scandal; they didn't actually lose the data, they just took a government contractor time frame to recover a corrupted database. I'm guessing the worst part of LockMart's inability to figure out which form to use to report the problem.

  3. This Sounds Familiar by Anonymous Coward · · Score: 5, Interesting

    This sounds like an incident at my workplace. HR had an ancient SCO UnixWare server that we (IT) never knew about. Damn thing crashed, 2 failed hard drives on a RAID 1 array. Turns out they knew for years that one drive had failed AND there were memory errors, but they never bothered to tell us or anyone about it to fix the issues. They had a tape drive backup, but that also failed with hardware problems.

    Naturally, once the server wouldn't boot, they went into full panic mode and are trying to recover the data from a failed RAID and tape backups dating back to the 1960's (don't ask me how they plan to do that, I have no idea).

    Now they are "investigating the incident" and "working to prevent future failures like this in the future". It's the same at every workplace.

    1. Re:This Sounds Familiar by l0n3s0m3phr34k · · Score: 1

      LOL, indeed. Your story reminds me of when I finally got to go into the "secure bunker" area of the old SABRE cold-war underground data center. Some ancient SCO box, making a horrible whining noise...a stack of other boxes of the same model next to it, we had already scavenged them for all their parts and "rebuilt" this box before. No more replacement parts. We didn't even have a backup either, and this box was considered somewhat important for all the scripts it ran that monitored other systems...just not "important" enough to allocate time for the Front End team to even migrate over to a VM. Since our client is so hardcore ITIL, we couldn't just do a VM on our own without proper change control procedures, a dozen sign-offs, etc.

      The bunker area itself was pretty cool though; two stories underground, foot-thick old-school blast door, etc. Took me two years of asking just to finally get in and see it...real piece of history there!

    2. Re:This Sounds Familiar by TheCarp · · Score: 1

      > This sounds like an incident at my workplace.

      I was thinking the same thing about a different part....corrupted backups. Old job had a mail server update that broke the backups in a subtle way: it corrupted the mailbox names such that the path wouldn't match when the restore looked for it.

      As support waved a dead chicken over it, I learned a few commands that turned out to be how to dump data streams off the tape from the command line, and was able to write a perl script to extract mailboxes that we needed, but, we were not exactly able to put the kind of pressure on vendors that gets them to go any further than "Sorry your backups are corrupt"; we were on our own to solve it after that.

      --
      "I opened my eyes, and everything went dark again"
  4. Once again. . . by smooth+wombat · · Score: 0, Troll

    private industry doing it better than government:

    the effort to recover the files involved Lockheed Martin and Oracle, the two defense contractors that run the database,

    Wonder how much these two billed the taxpayers to clean up the mess they created?

    --
    We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
    1. Re:Once again. . . by Anonymous Coward · · Score: 0, Troll

      Apparently private industry does it better than government - these records were found inside of a week.
      - the IRS took over a year to find Lois Lerner's e-mails
      - Hillary's e-mails are still missing

    2. Re:Once again. . . by gtall · · Score: 1

      "private industry doing it better than government" The thing about private industry is they get to decide who will be their clients. Government has to take everyone as client. Ever listen to CSPAN's call in show? Grandma the gun-nut is sure the Jews are behind Islamic terrorism and taking her tax money...and she regularly calls the IRS to explain to them the depth of her problems.

    3. Re:Once again. . . by smooth+wombat · · Score: 0, Troll

      I guess you're right. The thousands of daily emails warning Bush of an impending attack were never recovered either.

      --
      We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
    4. Re:Once again. . . by Reziac · · Score: 1

      From: opportunity@turbans.iq
      Subject: They're at it again
      Disposition: spambinned

      rinse and repeat thousands of times...

      --
      ~REZ~ #43301. Who'd fake being me anyway?
  5. At this point , I just need to ask: by Izhido · · Score: 3, Interesting

    What the hell does "leverage" mean?

    I'm no longer able to pretend to know what these nonsense buzzwords mean.

    1. Re:At this point , I just need to ask: by zenlessyank · · Score: 1

      It is the age of levers. Or. It is when someone has a gun to your head and uses the fact of leverage over your life to make you do something you don't want to do. I means someone uses an imaginary power over something to get something done. That is my GED version of the definition.

    2. Re:At this point , I just need to ask: by SeaFox · · Score: 2

      What the hell does "leverage" mean?

      In this case it means they actually tried to recover the files, instead of just claiming they couldn't because they're lazy, or because they would rather the files be lost anyway.

      You see this happen in private industry too, when files needed for legal proceedings are "lost" and the judge isn't going to let that get by. Miraculously, "unrecoverable files" are suddenly retrieved.

    3. Re:At this point , I just need to ask: by Impy+the+Impiuos+Imp · · Score: 1

      What the hell does "leverage" mean?

      I'm no longer able to pretend to know what these nonsense buzzwords mean.

      What the hell does "leverage" mean?

      I'm no longer able to pretend to know what these nonsense buzzwords mean.

      I'm sorry they removed the study of simple machines in 6th grade in favor of a unit on how not to offend on twitter. It is a wedge issue, to be sure, when it was wheeled out without much discussion by people with undue pully on the situation. I'm inclined to think they've put the screws to future generations.

      --
      (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    4. Re:At this point , I just need to ask: by MountainLogic · · Score: 1

      They finally paid Cozy Bear to give them copy of the files

    5. Re:At this point , I just need to ask: by Toad-san · · Score: 1

      Accompanied by some serious threats to never give the careless bastiges another contract.

  6. we lost it, oh my bad, there it is. by Anonymous Coward · · Score: 0

    but they sound so much more in control and macho (no, not really it just sounds like they had to pay some else to do the finding)

  7. International Cooperation by NMBob · · Score: 4, Funny

    They must have asked for a copy the Chinese and Russians had stolen.

    1. Re:International Cooperation by Anonymous Coward · · Score: 0

      I bet this is actually how they got the records back, the story about cooperating with Lockheed and Oracle was created just to save face.

  8. Lost, found again? by Yvan256 · · Score: 1

    This reminds me of the Vogons. They are one of the most unpleasant races in the Galaxy. Not actually evil, but bad-tempered, bureaucratic, officious and callous. They wouldn't even lift a finger to save their own grandmothers from the Ravenous Bugblatter Beast of Traal without orders – signed in triplicate, sent in, sent back, queried, lost, found, subjected to public inquiry, lost again, and finally buried in soft peat for three months and recycled as firelighters. The best way to get a drink out of a Vogon is to stick your finger down his throat, and the best way to irritate him is to feed his grandmother to the Ravenous Bugblatter Beast of Traal. On no account should you allow a Vogon to read poetry at you.

  9. Color me skeptical by Anonymous Coward · · Score: 0

    Why do I have the feeling that there will be a number of situations where they say..."Sorry, while we have some of that data, there were some key parts lost."

    New mantra for the Air Force "Blame it on the floppy"

  10. Chain of Custody? by Irate+Engineer · · Score: 1

    If these are active AF investigations, do they have a documented chain of custody for these files if they are evidence (emails, etc.)?

    Even under the UCMJ, I think that the validity of any evidence files on these servers could be questioned.

    --

    Left MS Windows for Linux Mint and never looked back!

    Vote for Bernie in 2016!

    1. Re:Chain of Custody? by ColdWetDog · · Score: 1

      "Hey guys! Look at this! Push the button and the whole thing boots up again!"

      "We fixed it!"

      --
      Faster! Faster! Faster would be better!
    2. Re: Chain of Custody? by Anonymous Coward · · Score: 0

      Well, the IG doesn't do judicial investigations, that's the OSI and base law enforcement. However, we are way too incompetent to have done this as a cover to delete some information. And yes, ongoing IG investigations are done on Lt Col Snuffy's laptop that might maybe be backed up to a external HDD and the forms and reports are uploaded to the DOD IG system.

  11. No they did NOT. by Anonymous Coward · · Score: 0

    They are lying.

  12. What's also important by Anonymous Coward · · Score: 0

    Making sure that the smaller set of deliberately lost content is inaccessible to the restoration procedures. That was the whole poin tof losing them, right?

    The files probably contained the missing 17 minutes form Nixon's illegal audio recordings form the Oval Office.

    1. Re:What's also important by Anonymous Coward · · Score: 0

      Making sure that the smaller set of deliberately lost content is inaccessible to the restoration procedures. That was the whole poin tof losing them, right?

      The files probably contained the missing 17 minutes form Nixon's illegal audio recordings form the Oval Office.

      No. They recovered specifically 0.00% of that data.

  13. *** JOIN US TO FIGHT THE SPYING *** by Anonymous Coward · · Score: 0

    https://www.eff.org/nsa-spying

  14. Dude! by Anonymous Coward · · Score: 0

    Send them to bitsavers.org!
    They're doing recovery/scanning/documentation of *ALL* that stuff.
    Additionally they have archive.org helping to maintain a mirror (They have pdfs of scans of electronics documents dating from ENIAC through PDP/VAX to late 90s PARISC/SPARC/x86/Alpha, including firmware and disc backups.)

    Their archive is 250+Gb and growing weekly.

    1. Re:Dude! by eric_harris_76 · · Score: 1

      So, what's the deal? Do they just do this for grins, or do they expect something in return -- like retain a copy of the contents?

      There's got to be someone who -- for a price -- will take the contents of those tapes and transfer them to another medium. Maybe some kind-hearted and mostly trustworthy OpenVMS Sys Admin.

      Which medium will eventually become obsolete, I expect. Lather, rinse, repeat.

      --
      There's no time like the present. Well, the past used to be.
  15. What's all the fuss about? by Anonymous Coward · · Score: 0

    Couldn't they have just asked China for a copy of the files?

  16. We have FOUND THE ONE by Anonymous Coward · · Score: 0

    case we really wanted to ensure went away and made sure it is never recovered.

  17. Re:At this point, I just need to ask: by wwphx · · Score: 1

    I have in front of me of my recently arrived DVD complete Yes Minister, and I am reminded of an exchange between Sir Humphrey and Jim Hacker. To paraphrase:

    Sir Humphrey: There are two responses. "It is under consideration", which means we've lost it.
    Hacker: And the other?
    Humphrey: "It is under active consideration", which means we're looking for it.

    --
    When you sympathize with stupidity, you start thinking like an idiot.
  18. Ransomware by peawormsworth · · Score: 1

    ...plus Air Force cyber and defence cyber crime personnel

    I suspect ransomware was at play... and then they just paid it.

    why else bring in the cyber crime personnel... who will just say "pay the ransom"