Slashdot Mirror


Air Force Has Lost 100,000 Inspector General Records (thehill.com)

schwit1 shares an article from The Hill: The Air Force announced on Friday that it has lost thousands of records belonging to the service's inspector general due to a database crash. "We estimate we've lost information for 100,000 cases dating back to 2004," Air Force spokeswoman Ann Stefanek told The Hill in an email. "The database crashed and there is no data..." The database, called the Automated Case Tracking System (ACTS), holds all records related to IG complaints, investigations, appeals and Freedom of Information Act requests.... "We also use ACTS to track congressional/constituent inquiries."
The Air Force said they were "aggressively" trying to recover the data, adding that they had no evidence of malicious intent.

63 of 116 comments (clear)

  1. Simple: Restore from your backup by gavron · · Score: 4, Insightful

    You... do... have a backup, ... right?

    E

    1. Re:Simple: Restore from your backup by Anonymous Coward · · Score: 1

      "Real men don't do backups - but they often cry."

    2. Re:Simple: Restore from your backup by Anonymous Coward · · Score: 5, Funny

      Ask Israel or PRC or Russia, they probably have a copy.

    3. Re:Simple: Restore from your backup by khasim · · Score: 1

      That's my problem with this story.

      It's 2016. We know how to make backups. And databases compress nicely so the backup won't take anywhere near as much space as the original.

      We'll see ... but I'm willing to bet that there won't be ANY higher officers fired for this. Even though it means that some IG investigations/reports are now lost. Unless that is a feature that they wanted.

    4. Re:Simple: Restore from your backup by macs4all · · Score: 5, Informative

      You... do... have a backup, ... right?

      E

      Exactly what I came here to say.

      At this point, people employed by a Government Agency claiming "Lost Data" that cannot be restored from a Backup should be CRIMINALLY culpable for Obstruction of Justice and Breach of the Public Trust. Period.

      It's 2016, FFS, there is ABSOLUTELY no excuse for not having Backups. Especially not with the Goddamn AIR FORCE'S Budget!!!

    5. Re:Simple: Restore from your backup by Anonymous Coward · · Score: 1, Informative

      You did read (the very short ) article right?

      "The Air Force said it was notified on June 6 by a contractor that administers the database of records that the data within was "corrupted," according to a statement."

      Since the military thinks all their data and networks are safe in the hands of contractors, they'll probably just fire one contractor, put the blame all on them and hire another contractor.

      I really don't understand all the military confidence in contractors. IMO they use contractors far too often. They've also eliminated military personnel positions regarding military network administration and replaced with contractors.

    6. Re:Simple: Restore from your backup by JustAnotherOldGuy · · Score: 1
      --
      Just cruising through this digital world at 33 1/3 rpm...
    7. Re:Simple: Restore from your backup by ShanghaiBill · · Score: 2

      It is the job of the Inspector General to ensure that proper procedures and best practices are followed. I would say it was ironic that the IG office itself is the one with no backups, but then someone would say "That's not what 'ironic' means".

    8. Re: Simple: Restore from your backup by JcMorin · · Score: 1

      agree even if you don't have some automated backup I can't believe the machine was not backup once in a while in 12 years!!!

    9. Re:Simple: Restore from your backup by BoRegardless · · Score: 2

      Hey, Mr. General in charge of ACTS; no backup, no job, no pension ...

    10. Re:Simple: Restore from your backup by mrbester · · Score: 1

      There's more than five types of irony. The irony is most don't understand even the most commonly used type.

      --
      "Wait. Something's happening. It's opening up! My God, it's full of apricots!"
    11. Re: Simple: Restore from your backup by Anonymous Coward · · Score: 1

      The DoD has a severe problem of not recognizing and retaining real talent. In the last forced riff, they randomly deleted jobs. In fact, most of their knowledge level of leadership in IT is extremely low). It surprises me not one bit that they don't have backups, they probably fired a key person that maintained them, replaced by someone who does three jobs and not well, and never realized the difference.

    12. Re: Simple: Restore from your backup by jader3rd · · Score: 1

      Well the SAN vendor sells you on the superior redundancy of the SAN. It's so superior that there are no need for backups. Plus with disks as big as they are now, you can create databases so big that even incremental backups are taking more than 24 hours. What the SAN vendor won't tell you is that a SAN is a single point of failure in the system. When it decides to corrupt the data, there goes your data.

    13. Re:Simple: Restore from your backup by ShanghaiBill · · Score: 3, Interesting

      The reason why contractors are used is simple... Campaign contributions.

      Campaign contributions are only part of the problem. Another big factor is revolving door jobs. Most senior military officers serve for 30 years, from their early 20s to their early 50s. Then they "retire" on 75% pay, and are ready to move on to a civilian career. If they steer deals to the contractor during their service, there is often a wink-wink deal that they will get a job offer upon retirement. Then they can use their connections and contacts to work out similar deals with their former subordinates who have now moved up in rank.

      I used to work for a defence contractor, and all the senior executives were former generals or colonels.

      Of course, you can avoid all the expense of hiring they officers, by instead just providing them with hookers and cigars.

    14. Re:Simple: Restore from your backup by Anonymous Coward · · Score: 1

      "I'm sorry, sir, we have no record of you earning a pension. I'm afraid those records were lost and there were no backups. If only someone had done a better job..."

    15. Re: Simple: Restore from your backup by Anonymous Coward · · Score: 1

      The linked article does not state there are no backups. It does not provide data to assume that this is anything more than a temporary access issue which is being investigated. Keep in mind that this is the military. Investigative reports are a matter of course for all deviating events affecting efficiency (whether catastrophic or circumstantial) in order to prevent or mitigate occurrences which could adversely affect combat effectiveness. (IE, Incident reports are SOP, as opposed to the civilian world where they often imply identification of what parties to blame or punish).

      Note that the journalist selectively quotes a spokesperson and does not address key information... causing readers to make the assumption that there are no backups. The database being non accessible would plainly affect open investigations and research, and that is the reason why this issue was important enough to be reported via media. That the journalist chose not directly address data retention is rather curious.

    16. Re:Simple: Restore from your backup by Anonymous Coward · · Score: 2, Interesting

      You... do... have a backup, ... right?

      E

      Sadly, I can attest to this being frighteningly common. I work in the data storage industry (on the software side) and some of our customers don't have proper backups. I don't want to think of all the times we've had a customer escalation that was almost exactly like this. The customers frequently expected our support staff to be able to wave a magic wand and get their data back. It's really impressive how incompetent some system and storage administrators can be. And incidentally, only one of the ones I remember was a government account. Most of them were medium or large sized companies that just didn't have good people running the place. Although, the one that was government was the most spectacular one. They ended up paying a big consulting team to actually go back in and reconstruct most of the data by hand from the bits and pieces left intact on the storage. It was also the one that was basically admitted to have been intentional sabotage.

    17. Re:Simple: Restore from your backup by Anonymous Coward · · Score: 1

      Ask Israel or PRC or Russia, they probably have a copy.

      Why so complicated man ? Just ask the NSA, I'm sure they'd be happy to oblige.

    18. Re:Simple: Restore from your backup by AK+Marc · · Score: 1

      They had enough budget to buy planes, but were likely not allowed to buy a backup, they were required to use a Congressman's son's IT services company to do it, which is why it didn't happen. They don't have the power to be competent. The government has outlawed government competency. But that's what you get when one of the major parties campaigns on a "hate the government" platform.

    19. Re: Simple: Restore from your backup by Gr8Apes · · Score: 2

      Once again, if you can't detach and take that "backup" with you (even if you need a forklift and an 18 wheeler or three) and plug it into a different system and view the data, it's not a backup.

      --
      The cesspool just got a check and balance.
    20. Re:Simple: Restore from your backup by DraconPern · · Score: 1

      Bet they are in the same situation as I am. I have a back up. The backup restores correctly, all the data is there. Except the application can't connect to the db because it is trying to login with a hard coded password. And the database is on mssql 7. The company that made the software has long been dead. We still use the application, but trying to upgrade it to sql 20xx has been a challenge.

    21. Re:Simple: Restore from your backup by hackwrench · · Score: 1

      Yeah, ""talent". Just another word for "magic". Can't say for certain, but I suspect that most people have what it takes for the military's "advanced training" to get them the rest of the way, but then who trains the trainers?

    22. Re:Simple: Restore from your backup by ExXter · · Score: 1

      If not then you know thatit was deleted (crashed*) in the first place. Something was in there that should not be touched/uncovered and or made public. But lets stay with the truth. America is fast with features but not safety. The word backup is not from America its something that is despised. Sorry being nonconstructive.

    23. Re:Simple: Restore from your backup by cusco · · Score: 1

      Wonder how much of the lost data pertained to investigations of the DB contractor . . .

      --
      "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
    24. Re:Simple: Restore from your backup by cusco · · Score: 1

      The best way to prove to your constituents that government doesn't work is to break it yourself.

      --
      "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
    25. Re:Simple: Restore from your backup by davester666 · · Score: 1

      We're throwing them into a fire as fast as we can in our attempt to recover the data.

      --
      Sleep your way to a whiter smile...date a dentist!
    26. Re: Simple: Restore from your backup by Anonymous Coward · · Score: 1

      The NSA, they have a copy but can't find it anymore...

    27. Re:Simple: Restore from your backup by currently_awake · · Score: 1

      Failure to keep backups of official government records should be considered negligence. Proper SOP should be a monthly full backup with daily incrementals..

    28. Re:Simple: Restore from your backup by HornWumpus · · Score: 1

      You don't ground a showerhead.

      What incompetent electricians will do is use the cold water pipe as a ground. Still more incompetent electricians will attach a hot wire to the the cold water pipe (that isn't even grounded because it's attached to plastic pipe).

      --
      John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
    29. Re:Simple: Restore from your backup by HornWumpus · · Score: 1

      Why would you have to? It's plenty broken, mostly by the do gooders trying to make it do stupid things.

      --
      John McAfee 'It was like that time I hired that Bangkok prostitute; to do my taxes, while I fucked my accountant'
    30. Re:Simple: Restore from your backup by BlueCoder · · Score: 1

      Contractors are used because they often fall outside regulations. Requirements are stipulated as part of their contract which is physical and the size of a phone book. This is also how the executive branch of government currently gets around laws. And intentionally lose information. And lie.

      A contractor can be fired. He can then go bankrupt and or reform under a new business name and then be rehired. We don't currently hold the government responsible for the actions of it's contractors. There is no buck stops here. By definition the contractor can only be held liable for what's in the contract and it's vary easy for those contracts to be vague. The governments can always point clauses to the public and say see this clause... but the contractors know what is legally enforceable. And again the contractors can more or less at any time jump ship or be fired and then rehired with a new name and contract. Win Win for the executive branch. The contractor is happy so long as it and it's employees get paid very well.

      For example Snowden was an employee of a Dell. Then he became part of the NSA contractor, Booz Allen Hamilton.

      Real spies don't work directly for the government. Plausible deniability.

  2. This calls for a Congressional Inquiry! by mykepredko · · Score: 1

    I wonder where they'll store the request?

  3. Re:Lucky bastards! by Tailhook · · Score: 4, Interesting

    Yep. Something was in those records someone wanted disappeared. This is SOP in government now; systematically destroying disk drives, deleting PST file content, wiping servers.... just another cover-up.

    --
    Maw! Fire up the karma burner!
  4. Re: Did Hillary Wipe It? by Anonymous Coward · · Score: 1

    That database had it coming.

  5. Obama's Razor by Rick+Zeman · · Score: 4, Funny

    Never attribute to happenstance which can be attributed to a cover up.

  6. Re:Whose airforce? by Rick+Zeman · · Score: 3, Funny

    Let's see, it's a US Capitol Hill newspaper, and the article shows US airmen and a US transport in the photo. So, obviously, it's about the Kenyan Air Force.

  7. Yes Minister said it best by whoever57 · · Score: 4, Funny

    James Hacker: Was 1967 a particularly bad winter?
    Sir Humphrey Appleby: No, a marvellous winter. We lost no end of embarrassing files.

    --
    The real "Libtards" are the Libertarians!
    1. Re:Yes Minister said it best by Anonymous Coward · · Score: 1

      or this from the same show:

      Bernard: “I’ll just say, ‘The Minister has asked me to thank you for your letter’ and something like ‘The matter is under consideration’, or even ‘under active consideration’.”

      Hacker: “What’s the difference?”

      Bernard: “Well, ‘under consideration’ means we’ve lost the file, ‘under active consideration’ means we’re trying to find it.”

  8. Re:"aggressively" by war4peace · · Score: 1

    Slam the HDDs with a big hammer until data confesses...

    --
    ...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
  9. Newspeak by qbast · · Score: 2

    What is aggresive recovery? Waterboarding the hard disks or straight to rubber hoses?

    1. Re:Newspeak by Tablizer · · Score: 1

      Don't give Cheney any new ideas.

  10. They have to be in violation of something by meerling · · Score: 1

    Let's mention the first 3 laws of computing:
      1 - Backup
      2 - BACKUP
      3 - See Rules 1 & 2

    On top of that, when I was in the USAF working in the missile shop, we had FIVE copies of all the records, and they were stored in different places. Losing the records to a missile would mean your ASS! More than one would crucify the entire shop!
    And now someone is trying to complain that the Inspector General has lost a huge amount of records because of a single database crash?
    Is someone incompetent running the Inspector Generals Office?
    Do they freaking need some of us old time computer geek veterans to come over there and show them how NOT to be a total embarrassment?!

    1. Re:They have to be in violation of something by Rick+Zeman · · Score: 4, Insightful

      Let's mention the first 3 laws of computing:

        1 - Backup

        2 - BACKUP

        3 - See Rules 1 & 2
      ]

      1 - Backup
      2 - BACKUP
      3 - Test restoration of backup

      There, fixed it for you.

    2. Re:They have to be in violation of something by cusco · · Score: 1

      Is someone incompetent running the Inspector Generals Office?

      They contract out the IT work, so yeah.

      --
      "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
  11. Obviously no malicious intent by houghi · · Score: 1

    If what you did had the results you where going for it is not called "malicious intent". Oh, you mean in general, yeah, well, awkward.

    --
    Don't fight for your country, if your country does not fight for you.
  12. Re:No need for RAID, Distributed DBs or Backups by hawguy · · Score: 4, Informative

    No need for that fancy stuff since it wasn't mission critical data
    Must have been designed by Carnegie Mellon grads

    RAID and distributed DB's are for HA, they are not a substitute for backups. Neither RAID nor a distributed DB will protect against corruption or accidental data loss - if someone deletes the wrong records, they'll be gone from both the primary and secondary database.

    Any many people still think RAID-5 gives adequate protection against drive loss, which is no longer the case with modern large hard drives.

  13. Obama does what he wants. Clinton cover-up since by raymorris · · Score: 2

    I don't know about that; Obama has been pretty open about wiping his ass with the Constitution.

    Contrast Hilary Clinton, who has been involved in government since 1977 and her primary responsibility for 38 years has been cover ups and white-washing.

    Trump is another who is pretty open about doing and saying things that people don't like.

  14. THIS. Most untested backups don't work by raymorris · · Score: 3, Insightful

    You hit the nail on the head. I've probably encountered more broken backups than ones that work. Web hosting providers frequently provide backups that stopped working 10 months ago, but nobody noticed. If you haven't recently tested restoring your backups, you probably have no backups.

    I like to use remote backups that I can restore from conveniently, so that I restore a file from time to time just because I messed up a couple paragraphs of text or something. These real-life, low-impact restores serve to verify backup and restore is working properly.

  15. Re:Trump 2016 by mrbester · · Score: 1

    Blame the BBC for that one

    --
    "Wait. Something's happening. It's opening up! My God, it's full of apricots!"
  16. Bobby Tables by Macdude · · Score: 4, Funny

    Sound like the Air Force may have added Booby Tables to the Inspector Generals Records.

    https://xkcd.com/327/

    --
    "Grab them by the pussy" -- President of the United States of America
  17. Re:Lucky bastards! by ninthbit · · Score: 2

    So is gross negligence. I can easily believe these idiots only maintained a years worth of backups and that there was no process in place to verify data integrity over time. It's a very common problem with government systems. They backup corrupted data and eventually no longer have a good version to record from. This of course is when there actually is a backup to begin with. Often a productive underling will develop a system to make his job easier and it will grow into a poorly supported critical system. It's even worse in the DOD where the staff rotate every 3-5 years.

  18. Mission Accomplished! by rsborg · · Score: 1

    We'll see ... but I'm willing to bet that there won't be ANY higher officers fired for this. Even though it means that some IG investigations/reports are now lost. Unless that is a feature that they wanted.

    Money quote. Corruption is the feature, oversight/inspections are the bugs. Now you're thinking like upper brass and/or contractors.

    --
    Make sure everyone's vote counts: Verified Voting
  19. Fire contractor, announce name by myid · · Score: 4, Insightful

    From the article:

    The Air Force said it was notified on June 6 by a contractor that administers the database of records that the data within was "corrupted," according to a statement.

    How many contractors administered the database? I wonder if that was part of the problem: "Oh, I thought you guys were going to back up the database ... No you were supposed to back it up."

    If just one contractor was clearly responsible for the backup, then I wish the government would:

    1) Fire the contractor, and never use them again.

    2) Publicly announce the name of the contractor, so that we know not to use them.

    (Of course a lot more needs to be done, such as making sure this doesn't happen again in any govt. dept.)

    1. Re:Fire contractor, announce name by evilviper · · Score: 1

      2) Publicly announce the name of the contractor, so that we know not to use them.

      The contractor was Oracle... Be sure to NEVER use their products. Good luck.

      --
      Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
  20. Lost or purposely deleted? by shubus · · Score: 1

    Delete on purpose to hide IG findings or are Hillary's server guys moonlighting?

  21. Re:No need for RAID, Distributed DBs or Backups by aaarrrgggh · · Score: 1

    Honest question though: are distributed systems plus snapshots a "backup?" What really makes a backup-- beyond ensuring no common mode failure option.

  22. Just let it go by h33t+l4x0r · · Score: 1

    So you lost all your data and didn't have a backup? It's happened to all of us and somehow, life goes on.

    At least you didn't spend the past 10 years implementing, maintaining, and testing a foolproof backup system, because that sounds like a real drag.

  23. Cross-reference former assistant IG at Pentagon by mcleland · · Score: 1

    Take a look at this: http://www.thenation.com/article/the-national-security-expose-so-secret-even-edward-snowden-didnt-know-about-it/ where a former assistant Inspector General for the Pentagon claims whistleblowers were treated illegally. Neither the parent article nor that linked one inspire confidence in any DoD related Inspector General office.

  24. Ah, I remember stage 1 by raymorris · · Score: 1

    I remember when I said stuff like that. You're pissed that parties are partisan. So cute. At this stage, you're still rooting for the team you picked, thinking that they aren't just as corrupt, playing you like a piano. Clinton has your name on a list of people like you. At this stage, you don't quite know the politicians' names (Chaney and Regan lol), but you're sure that the politicians on the other team are evil, while the politicians on the team you chose are good. You don't know the names of the Congressional commitees or agencies, or treaties, but you can almost get them half right as you parrot the propaganda you heard on last night's comedy show.

    Later, you'll learn the names and start to understand a little bit about what they do. You'll be able to parrot the party propaganda and actually get it right. Hillary will move your name to another list.

    Assuming you have an IQ above 87, after that you'll eventually realize what you've been regurgitating is silly propaganda, that Clinton's job since 1977 has been to figure out which lies stage 1 and stage 2 people will fall for.

  25. Re:Lucky bastards! by cusco · · Score: 1

    That's a point generally forgotten when setting up backup systems; do test restores frequently to verify that they work and that you're still backing up the data you need. Nothing like having supposedly successful error-free backups for months, then when you need to restore something finding that all the tapes are blank (thanks Backup Exec!)

    --
    "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
  26. Re:No need for RAID, Distributed DBs or Backups by cusco · · Score: 1

    Better than the Compaq tech which removed the GOOD drive, degaussed it, then couldn't figure out why the RAID wouldn't rebuild.

    --
    "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
  27. Re:No need for RAID, Distributed DBs or Backups by Blaskowicz · · Score: 1

    You should have used triple drive RAID 1 then. The probability of deleting everything is less than 50%.

  28. Files by Winkkin · · Score: 1

    figures