Slashdot Mirror


So You've Lost a $38 Billion File

smooth wombat writes "Imagine you're reformatting a hard drive so you can do a clean install but then realize that you have also reformatted the back up hard drive. No problem. You reach for your back up tapes only to find out that the information on the tapes is unreadable. Now imagine the information that is lost was worth $38 billion. This scenario is apparently what happened in July to the Alaska Department of Revenue. From the article: 'Nine months worth of information concerning the yearly payout from the Alaska Permanent Fund was gone: some 800,000 electronic images that had been painstakingly scanned into the system months earlier, the 2006 paper applications that people had either mailed in or filed over the counter, and supporting documentation such as birth certificates and proof of residence.' Using the 300 cardboard boxes containing all the information, staff worked overtime for several months to rescan everything at an additional cost of $200,000."

15 of 511 comments (clear)

  1. Redo the work? by Anonymous Coward · · Score: 5, Insightful

    For that kind of money, I'd probably just send the HD to data recovery specialists.

    1. Re:Redo the work? by omeomi · · Score: 4, Insightful

      For that kind of money, I'd probably just send the HD to data recovery specialists.

      Well, this is the government. They probably didn't have a budget for data recovery, but they did have a budget for scanning documents...the actual dollar amounts of each probably matter very little ;-)

  2. $38 billion? by pipatron · · Score: 4, Insightful

    How did they figure these files were worth $38 billion when it only cost $200000 to create them from scratch?

    --
    c++; /* this makes c bigger but returns the old value */
    1. Re:$38 billion? by __aapbzv4610 · · Score: 4, Insightful

      The value is based on what the data represents, not the material labor in re-scanning them. loose example, I could spen $30 on painting supplies and create a $100,000 masterpiece (well not by me, but, anyway...)

  3. Re:Tapes? by teflaime · · Score: 3, Insightful

    Most ENTERPRISES still have tape at some level as part of a comprehensive disaster recovery plan. Tape is easy to offsite, fairly reliable overall and still have comprehensive support available in all platforms. Most INDIVIDUALS don't do backups at all.

  4. Backups are the devil by Maxo-Texas · · Score: 5, Insightful

    Because no one ever restores them regularly to test them.

    I was at a company years ago and argued for both a ton more backups than they were making and for a test restore. They were not in the mood to do either. After about nine months, for some unknown reason they had to restore a file.

    And the backup tape was unreadable. The next good backup was 17 days older.

    After that we got $30 bucks of backup tapes every week and we had a 7 day rotation with the 7th day going in the vault. And we did regular test restores once a quarter.

    You should REGULARLY test your backups.
    You should have LOTS of backups.

    --
    She was like chocolate when she drank... semi-sweet at first and then increasingly bitter.
  5. Re:Tapes? by daeg · · Score: 4, Insightful

    Hm. Tapes with a proven shelf life of many, many years, or DVDs where a single scratch can render 4GB of data worthless. I wonder which enterprises (or governments) should chose?

  6. Damn! by Penguinisto · · Score: 4, Insightful
    That has got to rank up there w/ the all-time worst 'oh-shit' moments in that poor bastard's career (if it still exists). I wonder what the sysadmin was thinking, storing data on the same partition as the OS. No sane production environment rig that I know of would (or at least should) have that. It may be a Windows thing, but on most servers I've dinked with, the OS sat on a pair of RAID disks by itself, and all the data sat on the monster pile of disks on their own logical RAID drive (at least RAID 5... 5+0 w/ a hot spare preferred).

    That, or you'd think they'd at least have that kind of stuff stored on more than one server if it were that valuable?

    /P

    --
    Quo usque tandem abutere, Nimbus, patientia nostra?
  7. Re:Tapes? by Red+Flayer · · Score: 4, Insightful

    Well, the summary states that the files were rescanned at a cost of $200,000 -- so it sure sounds like the hard copies were preserved.

    --
    "Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
  8. Re:Tapes? by Itninja · · Score: 4, Insightful

    Easy to offsite???

    I work for an IT organization and we pay a company called Iron Mountain $100's monthly to schlep our boxes and boxes of backup tapes to their offsite storage facility.

    And remember there is a difference between making 'backups' (store my important files somewhere else so I can get them in case of a system failure) and preparing for 'disaster recovery' (store everyones files somewhere else so we can rebuild the entire infrastructure in case the building burns to the ground).

    --
    I judt got a nre Kinesis keybiartf so please excusr ant egregiou typos.
  9. Re:I could be a douche and say it has never happen by Overzeetop · · Score: 4, Insightful

    What happens when you fuck up that big? If you the CEO of a major corporation, you get a severance check that looks like a phone number. If you're the IT plebe, you get to look for a new job.
    --
    Is it just my observation, or are there way too many stupid people in the world?
  10. Re:Data recovery? by mwvdlee · · Score: 4, Insightful

    I'm just assuming the harddisks were secure erased, considering that is what pretty much every govenment in the world does when formatting harddisks.

    Simply put, secure erasing is a process whereby (semi-)random data is written to the harddisk, overwriting previous data, and doing it enough times to ensure no residual traces of data exists.

    --
    Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
  11. Someone is trying to cover their ass by Unique2 · · Score: 5, Insightful

    Primary disk: Accidently deleted.
    Backup disk: Accidently formatted.
    Tape: Unreadable.

    What about the other tapes in the cycle? Did you not test it before? What about data recovery on the hard disks?

    Thats a lot of unfortunate co-incidents and a lot of questions. It sounds more like the reality is that none of these ever existed and someone got caught-out.

    --
    No trees were harmed in the posting of this message. However, a great number of electrons were terribly inconvenienced.
  12. DVDs are a joke. by mosch · · Score: 4, Insightful

    An LTO Ultrium 3 tape holds 400GB uncompressed, and you can buy libraries that hold hundreds or thousands of these tapes (and dozens of drives).

    Disc to disc backup is gaining acceptance for some applications, but there are other places where the massive storage capacity of tape just can't be beat.

    The idea of DVD as a business-class backup medium is almost perfectly slashdottastic.

  13. Re:Time for... by jadavis · · Score: 5, Insightful

    It was actually only worth $200k, since that's the amount of money it took to recover from the problem.

    The fact that it was related to an account worth $38B is scary, but not the actual cost.

    --
    Social scientists are inspired by theories; scientists are humbled by facts.