Slashdot Mirror


Tech Magazine Loses June Issue, No Backup

Gareth writes "Business 2.0, a magazine published by Time, has been warning their readers against the hazards of not taking backups of computer files. So much so that in an article published by them in 2003, they 'likened backups to flossing — everyone knows it's important, but few devote enough thought or energy to it.' Last week, Business 2.0 got caught forgetting to floss as the magazine's editorial system crashed, wiping out all the work that had been done for its June issue. The backup server failed to back up."

34 of 245 comments (clear)

  1. After the swearing stopped. by AltGrendel · · Score: 5, Funny
    The first words from management were "You're kidding me, right?"

    Then the swearing started again.

    --
    The simple truth is that interstellar distances will not fit into the human imagination

    - Douglas Adams

    1. Re:After the swearing stopped. by Anonymous Coward · · Score: 5, Insightful

      Actually that is what you get for having Geek Squad as your outsourced IT staff.

      honestly, they CANT have competent IT. The FIRST thing you do in the morning is check the backups.

      I have a HP sdat jukebox here and I STILL check the backup logs to make sure the backup and verify succeeded last night. if they dont I mirror the important files right away and then run a manual backup to not lose the last 24 hours of backup.

      I hope that Business 2.0 learned that paying top $$$ for competent IT is a good idea and they should run a article about it.

    2. Re:After the swearing stopped. by IdleTime · · Score: 4, Insightful

      I am not surprised.

      There is not a week going by without me getting an issue from one of out regular analysts with question about how the customer can salvage their data because they don't have a backup. My standard answer is that we may be able to save some data, but it's going to cost a lot of $$$. And I also say: "When you don't have a backup, you have either deemed that you can easily recreate the data or that they are not important for the company"

      And these are not mom&pop companies but big multi million/billion dollar companies.

      --
      If you mod me down, I *will* introduce you to my sister!
    3. Re:After the swearing stopped. by Auntie+Virus · · Score: 4, Insightful

      I have a HP sdat jukebox here and I STILL check the backup logs
      HP DAT? You'd better do more than check the logs. A test restore (if your users don't already test for you by deleting files) at least a few times a week might save your butt one day. Actually DAT or not, test restores are a must. Logs lie.

      --
      Why yes, I *AM* new here. Why?
    4. Re:After the swearing stopped. by loafing_oaf · · Score: 5, Informative

      The problem is that tech magazines are in the advertising business, not the tech business. I write content for the Web site of a tech radio show, and it's just a bunch of us in cubicles looking stuff up on Google. No tech people involved.

      --
      Always someone has power over you. The thing to consider is this: Is the power good, or bad?
    5. Re:After the swearing stopped. by Sandbags · · Score: 3, Insightful

      I work for a backup company that makes D2D backup appliances supporting more than 20 operating systems.

      First, no one really understands best practices for backup, and a lot of systems that are backed up "successfully" can't be restored anyway (in fact, most commonly this is Microsoft Exchange, the most important system in most companies!). Second, Tape sucks! You MUST have Disk-to-disk backups to have any true recoverability in today's world. Third, check you logs EVERY day, there's no excuse! Fixing a failing backup should be the number 1 priority second only to an actual failed server you are recovering. Next, nobody spends enough on IT disaster recovery, and no one documents the recovery process properly. Your IT spending on DR should be approximately 25% or more of your total IT budget for server systems. At least 1 day per month should be used to practice system recovery or update the documentation covering it. Next, nothing should ever be considered backed up until the server has been test recovered, completely from scratch, at least once. At least some data should be recovered from backup media every day just to be certain it can be done when needed. The test recovery should be of a random critical data folder, or database, not the same stuff each time.

      Off-site DR is also important. Making sure that your entire data set for all critical systems is moved off site every 24 hours is a must. Included in this should be any media required to process a restore (not just the backups, but the install CDs, BareMetal recovery disks, licenses keys for all servers and applications, the DR documentation itself, network architecture information, hardware and software configuration of each server, and all information regarding your ISP contract, and system warranties from each manufacturer. If you don't have all this stuff, contract someone who knows what they are doing to make it for you.

      For each unique mission critical system you have (Mail, critical database server that allow the business to operate, point application server, Citrix box, etc) you should have a complete spare system meeting the system requirements so that system can be restored immediately in the event of a system outage. Your system recovery tests should be performed regularly to that hardware. Best practice is also to keep those test boxes off-site when possible, but nearby enough to get in a jiffy. If you don't have spare lab equipment, and don't have enough budget to have it, you can't afford to have those critical systems in house, and should consider outsourcing a data center who does have those resources. Clustering is complicated and expensive, but spare chassis and a few spare drives don't amount to a huge IT burden. You don't have to have 1 for each server, just one that can handle the job of each unique mission critical system (if you have 5 SQL servers, 1 exchange, 1 citrix, and 4 file servers, you only need 4 total spare system).

      The average business that goes through a critical system disaster that interrupts business for more than 48 hours requires 1 month of revenue to overcome the loss of each day of downtime. 40% of businesses that have a site disaster lasting more than 3 days go bankrupt within 90 days of the event. How much money will your business loose if you have to roll your purchase database back 2 days and loose all records of those transactions? How will your business survive if e-mail is out for 3 days? How much will you loose if your online store is gone for several days? How many customers will you loose if your support department is off-line for 2 days? How much will you be sued for if you miss a contractual deadline due to data loss? Can you afford to NOT spend the money to make sure this doesn't happen!?!?!

      --
      There is no contest in life for which the unprepared have the advantage.
  2. With this much free advertising by Anonymous Coward · · Score: 5, Funny

    who needs a magazine?

  3. Nelson Muntz by erroneous · · Score: 5, Funny

    Some stories should just come with Nelson Muntz sound files embedded.

    Ha-ha!

    --
    erroneous: look me up in a dictionary
  4. err... by cosmocain · · Score: 3, Insightful
    HAHA!

    *coughs

    TFA:

    Business 2.0 never had to rely on their backup software until that day, which is why they probably did not realize that it was either obsolete or dysfunctional.

    sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?
    1. Re:err... by Lumpy · · Score: 5, Informative

      hell with that. ever heard of competent IT staff? why has their CTO not been fired yet?

      honestly though, talking management into backup solutions is like pulling teeth, then they blame you for not having it in place when the failure does happen.

      Last place I worked at we were using 4 year old DLT tapes because management was too stupid and cheap to buy new ones.

      "we will buy new when those fail" is what we were told.

      --
      Do not look at laser with remaining good eye.
    2. Re:err... by morgan_greywolf · · Score: 4, Funny

      ever heard of verifying backuped data?


      Errr...uhh....umm...'verifying'? Uh, I'll be right back!

    3. Re:err... by dal20402 · · Score: 3, Informative

      /grabs hammer...

      *bang* *bang* *bang*

      Oops, it looks like a couple of those DLT drives are running into problems. We need replacements. Did you see what happened to Business 2.0?

    4. Re:err... by radtea · · Score: 5, Funny

      sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

      I'm sure they've heard of it, in a conversation that went something like this:

      IT Guy: We need a system for verifying our backups.

      Suit: How come? Don't the backups work?

      IT Guy: We need to be sure that if there is a failure, the backups will be ok.

      Suit: But they're just copies, aren't they? I copy files all the time and it never goes wrong.

      IT Guy: This is a little more complicated than that.

      Suit: How hard can it be?

      IT Guy: Well, I was thinking we might need to hire a part-timer just to take care of backups and verification.

      Suit: But we've never had a failure! Sounds like empire building to me. I know that's what I'd be doing in your position. Nice try. We'll keep the backup system the way it is, thanks.

      IT Guy: But..!

      Suit: Moving on to the next item on the agenda... ok, Executive Bonuses!

      --
      Blasphemy is a human right. Blasphemophobia kills.
    5. Re:err... by Speare · · Score: 3, Insightful

      "we will buy new when those fail" is what we were told

      "Your successor will buy new when these fail." is the correct response to this.

      --
      [ .sig file not found ]
  5. They probably still have most of it by ZachPruckowski · · Score: 4, Insightful

    I imagine that they still can resemble a lot of it from other files - they should still have all the layout pieces for one, and all the authors ought to have at least rough drafts of their stories on their personal computers. The deadline's screwed, but they can probably get it out a few weeks late (or in July, depending on how often they normally publish).

    1. Re:They probably still have most of it by Red+Flayer · · Score: 3, Funny

      Yeah, great, that's the content - now how about the advertising? That's where they make their money.
      Editorial department server content was lost. Advertising content is normally handled by the production department.

      I think we can all relax and rest assured that the June issue of Business 2.0 will have all its intended advertising.
      --
      "Trolls they were, but filled with the evil will of their master: a fell race..." -- J.R.R. Tolkien on Olog-hai
  6. High profile SNAFUs by Rob+T+Firefly · · Score: 5, Insightful

    This reminds me of the recent uproar over a car crash involving the New Jersey governor. He was critically injured because he wasn't wearing his seatbelt, and people freaked, asking what sort of role model he could possibly be. I argued that he was an awesome role model, because sometimes people need to see a mistake end badly for someone else before they'll do what's necessary to protect themselves from making the same mistake. Seeing a high-profile magazine get hit like this can do the same for backup slackers the world over.

    I don't know about you people, but after reading this (and giving it the "haha" tag) I'm going home and catching up on a couple of backups I've been slacking off on for a while.

    1. Re:High profile SNAFUs by Hoi+Polloi · · Score: 5, Insightful

      I wouldn't use the term "role model" for things like that. I'd say "examples" is the better word. The governor was an example of what NOT to do.

      --
      It is by the juice of the coffee bean that thoughts acquire speed, the teeth acquire stains. The stains become a warning
  7. Re:What was the nature of the crash? by Chris+whatever · · Score: 3, Insightful

    Hum!!! Unless you are there looking at the data being backed up there is no way unless you get notification from your system that it has completed.

    usually that is the case but it has happened when one of my backup failed one night and someone needed a file restores from the previous day, if that company never checked it's backup or never configure some kind of noticaition upon failiure or success then they are very lame

  8. How does this actually happen? by Orange+Crush · · Score: 4, Interesting

    There aren't a lot of ways for a machine to "crash" that loses all its data. Even a lightning-fried hard drive can have its platters removed by a data recovery lab and many files can be pulled off. A mechanical failure doesn't grind the platters into sand. As a network server it really should have a RAID too. So how exactly can "the server crash" so spectacularly that the RAID, backups, and widely available data recovery services all fail? Did the building blow up?

    1. Re:How does this actually happen? by Rob+T+Firefly · · Score: 3, Insightful

      IANA publisher, but I would also imagine that in such a deadline-intensive business, data from a fried disk is about as good as lost. Sure, they can send their drives off to data recovery labs who could slowly recover an uncertain portion of the data for a pantload of money, but by the time that's done it'll be time for the next issue anyway. I'd guess it would be a lot quicker and cheaper to write off the disks and salvage what they can from everyone's local copies of the data.

    2. Re:How does this actually happen? by Paulrothrock · · Score: 4, Informative

      A mechanical failure doesn't grind the platters into sand.

      Doesn't it?

      --
      I'm in the hole of the broadband donut.
  9. At this exact moment across the world by Timberwolf0122 · · Score: 3, Funny

    Tens of IT managers are getting Hundreds of IT minions to check Thousands of backup tapes and befor a senior manager walks in.

    --
    In the not too distant future, next Sunday A.D.
  10. Wrong problem by mseeger · · Score: 5, Insightful
    Hi,

    the problem was, as always, not the backup. I've rarely seen problems resulting from the backup process. The troublesome process is the restore. Or as a friend put it once:

    Nobody wants backups, what everybody wants is a restore.

    In my twenty years of IT i've seen several companies making backups like a well oiled machine. The backup process was well documented and everyone was trained to a degree, they could do it with their eyes closed. But everything fell apart in the critical moment, because all they had planned was making the backup. Nobody ever imagined or tried a restore on the grand scale. So they ended up with a big stack of tapes with unuseable data.

    Backup is the mean, not the goal.

    Regards, Martin

    1. Re:Wrong problem by RetroGeek · · Score: 5, Interesting

      The troublesome process is the restore.

      I heard a story about a LAN admin who was doing backups every night. The tapes would go into a safe, then would go offsite, then be used again.

      Everything worked well(?) until they needed to do a restore. The tape in the safe was corrupt. The tape at the offsite storage was corrupt. No tape was good.

      It seems that the LAN admin made tea every morning. The electric kettle sat on top of the steel safe.

      So the backup tape was placed into the safe, then the kettle was started, magnetizing the safe, and erasing the tape.

      Not ONCE did anyone try to do a test restore to prove the system.
      --

      - - - - - - - - - - -
      I am a programmer. I am paid to produce syntax not grammar. Deal with it.
    2. Re:Wrong problem by mseeger · · Score: 3, Interesting
      > Would mirrored drives be a more effective solution?

      Yes and No:

      • Mirrored drives are a good protection against drive failures and (usually) offer an easy restore process. If you mirror a drive and put the copy away (e.g. into a safe) this is a real and widely used backup method. As always you should at least try once to boot the system while removing the primary disk. Somtimes RAID controllers have some irks too.
      • This method usually depends on the availability of a certain hardware, if you cannot get a new mainboard or raid controller of the same type, the mirrored disk contains data you may have trouble getting at. You may ignore this issue, if you have the same hardware at a safe location again.
      Regards, Martin
    3. Re:Wrong problem by ByteSlicer · · Score: 3, Informative
      I highly doubt that the kettle could demagnetize the tape in the safe, due to the Farraday shielding. Even if the kettle was on top of the tape (outside the safe), the generated magnetic field would not be strong enough (although the heat would probably melt the tape).

      Nice story, though. Reminds me of the sysadmin in my first company who automatically back-upped our server every day. Only problem was: the proces put a copy of the backup on a drive that was being back-upped. You can imagine what happened after a few weeks (it failed, disk full). He only noticed a few months later when we asked him to restore some files.

  11. Re:Rag by UbuntuDupe · · Score: 5, Insightful

    nobody reads Business 2.0 anyway.

    I wish. I wish people didn't read Time, either (the publisher), but they do. Time's writing style is the dumbed down, try-to-be-hip crap I wouldn't have gotten away with in sixth grade. Seriously. Like I said before, to understand why its writing is like fingernails on a blackboard for me, consider how the same information would be conveyed by two sources:

    8-year-old: "6 divided by 3 is 2."

    Time magazine: "Okay, imagine you've got a half-dozen widgets, churned out of the ol' Widget Factory on Fifth and Main. Now, say you've gotta divvy 'em up into little chunklets -- a doable three, let's say -- and each chunklet has the same number that math professor Gregory Beckens at Overinflated Ego University calls a 'quotient'. The so-called 'quotient' in this case? Dos."

    Based on how that post got modded, I'm not alone in this.

  12. Re:Why isn't this a default by metamatic · · Score: 3, Informative

    Why isn't it a default for an OS to ask where the backups should go when it is installed?

    Wait for OS X 10.5 and "Time Machine".

    --
    GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
  13. We can't backup, its too expensive. by Stu101 · · Score: 5, Insightful

    This is my story and I bullshit you not! I work for a manufacturing company, the second largest in its field in the world. Great. However the boss really does not like spending money. We eventually got a backup system using offsite backups (with a special client) and it seems to work ok. However, when it got to 100 GBs I was told to start pruning stuff. So I did. Long and short of it, even with the most important files backed up, we still have most things not backed up. Basically I have almost half a TB of data that I am not allowed to back up because its expensive. I can only backup 5 days worth of data as they are unwilling to pay anymore money for it. The fun will come when someone wants a restore from last year. This people, is the reality sometimes. Me, well, I really dont care anymore. Im sick of having servers, important, mission critical machines sitting on single IDE disks. We sell online, great, problem is our firewall is non redundant single IDE disk. If it goes (like it has in the past) we were down for days, loosing emails, web traffic, web orders, remote ordering systems, EDI data, remote sessions, ftp, everything. DR? the solution proposed by upper management is, oh we will buy some dells and restore. Yeah thats a good idea. After waiting a week for them to arrive, what exactly are you going to restore ? This is more typical than you think, unfortunatly. Im just the guy that has to make do with what i can. No doubt when it fucks up I will be blamed.

    --
    http://www.writeitfor.us - Writing IT for the IT generation.
  14. Re:We've all been there. Don't be too pious, here. by LurkerXXX · · Score: 3, Informative

    And if the data is that important then a suitable RAIDed disk array will sort things out.

    The topic here is backups, not RAID.

    Say it again with me everyone "RAID IS NOT A BACKUP"

    RAID increases-uptime by decreasing/eliminating the downtimes needed to do restores when an individual drive bites it. It is *NOT* a backup.

    RAID does not save you if someone accidentally deletes a needed file.
    RAID does not save you if your machine gets nailed by a virus/upatched-exploit.
    RAID does not save you if the drive power supply fries taking out attached hardware.
    RAID does not save you if a bugler steals your machine.
    RAID IS NOT A BACKUP.

  15. Paging Jerry Seinfeld by shrubya · · Score: 5, Funny

    Jerry: I don't understand. Do you have my data?
    IT: We have your backup, we just can't restore it.
    Jerry: But the backup keeps the data here, that's why you have the backup!
    IT: I think I know why we have backups.
    Jerry: I don't think you do. You see, you know how to MAKE the backup, you just don't know how to RESTORE the backup. And that's really the most important part of the backup: the restoring. Anybody can just make them.

  16. Re:Didn't you read the article? by njchick · · Score: 3, Funny

    How about doing a restore practice run whilst at it?
    You mean, buckle up the governor and make another crash to test the seatbelt?
  17. Re:We've all been there. Don't be too pious, here. by Nick+Number · · Score: 4, Funny

    RAID does not save you if a bugler steals your machine.
    But can he carry my server and his horn at the same time?
    --
    Promote proofreading. Don't mod up sloppy posts.