Tech Magazine Loses June Issue, No Backup

Gareth writes "Business 2.0, a magazine published by Time, has been warning their readers against the hazards of not taking backups of computer files. So much so that in an article published by them in 2003, they 'likened backups to flossing — everyone knows it's important, but few devote enough thought or energy to it.' Last week, Business 2.0 got caught forgetting to floss as the magazine's editorial system crashed, wiping out all the work that had been done for its June issue. The backup server failed to back up."

Comments

After the swearing stopped.

[AltGrendel]

The first words from management were "You're kidding me, right?"

Then the swearing started again.

Re:After the swearing stopped.

[xzvf]

That's what I said when Byte went out of business and instead of refunding my subscription they sent me "Business 2.0". Multiple letters latter, they stopped sending the piece of crap magizine, but I never got my refund. It was only like $10, but it was mine.

Re:After the swearing stopped.

[sjwest]

We do backups, but i dont floss. Quite never saw the point of flossing. - perhaps they should have flossed less.

Re:After the swearing stopped.

[Anonymous Coward]

Actually that is what you get for having Geek Squad as your outsourced IT staff.

honestly, they CANT have competent IT. The FIRST thing you do in the morning is check the backups.

I have a HP sdat jukebox here and I STILL check the backup logs to make sure the backup and verify succeeded last night. if they dont I mirror the important files right away and then run a manual backup to not lose the last 24 hours of backup.

I hope that Business 2.0 learned that paying top $$$ for competent IT is a good idea and they should run a article about it.

Re:After the swearing stopped.

[TrippTDF]

Agreed- this is just a no brainer.

What does it say about a tech magazine that can't even handle the basics of technology?

Re:After the swearing stopped.

[IdleTime]

I am not surprised.

There is not a week going by without me getting an issue from one of out regular analysts with question about how the customer can salvage their data because they don't have a backup. My standard answer is that we may be able to save some data, but it's going to cost a lot of $$$. And I also say: "When you don't have a backup, you have either deemed that you can easily recreate the data or that they are not important for the company"

And these are not mom&pop companies but big multi million/billion dollar companies.

Re:After the swearing stopped.

[Auntie Virus]

I have a HP sdat jukebox here and I STILL check the backup logs
HP DAT? You'd better do more than check the logs. A test restore (if your users don't already test for you by deleting files) at least a few times a week might save your butt one day. Actually DAT or not, test restores are a must. Logs lie.

Re:After the swearing stopped.

[Nutria]

Actually DAT or not, test restores are a must. Logs lie.

If the logs are lying to you, then that's a prime indication that you're using the wrong software.

Re:After the swearing stopped.

[maxume]

Off topic, but read up on the surgery for periodontitis(that's what comes after gingivitis. Basically, flossing be good for your gums).

Re:After the swearing stopped.

[JATMON]

It is also a very good practice to test the recovery of the backups on a regular basis. Just because the logs say that the backups ran successfully does not necessarily mean that the restore will work.

Re:After the swearing stopped.

[Hoi Polloi]

At what point did Nelson Muntz stick his head in the room and say "Ha Ha!"?

Re:After the swearing stopped.

[Nutria]

It is also a very good practice to test the recovery of the backups on a regular basis. Just because the logs say that the backups ran successfully does not necessarily mean that the restore will work.

Sure, if you have the capacity.

And don't reply "disks are cheap", because there's more to a disk subsystem than buying a few 500GB drives from NewEgg.

Re:After the swearing stopped.

[Anonymous Coward]

Verifying logs is only half the chore. They were probably doing that everyday. A good admin will have the backup software email his/her cell phone if there is a failure. The more important part is actually verifying the backup is working. I often copy random directories/files to my workstation, then delete them from the server and make sure I can pull them off tape. A "success" entry in your log != a successful backup - it just means the software thinks everything is ok.
 
We can laugh about what dumb asses they are, but I feel for them. In the crazy world that is network management, some times these important processes can get squeezed out by immediate demands. But still, you gotta do it.

Re:After the swearing stopped.

[Anonymous Coward]

First three Google results regarding the importance of flossing:

http://www.saveyoursmile.com/healtharticles/flossi ng.html
http://www.hopkinshospital.org/health_info/ENT/Rea ding_Room/flossing.html
http://www.robmaeder.com/archives/6

Don't be a filthy-mouth, floss your teeth!

Re:After the swearing stopped.

[AK Marc]

You can be using great software, have the logs tell you everything completed successfully, and still have useless backups. I was a consultant at a company. We were brought in to run the IT department that was fired (new management, cleaned house for some good reasons, some bad reasons). They skimped on the number of people they brought in. We were just keeping up with the help desk functions, and mentioned in the daily report that backups hadn't been checked, UPSs were bad, and all sorts of things that required more attention. They specifically told me not to work on the bigger issues (specifically because they thought consultants were evil and I'd just be recommending them spend more money to fix stuff they didn't need, after all, it's working now, right?). Well, one day, we needed to get something from a backup. It turns out that, long before we got there, the tape library was set to record to only one tape. There were 6 tapes in the cartridge. So, there was one successful of the last server backup (the least important server, hence why it was backed up last), and the other 5 tapes were blank. But the job logs correctly relayed that the backups had completed successfully.

So yes, you can have the logs "lie" (report success for useless backups) and have perfectly fine software. Oh, and when I finally left, they had 20 or so desktop UPSs still daisy chained for their server backup power plan, and nothing telling servers to auto-shutdown, so the occasional power failure would corrupt open databases and they had so little uptime that some servers couldn't even finish the shutdown procedure before battery power ran out.

Re:After the swearing stopped.

[Gilmoure]

Ah, I thought I was going psychotic when I heard his laugh, reading the blurb this morning. At least with auditory hallucinations, I don't have to ever be alone.

Re:After the swearing stopped.

[ElectricRook]

I would imagine they had a "complete backup solution". So simple anyone who could drive a mouse could do backups.

Then they found someone who could drive a mouse. And that someone could click on "do backup". And the magically complete fully integrated all comprehensive patented backup solution provided a nice friendly reassuring pop-up window that said something like "Your computer is protected".

Instead of browsing the log files, or even scripting the browsing of the log files. The mouse driver went back to making very nice power point presentations that showed off their fantastically technical prowess.

Re:After the swearing stopped.

[iminplaya]

I hope that Business 2.0 learned that paying top $$$ for competent IT is a good idea and they should run a article about it.

Yeah, but maybe this time they should just use a typewriter and mimeograph machine.

Re:After the swearing stopped.

[Anonymous Coward]

Easier said than done. Without fulltime IT, what anal retentive office drone is going to check the logs first thing every morning. Does the receptionist do it? The CEO? Have you every looked at modern backup logs? One minor hicckup and it looks like a catastrophic failure.

Backup systems are ridiculously complex. Tapes are overpriced relics that are good for offsite and thats about it. What people (clients) need to understand about backup is that it costs A LOTof money to maintain. Most IT types are not good enough salesmen to convince small companies that they need to spend a ton of money for the insurance of a realiable backup system.

Re:After the swearing stopped.

[loafing_oaf]

The problem is that tech magazines are in the advertising business, not the tech business. I write content for the Web site of a tech radio show, and it's just a bunch of us in cubicles looking stuff up on Google. No tech people involved.

Re:After the swearing stopped.

[Xeriar]

A place a relative of mine worked at had spent loads for this 'backup' software.

It came with no recovery features. It would 'back things up', but there was no actual process for recovering said data.

Re:After the swearing stopped.

[PinkPanther]

Backup software cannot confirm that your restore process will work. Only a test restore will.

The software might be able to ensure that all the bits put on the backup medium were recorded correctly, but it cannot possibly tell you that restoring that database and firing up the application will "work", or even that the resulting blob restored to disc would be even usable.

Re:After the swearing stopped.

[rbanffy]

A periodic test restore (or better - a disaster recovery drill) tells you that the backup is readable and it is usable to recreate the environment. A perfectly correct backup can be useless if it backs-up wrong or incomplete data.

Having proper UPSs, decent servers with RAID, fast disk-to-disk backups and a hardware retirement policy is essential. And, when compared to the cost of restoring lost data (which may include some jail time for the executive officers) it is trivial.

Anything less is risking too much.

Re:After the swearing stopped.

[Sandbags]

I work for a backup company that makes D2D backup appliances supporting more than 20 operating systems.

First, no one really understands best practices for backup, and a lot of systems that are backed up "successfully" can't be restored anyway (in fact, most commonly this is Microsoft Exchange, the most important system in most companies!). Second, Tape sucks! You MUST have Disk-to-disk backups to have any true recoverability in today's world. Third, check you logs EVERY day, there's no excuse! Fixing a failing backup should be the number 1 priority second only to an actual failed server you are recovering. Next, nobody spends enough on IT disaster recovery, and no one documents the recovery process properly. Your IT spending on DR should be approximately 25% or more of your total IT budget for server systems. At least 1 day per month should be used to practice system recovery or update the documentation covering it. Next, nothing should ever be considered backed up until the server has been test recovered, completely from scratch, at least once. At least some data should be recovered from backup media every day just to be certain it can be done when needed. The test recovery should be of a random critical data folder, or database, not the same stuff each time.

Off-site DR is also important. Making sure that your entire data set for all critical systems is moved off site every 24 hours is a must. Included in this should be any media required to process a restore (not just the backups, but the install CDs, BareMetal recovery disks, licenses keys for all servers and applications, the DR documentation itself, network architecture information, hardware and software configuration of each server, and all information regarding your ISP contract, and system warranties from each manufacturer. If you don't have all this stuff, contract someone who knows what they are doing to make it for you.

For each unique mission critical system you have (Mail, critical database server that allow the business to operate, point application server, Citrix box, etc) you should have a complete spare system meeting the system requirements so that system can be restored immediately in the event of a system outage. Your system recovery tests should be performed regularly to that hardware. Best practice is also to keep those test boxes off-site when possible, but nearby enough to get in a jiffy. If you don't have spare lab equipment, and don't have enough budget to have it, you can't afford to have those critical systems in house, and should consider outsourcing a data center who does have those resources. Clustering is complicated and expensive, but spare chassis and a few spare drives don't amount to a huge IT burden. You don't have to have 1 for each server, just one that can handle the job of each unique mission critical system (if you have 5 SQL servers, 1 exchange, 1 citrix, and 4 file servers, you only need 4 total spare system).

The average business that goes through a critical system disaster that interrupts business for more than 48 hours requires 1 month of revenue to overcome the loss of each day of downtime. 40% of businesses that have a site disaster lasting more than 3 days go bankrupt within 90 days of the event. How much money will your business loose if you have to roll your purchase database back 2 days and loose all records of those transactions? How will your business survive if e-mail is out for 3 days? How much will you loose if your online store is gone for several days? How many customers will you loose if your support department is off-line for 2 days? How much will you be sued for if you miss a contractual deadline due to data loss? Can you afford to NOT spend the money to make sure this doesn't happen!?!?!

Re:After the swearing stopped.

[asdfghjklqwertyuiop]

If the logs are lying to you, then that's a prime indication that you're using the wrong software.

How do you know if the logs are lying to you unless you try restoring?

Re:After the swearing stopped.

[JATMON]

Definitely not a cheap solution. Not only do you have to take into consideration the cost of hardware, but the time that your sys admin & whomever else that needs to be involved, will need to spend recovering the backup and then validating the recovery. It just comes down to is it worth the extra cost. If I am backing up the core of my business like databases that house my financial & customer info, or a system that has the only copy of the next issue of my magazine, then I would spend the money to have a system that I can use to test the recovery of my backups. If my systems are not critical to the success of my business and I really don't have the extra money to spend, then I would take the chance that I can recover my data.

Re:After the swearing stopped.

[1u3hr]

Long ago I found myself looking after the accounting system for a bookshop, purely because no one else in the company had half a clue. So I made a system to back up the large and expensive (for those days) database of inventory and sales; using a box of floppies with five sets. The staff would back up each day before shutting down. When the system booted it ran Norton Disk Doctor to check the integrity of the system.

One day I got a call that it wouldn't boot. Turns out the hard disk is corrupted. So time to get a new hard disk and restore the backup. EVERY backup set was corrupt. It turned out that for a few weeks, the Disk Doctor had come up with a series of warnings of disk errors. The staff had just clicked ignore. So damage in the file system grew until it was irretrievable, and after a week all the backups were of a corrupt database. Fortunately the company was going bankrupt by then so ultimately it didn't matter....

Re:After the swearing stopped.

[lucabrasi999]

in fact, most commonly this is Microsoft Exchange, the most important system in most companies

E-mail is probably considered important by the end-users. But, for most organizations, it should NOT be considered to be the MOST important. Consider the industry an organization is in:

A distribution company? The Supply Chain system should be most important.

Manufacturing? Customer Order/Manufacturing systems

Health Care Organization? Patient Records

Military/Defense? Supply Chain

Finance? Accounting Software/Cash Flow/Investments (depending upon specialty)

My company? Payroll

Ok, the last one is a bit of a bad joke. But, for most of the above types of companies, I would argue that e-mail is NOT number one. In fact, e-mail should be well down the list, after EDI, CRM, Accounting, HR/Payroll, and the examples I list above.

Re:After the swearing stopped.

[Anonymous Coward]

There is also evidence that it's good for your heart.
BTW, next time don't say "This is offtopic, but. . . " Automatic Offtopic mod for that, y'know.

Re:After the swearing stopped.

[Fulcrum of Evil]

If you ever believe everything the logs tell you, then you haven't been in the business too long.

Re:After the swearing stopped.

[dgatwood]

Doesn't take capacity. Just takes software that doesn't suck and a little bit of additional time. If your backup solution is of sufficient caliber, it should be able to "restore" without restoring any data, i.e. read back the tape and verify it against the current contents of the drives. It should then flag everything that has changed, and if it gets any I/O errors during the process, it should tell you that there is a problem. Finally, if the time stamps don't indicate that those files have changed since the time stamp on the backup, then something went wrong during the backup process, and it should tell you that there is a problem.

It really isn't hard to verify your backup---at least it shouldn't be.

Re:After the swearing stopped.

[davidsyes]

There's NO business like no BUSINESS!

Re:After the swearing stopped.

[Wolfrider]

MOD PARENT WAY thehell UP!!!11one!

Re:After the swearing stopped.

[Nutria]

So time to get a new hard disk and restore the backup. EVERY backup set was corrupt. It turned out that for a few weeks, the Disk Doctor had come up with a series of warnings of disk errors. The staff had just clicked ignore.

That's user stupidity, not log file lying.

Re:After the swearing stopped.

[Nutria]

So yes, you can have the logs "lie" (report success for useless backups) and have perfectly fine software.

That's operator/management error, and has nothing to do with whether the backup s/w wrote erroneous crap into the log file.

Re:After the swearing stopped.

[Nutria]

How do you know if the logs are lying to you unless you try restoring?

Dump the contents of the tape.

Re:After the swearing stopped.

[Nutria]

If you ever believe everything the logs tell you, then you haven't been in the business too long.

I've been in the business 19 years, 16 using OpenVMS. It must be that I've been spoiled by expensive, high-quality hardware & software that actually works and does what it says it does.

Re:After the swearing stopped.

[AK Marc]

Yes, exactly what I said. You can have logs report a success and have there not be a success. That is not necessarily the fault of the software.

Re:After the swearing stopped.

[Nutria]

little bit of additional time.

Not if you've got pot-loads of data.

read back the tape and verify it against the current contents of the drives.

And if the underlying data has changed?

It really isn't hard to verify your backup---at least it shouldn't be.

Hard? No. Expensive? Very possibly. (We regularly "test" database backups by restoring them to the QA system, and yearly do a DR of one of the systems.)

Also, the systems that we use (Rdb/VMS and OpenVMS BACKUP) have options to read every block on the tape, checking that they can be read and that it's reasonable.

Re:After the swearing stopped.

[Fulcrum of Evil]

Must be. I get cranky software that was written by people long departed, and the logging is spotty in places. Best to verify that files went where they should have.

Re:After the swearing stopped.

[sgt_doom]

Most important point of all - Business 2.0 is a total crap magazine.

Who reads that magazine?? Certainly no true geek - and certainly nobody with an IQ above a hamster.....

Re:After the swearing stopped.

[Nutria]

Must be. I get cranky software that was written by people long departed, and the logging is spotty in places. Best to verify that files went where they should have.

That's why I dread the planned migration to Linux & Oracle RAC.

For 7 years, Linux has been the only OS on my home PC, and OpenVMS is behind the times, but it's straightforward, religiously consistent across utilities and Just Works. Backup files properly and simply get restored from tape, database backups properly and simply get restored from tape, when there's a hardware crash and the machine is finally brought back to life, the 100 dead transactions roll back without a worry. The HPUX+Oracle DR test bombed because some obscure bit of software didn't get installed properly, but the Rdb/VMS DR test finished ahead of schedule.

Re:After the swearing stopped.

[1u3hr]

That's user stupidity, not log file lying.

Did I say otherwise? It's just a related anecdote.

Re:After the swearing stopped.

[Fulcrum of Evil]

That's why I dread the planned migration to Linux & Oracle RAC.

As well you should. Oracle RAC at our place has been a disaster. We vastly improved our reliability by switching to horizontally partitioned databases. It's more work, but less pager duty.

Re:After the swearing stopped.

[dgatwood]

Just to be clear, I'm not saying that it isn't a good idea to periodically restore onto a fresh system. That's certainly a good idea to do, if only to ensure that all of the data is getting backed up. I'm just saying that there are things you can do to dramatically cut your risk of problems even if a complete restore and test is impractical.

And if the underlying data has changed?

I assume that you time stamp data records and/or files when they change, right? If the data differs and the modification date on the live system is after the modification date on the backup, nothing is wrong (though marking it for a quick incremental "catch-up" backup might not be a bad idea). I think I addressed that point earlier.

The problem of data changing during the backup is trivially solvable with filesystem-level or database-level snapshotting as well. Do a backup of the snapshot, then verify the backup against the snapshot. If anything changed, something went very wrong.

A similar trick that I've heard of folks doing for databases that don't support snapshots is setting up a slave database. Clone the data to the backup DB, set up the master database to automatically forward a copy of any changes to the slave database to avoid any problems with data changing during the cloning, then after the clone is running in parallel, start dropping new transactions on the slave database until the database settles and/or roll back any pending transactions, then yank the slave database offline and back it up.

Last tip: while this is somewhat obvious, always do the verification after all backups are finished. That should catch situations in which a tape library wraps back on itself. :-)

Re:After the swearing stopped.

[Wolfrider]

Sandbags - pls email me off-board; $boss my be in the market for your company's D2D product... Thx

Re:After the swearing stopped.

[Bustergates]

"It's just a bunch of us in cubicles looking-up stuff on the web. No real tech people involved!" Really? You don't say! I'm flabbergasted! And how is this any different from any other blog, web site or magazine in the universe may I ask? -BG-

Re:After the swearing stopped.

[yellowalienbaby]

Checking it every morning is a bit laborious. I have monitoring set up, making nice webpages permanently displayed and constantly updating. It tells me if it's gone wrong, and will even recover from and re-run certain failures. I don't have to go and ask it. s'amazing what you can do with a bit of python, unix and apache.

With this much free advertising

[Anonymous Coward]

who needs a magazine?

Re:With this much free advertising

[Anonymous Coward]

My God! It's full of advertising!

*zap*

Re:With this much free advertising

[Anonymous Coward]

Come on, man.. I read it for the articles.

Oblig.

[cheese-cube]

Ha Ha!

We've all been there. Don't be too pious, here.

[ScentCone]

Maybe not so bad as losing your entire monthly product, per se... but it does happen. I'll bet their accounting, HR, and other back office systems are probably fine. This stuff is always ugliest at the department server level in smaller operations. I'll bet they get some good Mea Culpa 2.0 editorials out of it, though.

Re:We've all been there. Don't be too pious, here.

[Thyamine]

I agree (about not being pious about it). No matter how often I tell clients to backup files/software, or we sell them solutions to do it, it never fails that people lose important data. Including myself at home. You always assume things are running fine, or think 'hey, I'll just backup this weekend'. Then, system crash, power outage, etc, etc.

It happens, it will continue to happen, not much else to say really.

Re:We've all been there. Don't be too pious, here.

[Silver Sloth]

Maybe not so bad as losing your entire monthly product, per se... but it does happen If it happens to me then I'm looking for a new job. Honestly, losing 24hrs worth of data is the worst that should happen. Well, OK, maybe 48hrs if you happen to be unlucky enough to have a crash just before the backup after a failed backup (if you see what I mean) And if the data is that important then a suitable RAIDed disk array will sort things out. (we use mirrored RAID5 arrays in separate data centres but I work for a bank)

No, I'm not being pious, I would guess that the loss in revenue from a lost edition would cover the costs of a sufficiently resilient system that data loss would be next to impossible. In a business situation loss of data is nearly always unacceptable.

Re:We've all been there. Don't be too pious, here.

[daeg]

There are many, many industries where losing any data is unacceptable. Banking and health care (which I work in) being the front runners in my book. I can't imagine losing even 15 minutes' worth of patient data, or accidentally deleting test results, or losing even a single piece of a record. Not only is it potentially dangerous (for instance, losing the allergy information for a patient), it's bad service. Who wants to get their blood re-drawn because of an IT problem?

Re:We've all been there. Don't be too pious, here.

[LurkerXXX]

And if the data is that important then a suitable RAIDed disk array will sort things out.

The topic here is backups, not RAID.

Say it again with me everyone "RAID IS NOT A BACKUP"

RAID increases-uptime by decreasing/eliminating the downtimes needed to do restores when an individual drive bites it. It is *NOT* a backup.

RAID does not save you if someone accidentally deletes a needed file.
RAID does not save you if your machine gets nailed by a virus/upatched-exploit.
RAID does not save you if the drive power supply fries taking out attached hardware.
RAID does not save you if a bugler steals your machine.
RAID IS NOT A BACKUP.

Re:We've all been there. Don't be too pious, here.

[Silver Sloth]

Get down off your soapbox

I never said that RAID is a backup, indeed, I suggested that any professional would have valid backups or no job. However, if no loss of any data is acceptable then all the backups in the world won't save you and two mirrored RAID arrays in separate data centres will help eliminate two of your four reasons for data loss (we can survive the complete loss of one data centre - your #3, and no burglar is going to steal two machines in two secure centres - your #4)

So, repeat after me, backups alone are not sufficient for total data security without fault tollerant storage.

Re:We've all been there. Don't be too pious, here.

[networkBoy]

True.
RAID at geographically separate data centers with journaling(? you know, where every change is recorded like a CVS ?) file systems, however, are a backup. That's what we use, then take the second system to a disk to disk snapshot, then snapshot to tape, thus the two primary systems are always on-line.
-nB

Re:We've all been there. Don't be too pious, here.

[qwijibo]

"Why waste time and money arguing for things like backups or disaster recovery plans when your personal disaster recovery plan can be to drive across town and get a job with a company that didn't have a disaster."

This is not only a Dilbertism, but it's the sad/funny/haha-only-serious official backup/disaster recovery plan at work. We've gone for more than 3 years without backups and management is giddy with all of the perceived cost savings. It's amazing how many hardware failures you can pretend didn't happen if you have a good disk array for all of your primary storage. Fortunately, we're only the marketing organization for a major bank, not something that would be missed if all of the data and people vanished one day.

Re:We've all been there. Don't be too pious, here.

[Anonymous Coward]

...taking out attached hardware.
RAID does not save you if a bugler steals your machine.
RAID IS NOT...

Why would a bugler steal my machine... I could understand them stealing by bugle... I didn't even consider that in my data security plan...

Re:We've all been there. Don't be too pious, here.

[osu-neko]

If IT problems drew blood from those who caused them, there would be fewer IT problems. ;)

Re:We've all been there. Don't be too pious, here.

[Nick Number]

RAID does not save you if a bugler steals your machine.

But can he carry my server and his horn at the same time?

Re:We've all been there. Don't be too pious, here.

[Anonymous Coward]

Hardware has been doing it for years. We just need a clever developer to find some way to control the sharp edges with software.

Re:We've all been there. Don't be too pious, here.

[Statecraftsman]

And he's off...

Nelson Muntz

[erroneous]

Some stories should just come with Nelson Muntz sound files embedded.

Ha-ha!

Re:Nelson Muntz

[Kierthos]

Heck, I'm wondering why there's not a haha tag on this already.

Re:Nelson Muntz

[Anonymous Coward]

Heck, I'm wondering why there's not a haha tag on this already. I think the haha tag got blacklisted from the front page because it was showing up on so many stories.

Re:Nelson Muntz

[cyberchondriac]

I made this a few years ago:

http://www.phrets.com/muntzed/

I can feel for them though - there's nothing easy, routine, or "automated" when it comes to good backups. Constant vigilance !

Re:Nelson Muntz

[Yankovic]

I wish there was a way to post this as a player instead of a link: http://www.entertonement.com/188534/The_Simpsons/H a_Ha/Tone.aspx

err...

[cosmocain]

HAHA!

*coughs

TFA:

Business 2.0 never had to rely on their backup software until that day, which is why they probably did not realize that it was either obsolete or dysfunctional.

sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

Re:err...

[Lumpy]

hell with that. ever heard of competent IT staff? why has their CTO not been fired yet?

honestly though, talking management into backup solutions is like pulling teeth, then they blame you for not having it in place when the failure does happen.

Last place I worked at we were using 4 year old DLT tapes because management was too stupid and cheap to buy new ones.

"we will buy new when those fail" is what we were told.

Re:err...

[morgan_greywolf]

ever heard of verifying backuped data?

Errr...uhh....umm...'verifying'? Uh, I'll be right back!

Re:err...

[dal20402]

/grabs hammer...

*bang* *bang* *bang*

Oops, it looks like a couple of those DLT drives are running into problems. We need replacements. Did you see what happened to Business 2.0?

Re:err...

[radtea]

sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

I'm sure they've heard of it, in a conversation that went something like this:

IT Guy: We need a system for verifying our backups.

Suit: How come? Don't the backups work?

IT Guy: We need to be sure that if there is a failure, the backups will be ok.

Suit: But they're just copies, aren't they? I copy files all the time and it never goes wrong.

IT Guy: This is a little more complicated than that.

Suit: How hard can it be?

IT Guy: Well, I was thinking we might need to hire a part-timer just to take care of backups and verification.

Suit: But we've never had a failure! Sounds like empire building to me. I know that's what I'd be doing in your position. Nice try. We'll keep the backup system the way it is, thanks.

IT Guy: But..!

Suit: Moving on to the next item on the agenda... ok, Executive Bonuses!

Re:err...

[Anonymous Coward]

Verifying? That's simply not an option for many people. With our current Windows garbage backup software, we can't because the backup software can only be installed on the production systems since it has a serial number and a required Internet registration process (think something like Windows XP or many of the newer pieces of Microsoft software). That also means in order to be allowed to use the restoration software we have to get Windows going and have a working Internet connection to be allowed to register the software. We can't test a restore to one of the systems with the backup software because it doesn't understand the concept of restoring to a different location. So, we have almost $200k worth of software, drives, and tapes that we can't verify. That's the way the Windows world works. Of course the IT manager is very happy about the situation becase the constant Windows problems means that he and his many employees stay busy. Since he now has four dozen employees to manage our 15 servers, he makes a lot of money.

We used to use SunOS and tar. Of course it was possible to verify backup tapes done with tar onto a production system when you delete the leading / on the restore and make the destination dir something other than the production dir. We could also restore to development systems because tar is freely available. It's been over a decade since we switched from SunOS, but I still really miss it. At that time we had more servers, and I was the only IT employee. Now there's 49 people managing fewer servers. We do more with our servers and have more employees, but the amount of manpower required to keep Windows running is just ridiculous.

Re:err...

[Princeofcups]

>sorry, their MAIN problem is not in any way a dysfunctional backup system. ever heard of verifying backuped data?

"Yes, we guarantee 100% that your data is being backed up. Look at all those tapes going offsite. You need a recovery? I dunno. We never tested that."

jfs

Re:err...

[metamatic]

No, it's worse than that. He said the *tapes* are 4 years old. They've been using the same *tapes* for backup for 4 years.

That's like playing Russian Roulette with your data. Once a tape gets that old you might as well not bother backing up to it at all.

Re:err...

[Bastard of Subhumani]

ever heard of verifying backuped data?

I tried once, but I fuckupded it.

Re:err...

[tim_uk]

We can't test a restore to one of the systems with the backup software because it doesn't understand the concept of restoring to a different location.

What POS backup software are you using? And you say it cost 200K???

Re:err...

[networkBoy]

That's just scary.
When I was running a lab I was responsible for about 100G of data. This was engineering data and any loss directly resulted in having to re-test devices and re-create the data. I used a RAID5 array, and a 24gig/tape x 8 tape autoloader DAT system. I couldn't get them to spring for DLT, but I was able to get them to spring for 8 new tapes per week. So, I used 4 tapes per backup (thus two backups existed at a time), the last backup of the week was archived, the other four tapes were destroyed, and a fresh 8 tapes were loaded.
-nB

Re:err...

[Clover_Kicker]

No, it's worse than that. He said the *tapes* are 4 years old. They've been using the same *tapes* for backup for 4 years.

That's like playing Russian Roulette with your data. Once a tape gets that old you might as well not bother backing up to it at all. 4 year old tapes is more like Russian Roulette with 5 bullets and 1 empty chamber.

Re:err...

[Speare]

"we will buy new when those fail" is what we were told

"Your successor will buy new when these fail." is the correct response to this.

Re:err...

[Fulcrum of Evil]

"we will buy new when those fail" is what we were told.

Schedule a meeting with them. Show up with a stack of tapes and a mallet. Tell them "These tapes are 4 years old. They have failed."

Re:err...

[Anonymous Coward]

The correct responses are one of either:

To boss: "We need new backup software. The current software is usless. Here's why..."

To backup software company: "Can we have a temporary license to test our DR plan? If no, will you take responsibility if we are unable to restore our data?"

Re:err...

[Anonymous Coward]

i call bullshit. It all depends how often the tapes are rewritten. In a large tape library installation with hundreds of volumes it's not uncommon to use 4 year old tapes and it would be foolish and expensive to replace them every year. Of course provided we're talking LTO here, not some DDS which is not reliable even as out-of-box-fresh.

Re:err...

[Fulcrum of Evil]

If the suits are so cheap as to deny tape replacements, you're probably well past the safe number of writes already.

They probably still have most of it

[ZachPruckowski]

I imagine that they still can resemble a lot of it from other files - they should still have all the layout pieces for one, and all the authors ought to have at least rough drafts of their stories on their personal computers. The deadline's screwed, but they can probably get it out a few weeks late (or in July, depending on how often they normally publish).

Re:They probably still have most of it

[chiskop]

From the article:

Luckily for them copy-edited text of the lost issue has been mailed to the lawyers to get legal clearance. The page layouts had to be totally redone from scratch.

Re:They probably still have most of it

[Threni]

Yeah, great, that's the content - now how about the advertising? That's where they make their money.

Re:They probably still have most of it

[Red Flayer]

Yeah, great, that's the content - now how about the advertising? That's where they make their money.

Editorial department server content was lost. Advertising content is normally handled by the production department.

I think we can all relax and rest assured that the June issue of Business 2.0 will have all its intended advertising.

Why isn't this a default

[Anonymous Coward]

Why isn't it a default for an OS to ask where the backups should go when it is installed? Backups are like any other kind of security ... necessary. You shouldn't have to hunt for the instructions on how to back-up, they should be in your face.

Re:Why isn't this a default

[metamatic]

Why isn't it a default for an OS to ask where the backups should go when it is installed?

Wait for OS X 10.5 and "Time Machine".

Re:Why isn't this a default

[PygmySurfer]

Because everyone's backup needs are different?

You shouldn't have to be hand-held for every little thing, you should have a competent IT staff.

Re:Why isn't this a default

[mrchaotica]

The real question is, why did it take 23 years before even a single PC OS managed to get that functionality?! It's really the kind of thing that should have been in all OSs from day 1.

Re:Why isn't this a default

[toddestan]

Wait for OS X 10.5 and "Time Machine".

Time machine isn't really a backup. If your harddrive craps out, you still lose everything.

Re:Why isn't this a default

[nibblybits]

Time machine isn't really a backup. If your harddrive craps out, you still lose everything. That's not true---a backup is exactly what time machine is. It backs up to an external drive or server, it's not a versioning file system like zfs.

What we're missing.

[Anonymous Coward]

HOW TO TAKE YOUR IDEA FROM CONCEPT TO $30 MILLION - OVERNIGHT

Oh God! I'm sure we're missing soooo much!

HAHAHA

[Anonymous Coward]

HAHAHA

Rag

[BigDumbAnimal]

No June issue?

That's OK, nobody reads Business 2.0 anyway.

Re:Rag

[Fred_A]

Read ? Well, that's not a problem if you need something to read. But what am I going to line the litter box with ??

Re:Rag

[Jim Hall]

That's OK, nobody reads Business 2.0 anyway.

Well, certainly not anymore. :-)

Re:Rag

[UbuntuDupe]

nobody reads Business 2.0 anyway.

I wish. I wish people didn't read Time, either (the publisher), but they do. Time's writing style is the dumbed down, try-to-be-hip crap I wouldn't have gotten away with in sixth grade. Seriously. Like I said before, to understand why its writing is like fingernails on a blackboard for me, consider how the same information would be conveyed by two sources:

8-year-old: "6 divided by 3 is 2."

Time magazine: "Okay, imagine you've got a half-dozen widgets, churned out of the ol' Widget Factory on Fifth and Main. Now, say you've gotta divvy 'em up into little chunklets -- a doable three, let's say -- and each chunklet has the same number that math professor Gregory Beckens at Overinflated Ego University calls a 'quotient'. The so-called 'quotient' in this case? Dos."

Based on how that post got modded, I'm not alone in this.

Re:Rag

[Repton]

Back when I used to watch TV news here in NZ, the standard unit of weight was the "block of butter". Whenever the newsreaders quoted a weight, they would immediately follow it with a conversion to 0.5kg blocks of butter:

"This doodad weighs four kilograms. That's about eight blocks of butter. This other one weighs seven kilograms. That's like fourteen blocks of butter."

You can draw your own conclusions about what they think of their audience...

What was the nature of the crash?

[91degrees]

It seems unlikely to have crashed in such a way that a data recovery specialist would be unableto get most of the data back.

But whatever the case - there is a useful lesson here. Make sure your backups are backing something up.

Re:What was the nature of the crash?

[EveryNickIsTaken]

Sounds like the editor dropped his iBook.

Re:What was the nature of the crash?

[Chris whatever]

Hum!!! Unless you are there looking at the data being backed up there is no way unless you get notification from your system that it has completed.

usually that is the case but it has happened when one of my backup failed one night and someone needed a file restores from the previous day, if that company never checked it's backup or never configure some kind of noticaition upon failiure or success then they are very lame

The laws of the universe

[pseudosero]

Hold true, once again.

Re:The laws of the universe

[Architect_sasyr]

Never underestimate the power of human stupidity eh...

This happens all too often, it's funny because it was Business 2.0, but would we all be laughing if it was the main Ubuntu repository (multi-site distributions are conveniently ignored for the purpose of this post)? Or [insert OS/code here]? I know I'd be pretty pissed if this happened to my kit.

Let this be a lesson to you all... the laws of the universe DO hold true... and every man or woman can be struck down and proven wrong.

Re:The laws of the universe

[Anonymous Coward]

We'd all be laughing our asses off if it were the main Microsoft Windows repository though.

Re:The laws of the universe

[MightyYar]

Couldn't they just get it back from one of the P2P networks?

Re:The laws of the universe

[Anonymous Coward]

There are 10 kinds of people in the universe -- those who have lost data, and those who will!

High profile SNAFUs

[Rob T Firefly]

This reminds me of the recent uproar over a car crash involving the New Jersey governor. He was critically injured because he wasn't wearing his seatbelt, and people freaked, asking what sort of role model he could possibly be. I argued that he was an awesome role model, because sometimes people need to see a mistake end badly for someone else before they'll do what's necessary to protect themselves from making the same mistake. Seeing a high-profile magazine get hit like this can do the same for backup slackers the world over.

I don't know about you people, but after reading this (and giving it the "haha" tag) I'm going home and catching up on a couple of backups I've been slacking off on for a while.

Re:High profile SNAFUs

[CrackedButter]

... and I'm going home to floss!

Re:High profile SNAFUs

[SQL Error]

I argued that he was an awesome role model, because sometimes people need to see a mistake end badly for someone else before they'll do what's necessary to protect themselves from making the same mistake.

And then he was caught speeding on the way home from hospital.

Re:High profile SNAFUs

[Hoi Polloi]

I wouldn't use the term "role model" for things like that. I'd say "examples" is the better word. The governor was an example of what NOT to do.

Re:High profile SNAFUs

[Rob T Firefly]

Very good point, that's a much better way to put it.

Re:High profile SNAFUs

[shawn(at)fsu]

Reminds me of the demotivator Mistakes

Re:High profile SNAFUs

[PygmySurfer]

Was he wearing a seatbelt this time?

Re:High profile SNAFUs

[Anonymous Coward]

From your coworkers and friends:

(waving hand in front of nose) "Woowee! Please do...geeesh"

Re:High profile SNAFUs

[Anonymous Coward]

Then the same arse hole was seen driving away from the hospital at (a mere) 70 mph, not wearing a seatbelt again.... I guess some people never learn, too bad the fuck didnt die...

Re:High profile SNAFUs

[fbjon]

I recently indiscriminately backed up all important stuff on about 10 DVD's. The problem is, how should I now continue..? I have new stuff, that also need to be backed up eventually, preferrably in some organised manner.

Pr0n still on the way right?

[minotaurcomputing]

As long as those magazines that come in the smarmy black plastic covers still arrive I can't complain.
-m

What software?

[reed]

Anyone know what "editorial software system" (CMS) they use?

How does this actually happen?

[Orange Crush]

There aren't a lot of ways for a machine to "crash" that loses all its data. Even a lightning-fried hard drive can have its platters removed by a data recovery lab and many files can be pulled off. A mechanical failure doesn't grind the platters into sand. As a network server it really should have a RAID too. So how exactly can "the server crash" so spectacularly that the RAID, backups, and widely available data recovery services all fail? Did the building blow up?

Re:How does this actually happen?

[BVis]

The data "should" be recoverable, and the network server "should" have a RAID. I bet you anything that someone in IT asked for money for the RAID and it was denied, since lots of people with budget control think RAID is bug spray.

Backups and fault-tolerant hardware cost money. You can talk about potential losses and risks until you're blue in the face, until it *actually* costs the company money, nobody will listen. What's going to happen here more than likely is the person who asked for the RAID will get fired, as they're probably the same person in charge of the backups. This will also provide a scapegoat for that person's manager, since obviously if they got fired for it there need be no further repercussions or changes in behavior.

The only way they deserve to get fired is if they didn't advocate as hard as possible for enough backup hardware/software to allow for verification of backed up data and recovery in case of a mechanical hard drive failure. If they did, and were denied, then they did everything they could. (Which doesn't mean they won't get fired, it's just less deserved at that point. However, the thought there is that if they didn't want to get fired for incompetence, they should have tried to become a manager...)

Re:How does this actually happen?

[Rob T Firefly]

IANA publisher, but I would also imagine that in such a deadline-intensive business, data from a fried disk is about as good as lost. Sure, they can send their drives off to data recovery labs who could slowly recover an uncertain portion of the data for a pantload of money, but by the time that's done it'll be time for the next issue anyway. I'd guess it would be a lot quicker and cheaper to write off the disks and salvage what they can from everyone's local copies of the data.

Re:How does this actually happen?

[Paulrothrock]

A mechanical failure doesn't grind the platters into sand.

Doesn't it?

Re:How does this actually happen?

[SuseLover]

The primary purpose of RAID is only protection against HARDWARE failure along and enhancing performance. Backups are the only way to protect data from; user error, program errors, O/S errors (file system corruption), and virus infections/hackers, etc.

I always get worried when a scheduled backup fails and I make sure it is corrected ASAP and do regular restores to verify the integrity of data.

Re:How does this actually happen?

[igb]

`` I bet you anything that someone in IT asked for money for the RAID''

Perhaps, although my experience is that IT people are incredibly bad at framing business cases in terms more compelling than my daughter's request for a mobile phone for her birthday: a few vague reasons, followed by a sulk when asked for specifics.

I keep 20TB on RAID5, and replicate it daily to a RAID5 array that has no components or software above the spindle level in common (Solaris/EMC and Pillar Data). The data we really care about is on RAID 0+1, in some cases with three-way mirroring. We take it out to tape, in case the filesystem pukes over all the copies or the RAID controller decides to go bonkers. We're about to put ten miles between the two file servers. At no point have I had much pushback from management over the money, once the risks and rewards are explained. Too often, IT people convince themselves that some Dilbert-esque stereotype of a manager is going to say no, and therefore make their case in a passive-agressive style that will make anyone say no.

ian

Re:How does this actually happen?

[WillAdams]

Yep.

When you're under a deadline gun, sometimes re-creation is faster than:

  - logging a job ticket w/ the help desk
  - waiting for it to work its way up the queue
  - waiting for IT to figure out which file to restore
  - waiting for the file to be restored

William
(who keeps Quark set to save 10 revisions of each file that he's working on to a backup partition (this can be a couple of GBs of data for some projects) and at a previous job where he was administering an NT Server had everyone's copy of Quark set to make these backup files in a folder on the server which was cleared out after the Friday night backup)

Re:How does this actually happen?

[DebateG]

I once lived with a roommate who got home early one day:

Me: You're home early; not enough work to do?
Roommate: No, the server burned out
Me: Oh, that's no big deal; you just wait for them to get replacement parts and then you get back to it
Roommate: No, seriously, it's burned out. The air conditioning unit failed, the entire server room heated up to the point of spontaneous combustion and the entire server room caught fire

Lesson learned, keep your backups somewhere far, far away from the servers.

Re:How does this actually happen?

[afidel]

For a couple grand per disk there are a number of services that will turn around data in ~48 hours. Unless you are a daily that is probably good enough. The fee from most of the services for an analysis of how much data can be retrieved is generally in the couple hundred dollar range. For business critical data it's a nobrainer most of the time. It's not a substitute for backups, but when the CEO's laptop dies and he didn't follow procedures it can be a career saver.

Re:How does this actually happen?

[awfar]

I understand your point of view when a strategic IT position has already been decided upon, as yours apparently has, but cannot agree on many other situations.

If the IT staff is unable to present a cogent business case, I suggest their position has been deliberately isolated from the Business in general. It is not that hard.

My experience says it was often historical, political, or organizationally ill-placed middle management that would not deal with a unique proposal, formal or otherwise, and could not (or would not) effectively translate the business case and dollar cost up the funding chain for consideration.

Re:How does this actually happen?

[AK Marc]

So how exactly can "the server crash" so spectacularly that the RAID, backups, and widely available data recovery services all fail?

From what I've seen, almost no places verify backups. Sometimes, when adding a server to a backup plan the first time, there is one test restore, but never again. So, the backups were probably bad without their knowledge. This can happen with old media, an old drive with a head out of alignment, storage issues, poor selection of what files to backup, or other such reasons. RAID is useless as backup. RAID is for high availability, but not a "backup." I have seen RAIDs fail in a manner to destroy the usability of all disks at the same time. A bad SCSI cable caused bit errors that propagated until most files were corrupt. Failing RAM in the cache fed bad data to the disks, replicated across all of them until the RAID was effectively wiped out. If you can't conceive of simple ways to wipe out a RAID, then you shouldn't be a sysadmin. RAID does nothing to protect against stupid users and some will not do well in some power failures. As for data recovery, others have pointed out that it may be usable, but the time to get a result exceeds recovery through just re-writing the whole thing. But I'd be interested in what a data recovery place could do with 3 of the 5 disks from a RAID-5. They won't have enough to put back all of anything. And I've seen more than one RAID fail when building a new drive after a failure of a previous one, leaving you with 3 of 5 and scrambling for the backups. Rebuilds are hard, and if one disk failed from the batch of 5 that were all bought at the same time, it may just be near their time and the rebuild pushes one or more of the remaining disks over the edge.

There aren't a lot of ways for a machine to "crash" that loses all its data.

Work in a datacenter surrounded by computers all day, and you'll find there are many interesting ways a computer will crash that can make all the data unrecoverable without outside resources (data services or backups). And no, a data recovery service can *never* be considered a backup, even in such mental exercises as this. Even if they are 99% effective, that is not sufficient to rely on, or even consider a viable option (until you've screwed everything else up, which should never happen with an important server).

Re:How does this actually happen?

[jimicus]

IT people convince themselves that some Dilbert-esque stereotype of a manager is going to say no, and therefore make their case in a passive-agressive style that will make anyone say no.

As an IT person myself, I'm probably guilty of having done this in the past. The thing which trained me to think properly about what I was going to say was a former manager. And it's really not hard.

What is the business benefit for spending money in the way you propose?

That is ultimately what any manager means when they say "why do you want this?". To put it another way,

What's in it for the company?

There are only a handful of possible answers which are of any real interest:

1. It is likely to make the company money. Almost never happens in IT.
2. It is likely save the company money. Happens all the time. Maybe it automates a process, maybe it reduces the risk of something expensive happening, maybe it makes a bunch of people more efficient. Whatever. The important point is being able to understand in clear, concise terms how it will save the company money.

Remember there's a real possibility you're explaining this to someone may only have a rough idea of what the thing you're proposing is, and may not have joined the dots as to how it could help.

Taking the approach "but joining those dots is the manager's job!" (a common one in IT) generally doesn't get purchase approval. A few carefully thought-out sentences are far more likely to.

At this exact moment across the world

[Timberwolf0122]

Tens of IT managers are getting Hundreds of IT minions to check Thousands of backup tapes and befor a senior manager walks in.

Re:At this exact moment across the world

[doobie22]

This popped into my head too :)

Distributed backup

[Anonymous Coward]

They should have just installed p2p software and had backup with multiple off-site copies. (Okay, maybe some copyright and security issues, but one fix at a time.)

Wrong problem

[mseeger]

Hi,

the problem was, as always, not the backup. I've rarely seen problems resulting from the backup process. The troublesome process is the restore. Or as a friend put it once:

Nobody wants backups, what everybody wants is a restore.

In my twenty years of IT i've seen several companies making backups like a well oiled machine. The backup process was well documented and everyone was trained to a degree, they could do it with their eyes closed. But everything fell apart in the critical moment, because all they had planned was making the backup. Nobody ever imagined or tried a restore on the grand scale. So they ended up with a big stack of tapes with unuseable data.

Backup is the mean, not the goal.

Regards, Martin

Re:Wrong problem

[Anonymous Coward]

I see the backup/restore process as risky itself because small errors have major consequences. Secondly the time required for a fix may cost as much as the loss itself. Would mirrored drives be a more effective solution? I want my backup ready to run.

Re:Wrong problem

[swm]

I've been in quite a few shops where the backups worked better than the restores.

Re:Wrong problem

[RetroGeek]

The troublesome process is the restore.

I heard a story about a LAN admin who was doing backups every night. The tapes would go into a safe, then would go offsite, then be used again.

Everything worked well(?) until they needed to do a restore. The tape in the safe was corrupt. The tape at the offsite storage was corrupt. No tape was good.

It seems that the LAN admin made tea every morning. The electric kettle sat on top of the steel safe.

So the backup tape was placed into the safe, then the kettle was started, magnetizing the safe, and erasing the tape.

Not ONCE did anyone try to do a test restore to prove the system.

Re:Wrong problem

[RetroGeek]

I forgot to mention that the LAN adim tested the backup each morning with a sample file restore. Which is why they thought that the backups were valid.

Moral? Test the entire backup loop.

Re:Wrong problem

[mseeger]

> Would mirrored drives be a more effective solution?

Yes and No:

• Mirrored drives are a good protection against drive failures and (usually) offer an easy restore process. If you mirror a drive and put the copy away (e.g. into a safe) this is a real and widely used backup method. As always you should at least try once to boot the system while removing the primary disk. Somtimes RAID controllers have some irks too.
• This method usually depends on the availability of a certain hardware, if you cannot get a new mainboard or raid controller of the same type, the mirrored disk contains data you may have trouble getting at. You may ignore this issue, if you have the same hardware at a safe location again.

Regards, Martin

Re:Wrong problem

[ByteSlicer]

I highly doubt that the kettle could demagnetize the tape in the safe, due to the Farraday shielding. Even if the kettle was on top of the tape (outside the safe), the generated magnetic field would not be strong enough (although the heat would probably melt the tape).

Nice story, though. Reminds me of the sysadmin in my first company who automatically back-upped our server every day. Only problem was: the proces put a copy of the backup on a drive that was being back-upped. You can imagine what happened after a few weeks (it failed, disk full). He only noticed a few months later when we asked him to restore some files.

Re:Wrong problem

[ILikeRed]

That is why backups must be created in an open format.

I don't care what the front end is so much as I care if I can restore onto any system with a minimum of fuss. And as ugly and whatever else you want to call tar, it's format is beautiful and simple, accessible with a wide variety of different front ends, and you have to look really hard to find a system that will not be able to read the format. Even WinZip can read and pull files out of the tar format. I prefer not to have a proprietary back up program - but I absolutely insist on the tar format for the actual archive.

Re:Wrong problem

[lgbarker]

A previous employer's ops center did this. Stacked old mainframe-style reel tapes under a phone with a solenoid-driven bell. Phone rang, erased the tape. Took a couple of cycles to figure that they needed better tape storage than the Ops manager's desk.

Re:Wrong problem

[Hoi Polloi]

That is protection against hardware failure. It doesn't protect against user data loss or recovering old versions. Say someone wants to get an old directory back that was purposefully deleted a while back?

Re:Wrong problem

[afidel]

This is why I love HP, RAID drives can be dropped into almost any other HP machine and the RAID read. As HP puts it:

Data compatibility between models of Smart Array Controllers means customers can easily upgrade to future Smart Array, SAS products to get higher performance, capacity and availability. Unlike competitive products, successive generations of Smart Array products understand the data format of other Smart Array controllers, providing investment protection for your HP storage solution.

Re:Wrong problem

[petermgreen]

if by mirroring you mean raid 1 then it protects against drive failure but not against accidental deletion or complete system failure (e.g. mains gets onto the DC rails and kills all the drives in your array at once) or complete building loss (building burns down).

If by mirroring you mean periodic mirroring to another drive which lives in the same machine or in another machine on the same. that can be a good soloution but the limited number of drives in a machine means you can only back up the most recent stuff if you insist on each copy being a complete bootable system mirror and you still have the problem of something killing all hardware in the system or even all hardware in the building.

If you mean mirroring to a removable or external drive which is safely stored and with at least one copy always in safe storage that can be a very good system but hard drives aren't really built to be carted arround all the time or to survive being in a fire safe that falls through several floors as the building burns down.

if you mean remote mirroring over a WAN that can be a great soloution if you have the bandwidth to handle it. Remote mirroring gets better but more expensive as you expand the geographical area and/or the protection of the mirror sites from any relavent natural disaters.

Re:Wrong problem

[Anonymous Coward]

I call bullshit.

Re:Wrong problem

[Unique2]

Recently I've seen the words "backup" and "RAID" in the same sentence far too often.

Raid IMHO should only ever be seen as a convenience -- it means, for example, you don't need to drive out to your server at 3am and it lets you warn users of the possibility of downtime.

As many before me have said, RAID does not protect against user error or filesystem corruption but also it does not protect against floods, fire, walls falling in, power surges, theft, etc, etc. Off-site backups, it's the only solution!

Re:Wrong problem

[Jeff DeMaagd]

Back when there were consumer tape backup systems, all the programs had an option to verify the tape, which I always had it do. I never had any problems doing a restore.

Right now, I just backup to a local separate array and a remote array. When something went bad, I did lose a week's worth of my work because I didn't back up often enough, but that's far better than losing everything.

Re:Wrong problem

[RetroGeek]

Over time, maybe the safe became magnetized. It probably was not a single event.

Never the less, backup tape testing should encompass the entire tape loop, from tape machine through offsite storage.

Re:Wrong problem

[asdfghjklqwertyuiop]

Would mirrored drives be a more effective solution?

rm -rf /

You tell me.

Re:Wrong problem

[squizzar]

I remember a story (probably an urban legend) of the secretary asked to make copies at the end of each day. This was back in the era of the 5.25" floppy. When everything went wrong, the boss calmly asks said secretary to get the backups. She returns a few minutes later with several reams of paper, each with a floppy disk photocopied onto it...

I suppose that's another case were testing the ability to restore may have provided some insight...

Re:Wrong problem

[Anonymous Coward]

This is why I love HP, RAID drives can be dropped into almost any other HP machine and the RAID read. As HP puts it:

Same reason that some of us prefer Software RAID - giving us even more flexibility in what we use to rebuild after a disaster. I could us an HP controller, a 3Ware controller, or some other controller.

Re:Wrong problem

[turgid]

Where I work nobody cares. The tapes are lying about on a bench (home-made) above our homemade (out of junk) servers which are on a second-hand 100Mb hub.

There is a good old CRT on a KVM on the bench (left on all the time of course) against which all the backup tapes were snugly nesting. I moved them away a couple of inches to the edge of the desk.

No one (PHB-wise) cares. I gave up caring too because I am out of there as soon as a new job comes along.

We develop an "enterprise" product.

Re:Wrong problem

[RetroGeek]

Not at all an urban legend. Go to Tech Tales so see all the stupid things non-techy people do.

Re:Wrong problem

[DocSavage64109]

I believe you need a DC current to magnetize an object, and I doubt very much an electric burner/kettle/whatever is DC.

Re:Wrong problem

[RetroGeek]

AC will magnetize iron. With DC you can choose where North is. With AC North and South flips back and forth as the AC cycles through zero volts. When you turn off the AC, then the iron is magnetized in the last AC direction.

So to de-magnetize something, you need to put it into an AC field, then slowly pull it away. The magnetism will reduce as you get further away until it is effectivley nothing.

Re:Wrong problem

[imsabbel]

Nice try, its still bullshit. Just up there with all the other urban legends.
Sorry, just try it. You wont make an electric kettle delete even a credit card if you put it directly on it.

And sure as hell even IF you magnetized the save, there are also the kirchhoff rules of magnetic flux. The safe is a huge magnetic flux shortcut. It could fully magnetized at 1T internal field and _still_ have neglectable field >a few um away from the surface.

Re:Wrong problem

[Anonymous Coward]

In the 1980's, I was in charge of a computer room of Cray-1s. Remember the "seats" around the bottom of those babies? That's where the (gigantic) power supplies lived. If you put a CRT monitor on the seat, you'd have a hard time reading the display because of all the electromagnetic stuff going on. Of course, the operators just loved to stack our 6250-bpi round-reel backup tapes on the seats. None of the tapes was ever damaged in the slightest.

So, maybe the kettle had a larger field than the Cray-1, but I doubt it. Maybe the tapes in those days were harder to erase?

Re:Wrong problem

[DocSavage64109]

Thanks for the info. I stand corrected about the AC.

Re:Wrong problem

[damium]

RAID is for uptime and speed. If a drive fails you can rebuild the array on the fly (assuming you have hot-swap drives). Off-site backup is not the only solution (there are always fireproof safes) but it is one of the best in most cases.

Here we use a combination of short-term live snapshots and long term versioned network-off-site backups. It's worked well for us, 90% of the time I can pull files that were accidentally changed from the snapshots. The other 10% just takes a bit longer.

Re:Wrong problem

[mrchaotica]

This method usually depends on the availability of a certain hardware, if you cannot get a new mainboard or raid controller of the same type, the mirrored disk contains data you may have trouble getting at. You may ignore this issue, if you have the same hardware at a safe location again.

What I don't understand is, in the case of RAID 1, why the data on each disk isn't bit-for-bit identical to what it would be without any RAID at all. Or is it? It seems to me that you ought to be able to just hook any single disk up to any random disk controller and read it, but I don't know if that's really true or not.

Re:Wrong problem

[mrchaotica]

If you mean mirroring to a removable or external drive which is safely stored and with at least one copy always in safe storage that can be a very good system but hard drives aren't really built to be carted arround all the time or to survive being in a fire safe that falls through several floors as the building burns down.

For personal or small-business use, what are your thoughts on mirroring to a 2.5" external drive (which, being designed for laptops, ought to be built to be carted around) and storing the drive in a safe-deposit box?

Re:Wrong problem

[petermgreen]

sounds reasonable but make sure you have two drives at least one of which is offsite at all times and that you make new backups and swap the drives regularlly.

Re:Wrong problem

[toddestan]

Apparently many RAID controllers write to the harddisks information about the RAID. It's cheaper to it this way, as the RAID controller doesn't need it's own storage medium (flash, whatever). That's why my cheap motherboard RAID has to go out and detect the disks everytime I turn the computer on, then read them to find outI have them in RAID1. Of course, this design has some advantages - I should be able to plug my disks into a replacement motherboard and it should find them and just work. In the case of RAID1, I've been able to pull the drives out, and plug the disks straight into a standard controller or a USB enclosure and read them just fine, so atleast in my case the extra information the controller hides on the drive doesn't seem to affect anything else.

RAID1?

[burnin1965]

And the server wasn't setup with a RAID1 at least for the partitions that hold critical data?

Backup alone is not enough, in some cases there needs to be multiple levels of protection.

Re:RAID1?

[Backup To The Web]

Agreed. A good backup and DR plan requires multiple bullets. RAID stops the front line of things, then local backups (tape, etc). Additionally, online backup is becoming a much more viable solution. Not only could they recover their data, their writers could have retention copies of multiple versions of their articles to rebuild the issue (or even just fix their own accidental deletion.)

Re:RAID1?

[DTemp]

There's a difference between FAULT PROTECTION and BACKUP.

If you don't know the difference, I suggest you learn.

Sitting on your bum with only a RAID-1 protecting your data is only smart if you hate the company you work for and want to stick it to 'em.

Re:RAID1?

[rakeshk_371]

Its very sad and unbelievable with such a company like this to not to have a working backup server for their hard earned work for an international market. Definitely the software designer din't think of such an issue would happen in the future. This is a nice lesson to many a companies.

Mistakes

[catdriver]

Mistakes

"It could be that the purpose of your life is only to serve as a warning to others."

Caught out.

[auroran]

The International Herald Tribune http://www.iht.com/articles/2007/05/01/news/magazi ne.phparticle about it makes a good comment. Business 2.0 has drawn attention for skewering firms, including, occasionally, other magazines, in its annual list of the "101 Dumbest Moments in Business." I wonder if will include itself in the upcoming one? They should also have remembered the old saying "There are two types of computer users, those who have lost data and those who are about to..."

Re:Caught out.

[geek2k5]

They need to have an issue with "102 Dumbest Moments in Business" with a special award for themselves. And that cover should look like it came from a photocopied proof page.

Coming soon ...

[Stavr0]

The first issue of Business 3.0.

Better article

[pizza_milkshake]

http://www.nytimes.com/2007/05/01/business/media/0 1mag.html?_r=1&oref=slogin

Didn't you read the article?

[funkdancer]

How about doing a restore practice run whilst at it? Just kidding. Sort of. ;)

Re:Didn't you read the article?

[Nutria]

How about doing a restore practice run whilst at it? Just kidding. Sort of. ;)

You pony up the $300,000 (Large Systems cost a heck of a lot more than Windows servers) for a QA system and I'd leap at the chance to test the backups and procedures.

Re:Didn't you read the article?

[njchick]

How about doing a restore practice run whilst at it?

You mean, buckle up the governor and make another crash to test the seatbelt?

Got a

[kennylogins]

backup of that 2003 article?

The MPAA and RIAA...

[Anonymous Coward]

...may some day come running to me thanking me profusely.

I'm just providing them with free off-site backup, that's all.

I think I know how! Or at least why ...

[Bearpaw]

Maybe some White House officials had secondary email accounts on the system.

Practicing what they preach

[GuyfromTrinidad]

Since the parent company, Time named "you" the person of the year they were simply following "you" and not doing regular backups

Backups are usually no problems.

[Opportunist]

The problem comes with the need of a restore.

There are invariably 2 possible situations that can happen when a restore is in order:

Either the backups simply don't work 'cause they've been using the same media for 5+ years and nobody ever bothered checking the error messages (or because there were no errors produced in the first place 'cause the backup was done without a verify).

Or the last person who knew how to do a restore with those tapes was fired a year ago and the others just followed the backup procedure which has been documented down to the most trivial steps, but there isn't a single piece of information to be found how a restore would work. The reason for this is simply that, since that techie has to get a vacation sooner or later, too, even the secretary has to be able to do a backup (thus the documentation), but the techie alone is good enough to know the restore procedure.

This again is due to:
a) Him being too lazy to write down the procedure (hell, without his boss requiring him to write down the backup specs he wouldn't have done that either) and
b) His boss having not the foggiest idea that restores could be a tad bit more complicated than "doing a backup in reverse".

Re:Backups are usually no problems.

[raind]

Amen to that, being the lone admin can be painful.

Re:Backups are usually no problems.

[Anonymous Coward]

yeah ive seen that happen before.. we had to restore from a tape once.. we even had good documentation on how to do it.. however the password wasn't written down and it was required to restore.

luckily it was a simple password that was used for everything a few years earlier

Link to original article

[hargettp]

Huh, I guess I wasn't paying attention to when Slashdot turned into Digg, even though I read both. Here's a link to the original article, rather than what might be a splog. Especially since the article text was copied verbatim.

always test you

[geekoid]

recovery systems and methods.
Having a backup you can'r recover from is useless.

this happened to me too!

[yagu]

That's what I said when Byte went out of business and instead of refunding my subscription they sent me "Business 2.0". Multiple letters latter, they stopped sending the piece of crap magizine, but I never got my refund. It was only like $10, but it was mine.

OMG, this happened to me too! Instead of Business 2.0, they switched me to PC Magazine, one of the biggest and most annoying MS shills. Not only had I lost a cherished subscription, but it had been replaced with the very antithesis of what I had subscribed to. And!, I had a three year paid subscription.... I tried unsuccessfully to get a refund, and eventually just resorted to tossing the new PC Magazine issue when it arrived. What a ripoff!

Re:this happened to me too!

[siriuskase]

Yeah, they started sending me PC Magazine, too. But, IIRC, Byte was turning into one of those Shpper things where all the articles were concerned with which grey box packed the most megaflops, as if I cared. It had ceased to be the mag that taught me about PGP, Smalltalk, and similar exciting ideas.

Re:this happened to me too!

[rbanffy]

While I agree the last issues of BYTE was not up to the usual high standards, it was not even close to the low standards of PC Magazine.

You could still find great conceptual articles, decent coverage of non-PC platforms, lots of stuff about internet and such and was a much better read than the average computer magazine.

I felt the loss, deeply.

RAID =! BACKUP

[RMH101]

If they deleted the files in error, then the RAID would faithfully mirror that deletion across all physical disks...

I wonder if they run DR on a regular basis.

[TomTraynor]

I know that where I work we run a Disaster Recovery (DR) exercise every time we do either a hardware or software upgrade to prove that everything still works. If nothing has changed one is done every second year. We actually pull our off-site production tapes and restore to a new machine that is not in the same city where the current production machine resides. It may be over-kill for them, but, a test of that for them (or any other business) would be a fruitful exercise in that they prove that the backups are good and they can restore from a given point and carry on with a minimal loss of work.

For one of our server apps we actually have two laptops configured with all of the required software and we do restore production data from backups on a regular basis as we use that for our system testing on projects. This happens several times a year so we know that the backup and restore procedures truly work. It is also very cool walking in to the client site, plug in the laptop and show them that in an emergency they have a working machine very quickly. Not as fast as a server, but, it gets them a working machine until the replacement server arrives.

Check out their website: It's 'Dumbtastic!'

[rahimobius]

One of the 'Specials' on the Business 2.0 website:
101 Dumbest Moments in Business

See the video, test your Dumbest knowledge, and let us know what you think was the year's most boneheaded moves. (more)
Quiz: Test your Dumbest knowledge
videoBusiness: It's 'Dumbtastic!'

I think they might want to revise their list. I'm sure I would like to :)

I don't trust any single backup mechanism.

[Richard Steiner]

At home I back things up to my Buffalo LinkStation fileserver, but I also make secondary backups to other boxes and to the same box on which I work (different disk).

At work I tend to back the UNIX stuff I'm working on to my PC, my mainframe files to my PC, and my important PC stuff to either the UNIX box or to CD-R. The UNIX stuff is also in CVS.

I've invested a year or more in some of this software, and I don't want to lose it even if the entire data center fails.

Business 2.0wned

[Anonymous Coward]

SSIA

I can't believe I was the first to post that.

I wonder if the IT dept

[Cythrawl]

Felt a great disturbance in the Server, as if millions of files cried out in terror and were suddenly silenced...

Took about 20 minutes actually

[N.Muntz]

I hadn't had my morning coffee yet...

I'm gonna subscribe...

[rickb928]

Their next article on backups and recovery will be damned good...

And their next review of the backup software they use won't be written by the vendor's PR flack...

We can't backup, its too expensive.

[Stu101]

This is my story and I bullshit you not! I work for a manufacturing company, the second largest in its field in the world. Great. However the boss really does not like spending money. We eventually got a backup system using offsite backups (with a special client) and it seems to work ok. However, when it got to 100 GBs I was told to start pruning stuff. So I did. Long and short of it, even with the most important files backed up, we still have most things not backed up. Basically I have almost half a TB of data that I am not allowed to back up because its expensive. I can only backup 5 days worth of data as they are unwilling to pay anymore money for it. The fun will come when someone wants a restore from last year. This people, is the reality sometimes. Me, well, I really dont care anymore. Im sick of having servers, important, mission critical machines sitting on single IDE disks. We sell online, great, problem is our firewall is non redundant single IDE disk. If it goes (like it has in the past) we were down for days, loosing emails, web traffic, web orders, remote ordering systems, EDI data, remote sessions, ftp, everything. DR? the solution proposed by upper management is, oh we will buy some dells and restore. Yeah thats a good idea. After waiting a week for them to arrive, what exactly are you going to restore ? This is more typical than you think, unfortunatly. Im just the guy that has to make do with what i can. No doubt when it fucks up I will be blamed.

Re:We can't backup, its too expensive.

[Anonymous Coward]

i call bullshit. too expensive? so they would rather lose a days worth of sales (at minimum), then spend $200 on a 750GB hard drive in a usb case to offload stuff and throw it in a cabinet?

Re:We can't backup, its too expensive.

[Fulcrum of Evil]

No, they'd rather not buy a proper backup system (not a HD in a file cabinet) and pretend that everything is hunky dory.

Re:We can't backup, its too expensive.

[photomonkey]

Probably been said already, but it sounds like its time to look for a new job.

Best of luck.

Re:We can't backup, its too expensive.

[Stu101]

Man, I'm lookng! Problem is, although the tech is uber poor, the wages arent bad. Although yes, im looking. It's so broken I dont even try to fix it because its a case of ego of upper management over sanity. I have a mail server with 2% free disk space, which will crash in the next 48 hours but no one will give me the go ahead to either purge 4 year old accounts (I kid you not) or add extra disks, as it costs money.
Im waiting for the fallout. Upper management have known for 3 weeks it was heading a bad way, but they just arent interested.

Re:We can't backup, its too expensive.

[Maltheus]

i call bullshit. too expensive? so they would rather lose a days worth of sales

I absolutely believe that. I work for a major company that everyone is familiar with and I've been making this argument for years. In fact, it's even worse than that. We're losing data because we're working with 10 gigabyte disks that fill up before we can even process all the data we need to process. I use to bring up the fact that a 500 gig disks go for $150 and I should know, since my salary has paid for quite a few. Management always trails off with, well, then we'd have to justify it and that's a long a difficult process. Meanwhile, we give literally millions away to our competitors (and often even our corporate secrets) in exchange for nothing more that the privilege of being able to say we're partnered with them.

It has nothing to do with right or wrong or even the money, I can even convince management that it's right. It's just that large organizations are so bureaucratic, that they can't even get the simplest things done. Managers spend every waking moment in useless meetings and don't have time to argue for things that aren't exciting to their bosses. I've simply given up trying to do things right. It's a paycheck and I'm only rocking the boat but trying to protect those over me. I get the pleasure of having things done right through taking care of my personal server, workstation and HTPC. My backups, will NEVER fail.

Mirrored Servers

[boristdog]

Mirror your servers, not just your data.

One crashes, you reassign an IP and everything cruises along as before.

I've had to do this TWICE in the past year. It's a real timesaver.

Paging Jerry Seinfeld

[shrubya]

Jerry: I don't understand. Do you have my data?
IT: We have your backup, we just can't restore it.
Jerry: But the backup keeps the data here, that's why you have the backup!
IT: I think I know why we have backups.
Jerry: I don't think you do. You see, you know how to MAKE the backup, you just don't know how to RESTORE the backup. And that's really the most important part of the backup: the restoring. Anybody can just make them.

Read All About It: Special July Issue

[BoRegardless]

All about backups & Verification of Server Archives...

From first hand experience.

Re:Nelson Says

[MobileTatsu-NJG]

"I think I speak for most of us when I say .... (Score:-1, Redundant)"

Ha ha!

poor guys

[HazMathew]

Time to fire the infrastructure team :\

My Motto

[Cassini2]

My Motto for Backups is: "I have never had a backup that failed to loose information." or "All my backups have failed to restore in the critical moment."

After a few backup/disaster/restore cycles, I learned my lesson. A backup is only good if you check that you can actually restore it, and are able to restore it easily with freshly purchased computer equipment. Always assume that when the thieves/flood/disaster comes, you are left without any equipment whatsoever.

When disaster occurs, you need a credit card with a big limit, and the ability to restore onto the hardware that you just purchased that morning from the local Computer Store/Staples/Best Buy. Those expensive tape backup drives are difficult to replace quickly. If you use one, you need an additional tape drive off-site that works. Further, make sure you have the disaster recovery software and operating system software with you off-site.

A few hard drive images stored on portable USB drives can be a really handy component of the emergency backup strategy.

Incompetence

[Unique2]

Saying the "editorial system crashed" is the modern day equivalent of "my dog ate my homework". To all the PHBs out there, when you read a phrase like that I want you to replace it with "I am too incompetent to do my job!". I'm tired of hearing people talk about computer systems as if they are some wild, uncontrollable magic when anyone who have worked in I.T for long enough knows that exactly the opposite is true -- the only difference is knowledge of how the system works.

Word Police

[saltydogdesign]

smarmy: adjective ( smarmier , smarmiest ) informal - ingratiating and wheedling in a way that is perceived as insincere or excessive : a smarmy, unctuous reply.

SOX

[deets]

What I want to know is how they got that far with SOX auditing? We have to verify EVERYTHING for them.

Re:How does this actually happen? - I know

[Anonymous Coward]

I work for an IT department that had a serious crash that resulted in lost data.

Contracts, reference materials, source code, passwords, you name it, some of it was lost.

For us it happened because a tech without a lot of experience set up a new server many years ago. As years passed, he gained competence by leaps and bounds and the server he set up also gained a role as being relied on much more than it was originally expected to.

One day a drive crashed (hard) and it was then we discovered that it was RAID 0. Yes, the wails of anguish were painful to listen to. The tech was sorrowful but somewhat supported by the fact that management knew they were sending a rookie when he set it up the first time and also by the fact that nobody ever checked to see if the rookie had done the job right. Of course, he would never make that mistake now, but we all make noob mistakes when we're noobs.

So we sent the raid set out and they recovered the data, reconstructing piece by piece but many files were damaged beyond repair, some probably forever lost and the recovery process took a month when our department had come to rely on it for daily interaction.

Backup time right? Nope, it turns out that the backup software had been so overburdened that the department which managed it had started making judgement calls about what needed to be purged and what needed to run. Everything critical was on tape it was reasoned... but recall this server was not originally intended to be a critical server, and thus had no tape backup. The disk based backups had long since been purged and this server was (still considered non-critical by the people managing the backups because they'd never been told differently) without any backup, save those made before its role changed, which for all purposes were useless.

That is how it happens.

I see a lot of comments saying that you must test your backups, you must make sure your backups are successful but I don't see a lot of comments relaying this first rule of backups: MAKE SURE YOU ARE BACKING UP WHAT IS ACTUALLY NEEDED.

Loosers...

[gweihir]

Nothing else. Backup has to be tested regularly. And it costs money. These people deserve exactly what they got.

sad but true

[Anonymous Coward]

Too bad they don't have a rating for this other than funny. I've seen this mindset in too many companies regarding disaster recovery. Unfortunately, the IT guy ends up holding the bag when things go south.

Emergency flashlights

[geek2k5]

I suspect that, in the not too distant future, we'll be seeing instances where backup media gets destroyed by those battery free emergency flashlights that use simple shaking and a capacitor as a means of power. They usually have a warning saying not to have the things within a couple of feet of magnetic media.

If you DO have one of those flashlights in your desk or vehicle, check the instructions on them. You might be slowly killing any backup mag media you have around it.

Backups and restores

[geek2k5]

Lots of people do backups because they are relatively easy. But doing a restore, especially with expensive hardware, can be a major pain because you may not have a test machine to restore it on.

Then you need to add in the time factor of the restore. I remember one case where the backup and restore worked perfectly, but the amount of time spent restoring things was eating into the time available for running payroll. We had to do a partial restore of essential payroll files, and let the rest hang, in order to get payroll out on a timely basis.

Oddly enough, the machine was going to be upgraded to RAID-5 a couple of months after the crash. There had been a little complaining about the costs of the upgrade before the crash. That complaining disappeared about half way through the restore process.

CYA for Management Mistakes

[geek2k5]

If management is going to brag about cost savings, make sure that you get documentation on their comments and your warnings. That way, if/when things turn to slime, you can put it in your resume that you tried to warn them.

This may be needed for your personal recovery plan. It may also be needed if lawyers get involved and you end up facing charges.

Re:CYA for Management Mistakes

[qwijibo]

Our project manager is drawing up a risk memo about the backup and system maintenance problems. I seriously doubt anyone will acknowledge, let alone sign it. I don't think being in any way associated with worst-IT-practices-EVER is something anyone would put on a resume.

However, there are dozens of people who are aware of the problem and have been in on discussions with our VP and SVP when the risks were discussed. Our technologies group always references the SarbOx boogeyman as the reason to be afraid of everything, but our management considers malicious noncompliance to be some sort of sport. As a contractor, I don't think there's much they could do to me in a civil or criminal liability sense. And it's not like they would want to chance me having to testify in open court about all of the things that I know. =)

Backup stories

[dpbsmith]

That's the usual fly in the ointment with backups. Backups are difficult and expensive. Management always thinks they can make them easy and cheap by buying some automated solution. But the main cost of the backup is regularly testing the backups... which can only be done properly by doing a full restore... which requires available disk space equal to the size of the backup, and hours of operator time.

Story #1. Fortune 500 company. Lost some source. Big brouhaha. Edict went out: all files are to be backed up to diskettes and the diskettes sent to offsite storage which the management had contracted for with an outside firm. It took a lot of extra time, but people did it. After about two years, an important server with source code for a major product crashed. Developers tried to get the source back from offsite storage. It turns out that nobody at any point had taken any responsibility for cataloging, identifying, or indexing the diskettes. The diskettes might as well have not been labelled: the developers couldn't identify what diskettes were needed, and the offsite storage firm couldn't have retrieved them if they had.

Story #2. Medium-size scientific research organization with a Digital 11/70 running RSTS. Enlightened manager pays operator overtime pay to stay late three nights a week and do backups. Backups are performed with the "verify" option enabled. Tapes are placed in a fire-resistant tape vault every night. But no actual restores are performed. Database (Oracle, in the days when Oracle Corporation's name was still Relational Systems, Inc). is corrupted. A restore is attempted. It transpires that this version of Oracle uses the maximum record length for its files, which happens to be 65,536 bytes, and the Digital-supplied backup-restore utility... you guessed it... has a bug with records of that length. Yep. Writes 0 bytes, verifies 0 bytes.

Story #3. I worked at a place that recommended that individual developers perform individual backups using a cartridge tape system and some standard PC software. I set it up. There were two "verify" options. One used the cartridge system's read-after-write feature to read every block as it was written. The second performed the entire backup, then verified the entire backup in a second pass. Took twice as long, of course. I opted for the second method. The problem was: more than half the time, the verify would report one or two errors. And for some reason, probably efficiency of use of the tape, it didn't write file by file, it munged them into blocks. And it didn't even report the names of the files affected. Just "2 errors were encountered" or something like that. So, when that happened, I didn't see that a rational person had any alterative except to perform the whole backup again. And more than half the time, it would report a couple of errors the second time, and...

When I asked colleagues about this, it turned out that I was the only one ever to have picked the second verify option. Everyone else had picked the read-after-write-verify option, "because it was faster."

And told me not to fuss because "if it was only a couple of errors, the chances they were on files you needed to recover was too small to worry about."

coulda shoulda woulda

[flyingfsck]

used ext3cow versioning file system?

Bad old days

[jhines]

In the bad old days, one had to periodically backup, init the drive, and restore it, on a periodic basis for defraging the disk. This was before there were programs to do this automatically, and online.

This also had the effect of testing the full backup procedures. Not to mention spending a weekend at work.

Re:Bad old days

[Mazin07]

If the restore failed, wouldn't you be stuck with a bad backup and wiped original?

New name for the magazine

[Anonymous Coward]

Instead of Business 2.0 the June issue will carry the new name of the magazine:
Backups 101

Re:Better article (By a little)

[geek2k5]

It is a little bit better, but it still doesn't say much about the types of systems involved, both at the primary level and the backup/restore level.

If you are running relatively low cost stand-alone servers, then practice restores can be relatively easy with test machines.

If you are running something that is closer to a mainframe, practice restores can be hard because you may not have a test machine.

AND THEN THERE IS THE LITTLE GUY

[IamWhoIam]

Although most of the comments have been regarding corporate back ups of huge amounts of data. Backing up data is just as important for the Home EU. I can't count how many times I have had customers bring in their units with fried hard drives that had bunches of irreplaceable pictures, and documents of all sorts they had downloaded, or created but had failed to burn to discs, or put on flash drives. Sometimes I have been able to get the drive working long enough to copy said files, but in most cases not. It is a message I preach and preach and preach, is a .25 cent disc, and 30 min. of your time to much to pay for these memories????

Obligatory

[PPH]

Nothing to see here. Move along.

Re:Nelson Says

[nyclinix]

Mirrored disks are great protection against hardware failure of the disk only. If the server crashed due to a software bug or human error it would not be hard to imagine both copies of the data being damaged. As an earlier poster pointed out - the data is critical but only for a short period of time and then becomes out-of-date. A disk to 2nd server or disk to tape backup is necessary is this type of environment, it is similar to a large development project. Constantly changing data needs to backed up. Anything that is backed up needs to be validated by testing the restore and recovery process. A good trick is to backup the data and restore it to a different partition / drive / LUN with a different name and see if the application recoginizes the data.

Wrong magnetic principle?

[Krishnoid]

From the electromagnetic shielding wikipedia entry:

The shielding can reduce the coupling of radio waves, electromagnetic fields and electrostatic fields, though not static or low-frequency magnetic fields. (A conductive enclosure used to block electrostatic fields is also known as a Faraday cage.) The amount of reduction depends very much upon the material used, its thickness, and the frequency of the fields of interest.

Unless the safe was made of mu metal, I suspect the kettle could have demagnetized the tape if it was producing a low-frequency or static magnetic field.

Slowly recover?

[bogie]

The data recovery services I've worked with , Ontrack etc, can respond same day. If you can wait a few days, days not weeks, you will probably only get nicked ~$2,000. Unless the drive is physically damaged the entire process would be trivial for a company of any means.

I'm sure your right about the deadlines being a huge problem if you miss them, but unless its just literally about to print in most situations a recovery service should be able to help you out.

INEXCUSABLE.

[RyuuzakiTetsuya]

Setting up a regular, daily or weekly backup while not necessarily a super simple task, is pretty goddamn trivial. Esp. for a large business like Time.

Software to Blame?

[triso]

Soooo...What kind of software was running on the server machine(s) that failed... Linux, OS X, BSD, Windows? Also, what software gave empty or unusable backups without error or warning messages?

Tape library software

[Nutria]

That should catch situations in which a tape library wraps back on itself. :-)

???? Please clarify.

Re:Tape library software

[dgatwood]

Backup server A, compare server A, backup server B, compare server B: if tape library and/or support software for a tape library is stupid and starts reusing backup tapes, server A backups could get overwritten.

Backup server A, backup server B, compare server A, compare server B: if tape library and/or support software for a tape library is stupid and starts reusing backup tapes, you'll instantly realize that backup A was overwritten.

This was one of the problem scenarios somebody mentioned experiencing, and quite possibly the most horrifying failure case other than "tapes won't read after a month".

Re:Tape library software

[Nutria]

Backup server A, compare server A, backup server B, compare server B: if tape library and/or support software for a tape library is stupid and starts reusing backup tapes, server A backups could get overwritten.

The OpenVMS tape library software isn't that stupid.

You know, I'm starting to get a really bad feeling about the enforced migration off of VMS...

it's called backup software, not restore software!

[Gunstick]

it's called backup software, not restore software!
so it does backups. that's all.

You would be better off to buy a restore program. That usually would need to have a backup part build in to actually ba able to do the restore.

No! No! Not on personal computers!

[sbjornda]

and all the authors ought to have at least rough drafts of their stories on their personal computers.

Nonsense. They shouldn't allow files to be stored on their personal computers. Local hard drives fail all the time. That is why all files must be stored on network drives so that they can be properly ... er .... backed ... up.... :-$

--
.nosig