Slashdot Mirror
Why Do Companies Backup So Infrequently?
Orome1 writes "Businesses are on average backing up to tape once a month, with one alarming statistic showing 10 percent were only backing up to tape once per year, according to a survey by Vanson Bourne. Although cloud backup solutions are becoming more common, still the majority of companies will do their backups in-house. Sometimes they will have dedicated IT staff to run them, but usually it's done in-house because they have always done it like that, and they have confidence in their own security and safekeeping of data."
Portable HD is cheaper and faster, even for stacks of them. Small businesses may be using a bunch of these in place of tapes.
I run backups at least once per week, and if I make major changes to my system (I self web host), I backup several times per week. This week I lost a 500GB drive. I used ddrescue to recover all but 19MB, although after the filesystem was restored, I lost more than 19MB. Backups saved me huge amounts of time and trouble. The system was down for nearly three days. Data recovery took time, getting a new drive and installing an OS on it, and then rebuilding the system. Its been back for two days. I don't know how companies backup only once per year. Its like they are asking for disaster.
After one failure the costs alone will be realized. I personally never worked in a place that did not have a daily backup. Hell, I knew someone fired because he messed up a daily backup for 2 days but with no data loss just because if it *did* happen it would be catastrophic.
The only place I knew who did it weekly was a small computer shop.
Are companies this cheap today run due to excessive cost accounting cutting and right sizing? I find this too hard to believe
http://saveie6.com/
It's expensive, so management does not really want to pay for new tapes, a disk-based system or cloud backup. It requires personnel, which management does not want to pay for either. It's boring for the persons involved (who likes testing their backup?).
Most companies have no risk management, and no clear picture of the risks their business faces.
The result of "intuitive" risk-non-management is that the usual human flaws have full impact. Basically, aside from a narrow middle ground, all risks are wrongly estimated.
Assorted stuff I do sometimes: Lemuria.org
The result is that people just expect them to work flawlessly -- not to fail. They also ignore other risks. I put in a machine at some customers a couple of years ago. They did not want to pay extra for backups -- ''yes, we will do that later''. They knew that I configured mirrored (RAID 1) disks. I set up a backup from one part of the disk to another and reminded them every couple of months that they would loose everything if it was destroyed or stolen.
Then a few weeks ago another unit on the industrial was torched -- arson. I have finally pursuaded them ... I am putting in another machine on the far side of their factory that will take a daily rsync, and USB plugin disks that they will backup to weekly & take off site.
These guys are not stupid in what they do professionally, they have an annual turnover of more that £1 million. Why does it take a fire at a neighbour to make them see sense ?
Seriously. Most people aren't willing to put in the effort to determine just how bad things would be if they had to resort to their (often inadequate) backups, and therefore they aren't willing to pay the time and capital to get adequate backups.
If you want your company to get better backups, run a simulation of what would happen if something failed. What's the best recovery you could do? What business would you lose? Then calculate the probability of that failure occurring, and be generous.
Insert self-referential sig here.
especially when combined with 'find' and 'xargs', in what is supposed to be a simple task.
If you don't, you'll do something like what i just did ("worst typo in a decade"): you see, i was trying to update emacs and wanted to purge all the .elc files from ~/.emacs.d
Unfortunately, through a bad typo, some miss-applied keyboard shortcuts, and rushing through without mounting a scratch monkey... what actually ran was effectively "find ~/.emacs.d | xargs rm".
accidently deleted the 'grep'. Oops. 15+ years of elisp/etc destroyed.
Was it backed up? Nope! Been meaning to check it all into git, but always put it off as a "minor, unimportant" task I'd get to later. Of course, we all think that way up until the disaster hits...
*sigh*
Ce n'est pas une signature automatique.
RAID is not a replacement for a backup.
RAID will safeguard you against the failure of a single disk (if and only if you monitor the system and replace disks as they fail), but backup will give you back your data as it was before your application destroyed it or your user deleted it. That is something completely different.
Mod me down for trolling all you want, he deserved the pun after not making backups for 15 years consecutively.
I was promised a flying car. Where is my flying car?
I'm not sure how everyone gets so ecstatic about those cloud backups. When we would need to send all our data over the internet connection it would take an unworkable amount of time to complete the backup.
Even to the local LTO-4 drive, which runs at over a gigabit per second, the backup takes an appreciable amount of time.
Cloud backup may be good for a 3-man company doing document editing, but with the amounts of data that are common these days, and the speeds of internet connection that you normally have, I don't see it as a realistic possibility.
I've always wondered if /. bothers to back up the stories and posts...
Anyone here know?
How about for other 'trivial' sites, like reddit, youtube, etc?
To most PHBs a computer is a toaster. That's right, an appliance. Nothing more. Most have no idea of the nursemaiding a computer needs, or how vulnerable company data is when it's reduced to 0s and 1s. Until the "unthinkable" happens and the toaster breaks.
Now. I used to be in the biz of disaster recovery. It was lucrative when consumers who valued their potentially lost data found me and asked me "Can you do it?" and I said "Let's have a look". My success rate was something like 40%. Surprisingly (for me anyway), my better corner was recovering data from physically damaged flash drives. Out of several dozen of those I only had one that I couldn't recover anything from. Hard disks are a different animal, and the whole thing can be frustrating when you sit back for a minute or five and consider how much easier your work would be (and how much hardware you can sell) if your client had the benefit of hindsight and someone around who knew the shit of which he spoke - so a week-long recovery project that may or more than likely may not work turns into a two hour exercise in restoring on new hardware and sending the happy client on his way. Now, that is an exercise in getting repeat business through recommendation.
As far as "accidental" deletion of data: there is only so much you can do to protect the stupid from themselves - on a network share, for instance, you can deny users the right to delete files. Job done. In an environment that is supposed to be secure, that's a good start. VSC is another handy tool but you don't need to tell the stupid that their fuckups are (sort of) covered - it breeds complacency and does little to nothing to train responsibility.
Operation Guillotine is in effect.
Offsite backups are what a lot of companies don't do. They might back up to tape, but the tapes are stored in a pile next to the server. And they never test them.
Agreed. One of the many phrases in my litany on customers backing up is "Yes, that safe is fireproof. For paper. Plastic melts at a much lower temperature than paper burns." Not only do not enough companies run backups on a regular and timely basis, but too many of the ones that do run backups don't see the need for having the storage media offsite.
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
You're so wrong ...
Tape is the ONLY way to make serious backups and do archiving once you start having serious data volumes. 50TB of data is a LOT of disks... now do daily increments, weekly full backups with 2 months retention... also single disks are always slower than streaming to LTO4 or LTO5 tapes so your backup window becomes too big to handle.
Sure, if you only have like 1-10TB harddisks are fine...
But how to do you handle backups of say 50TB of data every week? Awful lot of disks to swap, copy times will be so long you'll need to start the next weekly while the last job is still running etc etc.
Tape still has it's place. It's not the universal solution for all backup problems, but for large datasets it's still king.
I backup to backup server every hour and then spool to tape every night.
Once a week that weeks tapes get sent offsite and old 3 month old tapes are returned for re-use.
If the company's IT department is NOT doing daily backups then it's IT department is ran by an idiot.
If management will not pay for it, then they need to be told, "so everything you do today does not matter and has no value? because that is what you are saying when you say backups are not important. Hard drives are not reliable, if they crash tonight and we lose everything, how much money do we lose?"
You have to talk to them in management speak... AKA money.
Do not look at laser with remaining good eye.
"RAID is no substitute for backups. Yes, RAID5 will handle a single disk crashing."
You know how many times I have seen a raid 5 go down hard with a single drive failure? too many to count.
Raid 5 lost a drive, no problem the hot spare will swap in and the rebuild starts..... and then drive #2 fails..... all the data is now utterly hosed as it failed during a rebuild, or better yet, the freaking hot spare was an old dead drive that someone failed to pull out of the cage during the 2 years when they cruised without a IT guy.
I have seen more raid 5 arrays fail with total loss of data than I can shake a stick at. what is more fun is when you lose all the data on a massive raid 50 because the engineers at dell made the scsi cables not lock in to the sockets, and the "new IT manager" that happens to be the CFO's 21year old son was touring the server room for the first time and grabbed a cable asking, "WHATS THIS FOR?"
Do not look at laser with remaining good eye.
So a memory hierarchy emerged: local use on solid-state or disk, first-level backups on disk, and second-level backups on tape. That sounds reasonable for organizations big enough to need tape.
"Wait until your boss deletes that important document and your RAID system has deleted it on all drives in the array immediately at his request."
Do you know how RAID works? it stripes the data across ALL drives. if you think RAID5 keeps separate copies you are a little nutty.
RAID 1 is mirroring, but nobody at all uses it other than home users that are paranoid and like paying 2X for their storage space.
and to stop idiot bosses, you put in place a RCS. revision control does wonder to solve the "oops I deleted all the document files on the server" problem.
Do not look at laser with remaining good eye.
I have had SDLT tapes I recovered from the floor of a server room after sitting for 3 days under water that read just fine in a drive after being dried out.
If you get a decent tape backup it has built in redundancy, 48 tracks of data with CRC checking and redundant data written in case that CRC fails.
Granted, I'd not rely on them surviving kicked around the floor, but a real tape backup system is far, far more robust than the under $1000 garbage sold at newegg.
Do not look at laser with remaining good eye.
I expect a lot of companies are just like my mom. having never experienced data loss, she doesn't see the point of backup.
Request your free CD of my piano music.
>showing 10 percent were only backing up to tape once per year
You mean showing 10 percent were only backing up to tape once per year to a paying third party vendor....
instead of in house which is cheaper when you know what you are doing...
I love when they twist these headlines to make them more attractive..
Pretend a guy prevents the 9/11 attacks by requiring all airliners to have a bulletproof, locked cockpit door. Hundreds of millions of dollars are spent, and prices go up. Everyone complains about government regulation. The attacks never happen. The instinct was that we just wasted a bunch of money. We spent hundreds of millions on H1N1 vaccine for an outbreak that never happened. But if it did, the vaccine would have saved many lives and prevented great financial loss. But it never happens, so it's seen as lost money.
You push for backups, and you spend tens of thousands on it for years. There is a disaster eventually that you recover from quickly, but perhaps you aren't around to take credit for it. Or you restore and everyone thinks that it was supposed to because we spent so much on it. No one will think, "Well, we just saved millions of dollars of downtime because of the backup." The thinking is just, "Oh, well it was supposed to. We spent so much on it."
This kind of myopia is commonplace in the world. We can invest money to treat everyone for chronic illnesses such as diabetes, high blood pressure, and high cholesterol with really cheap drugs. We can vaccinate everyone from childhood diseases. "Oh, but vaccines cause autism." Yes, but your child isn't paralyzed from polio. The avoided disaster is never quantified; only the cost spent shows up on your calculations.
A NYC lawyer blogs. http://www.chuangblog.com/
That's what I call living dangerously and very bad advice. Even my snapshots on spinning storage are in a different building.
I've been up all night nursing an array back to health so at this point I'm very dubious about relying on onsite spinning storage for absolutely everything.
at least 25 times, SDLT tapes are durable as hell.
Do not look at laser with remaining good eye.
There's no need for a robot when all the "tapes" (aka HDDs) are all accessible while stored: http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/
(IIRC this was on Slashdot before).
But if an org only needs 25 or similar magnitude of drives you can go Dell (e.g. stuff like MD1000 or MD1220) for a many times the price/capacity. Most orgs with those needs can afford it.
If I were running my own company I'd go with Dell/etc and HDDs first, but if my required storage capacity curve goes up steeply, I'd do the backblaze thing.
Not tape. I don't trust tape, in my experience tapes fail way more than hard drives. Both the tapes and tape drives wear out quite fast - there is physical contact between the heads and the tape, and the drives make many passes per backup cycle. So you'll actually need a higher number of tapes compared to HDDs which are mostly fine spinning at thousands of rpms for years.
Maybe I've been unfortunate? So far the HDDs seem to do better for: total number of HDDs, number of failures per year vs total number of tapes (or tape drives) and failures per year.