When Does Data Backup Become a Full Time Job?

nasteric asks: "Myself and 5 co-workers assume a number of responsibilities at my current job, including monitoring our tape backup jobs that run nightly. We do a good job of keeping non work related items (mp3's, wav files, etc.) off our servers, but our users keep eating up disk space very quickly. File storage, along with the fact that we have numerous projects 'going live' that each require one or more servers means more investment in our backup solutions. Fortunately, we have the capital to expand our backup solution (media, drives, autoloaders, software, etc.) but my boss cringes when I suggest hiring a full-time person to handle the backups. Ensuring the integrity of our nightly backups is critical, but my teammates are being spread thin due to the rapid expansion of our company. We really feel the best solution would be dedicated backup person. We currently backup approximately 3.5 terabytes of data and our enterprise expands to over 4 states. Does anybody have any suggestions as to when data backup becomes a full-time position? Are there any resources that specify when a full-time person dedicated to data backup should be hired? It would be nice to have some resources to refer to when proving my point to my boss."

Comments

Create a Whole Department!

[Sunlighter]

The Department of Redundancy Department! (With apologies to Richard Lederer...)

Obvious?

[Perdo]

When the cost of losing data exeeds the salery that would be paid to a dedicated IT guy

-or-

Backup will be taken seriously right after you loose a good chunk of data.

Then your boss will hire two people. One to do backups and one to replace the guy that cheesed the last one.

Re:Obvious?

[rasjani]

• "Then your boss will hire two people. One to do backups and one to replace the guy that cheesed the last one."

The Guy that cheesed the last one would then be the Boss himself ? =)

With all due respect...

[aaarrrgggh]

Is there a business need for the level of backup that is being provided? Specifically, is the backup strategy in-line with the value of the data? No timeframe is given for your 3.5TB, so it is difficult to discern your exact needs.

Maybe the easier battle to fight is to reduce the level of data backup, and show how it can save the company money. Chart the growth in number of tapes in your backup journal, and see if it is getting out of hand.

Of course, if you ever DO have a problem, you will be fired, but... what are the odds? ;)

automated solutions?

[Tumbleweed]

Have you looked into ways of only backing up data that really _needs_ to be backed up? Perhaps a backup solution that is configurable enough to block out mp3's and suchlike? Make the users do some of the work, or at least configure their machines so that the software they use saves files in places that get backed up automatically? (yet is still subject to the non-mp3,etc filters)

There's no way that this kind of thing *can't* be automated with enough work. The amount of billable time spent to get such a system going has to be much less than hiring a full-time backer-upper.

A place I used to work at had some shared drives on the servers - if you wanted your data backed up, you fricking saved it to those shared drives (dedicated folders for each employee & project), or else you didn't get your data backed up. And if something happened to your data that wasn't backed up, it's your problem (and thus, your ass).

Just a few thoughts.

Just a thought.

Re:automated solutions?

[madajb]

Interesting, but it doesn't work like that in The Real World(tm).

In The Real World(tm), users bitch and moan to their boss, who bitches and moans to his boss, who will then bitch and moan to your boss, who will bitch and moan to you.

When this happens, you _will_ drop what you are doing and restore said file.

My solution to this problem in the past has been simple. Start writing a log, simple spreadsheet, whatever, of how many hours per week you spend working on the backups (changing tapes/checking the logs/restoring). Do the same for your coworkers. If you and each of your coworkers spend 6 or so hours a day, that's about 40 hours, or one full time person. If it ends up being only a few hours a day, it might not be worth hiring a full timer.
Of course, this doesn't take into account intangibles, like how productive you are if you keep getting interrupted every hour to restore something. Or how much you know/don't know about optimizing backup solutions.

Another, perhaps more palatable option, is the rotating shift. Just like a rotating on call schedule, one person is responsible for all backup related issues during a given week. Presuming you have an automated system, this should really be limited to changing tapes, the occasional backup, and checking the logs to make sure things didn't break in the night.

===
-ajb

Re:automated solutions?

[Tumbleweed]

> Interesting, but it doesn't work like that in The Real World(tm).

Hmm. I think I smell a Market Opportunity(tm) here... :)

Re:automated solutions?

[qurob]

Another, perhaps more palatable option, is the rotating shift. Just like a rotating on call schedule, one person is responsible for all backup related issues during a given week.

Mike "Did you switch the tapes today?"

Steve "Switch the tapes? You're supposed to be doing them this week!"

Mike "Me? This week is Marty's week!"

Marty "My week was last week! It's Dave's turn!"

Dave "What?"

Re:automated solutions?

[Tim Macinta]

There's no way that this kind of thing *can't* be automated with enough work. The amount of billable time spent to get such a system going has to be much less than hiring a full-time backer-upper.

To add to this thought, here are some ways that you could set up such an automated system:

• As others have suggested, outsource it. There are plenty of storage service providers out there who specialize in this sort of thing and they will cost less than a full time employee.
• Keep all non-OS files on a network mounted drive and build the file server(s) using RAID arrays.
• Write a short shell script that tars up your files, scps them to another machine, and deletes any older backups from that other machine. Stick said script in your crontab. Now you don't have to mess with tapes (though you may want to copy the snapshots to tape every few weeks and take them offsite to guard against fire, theft, etc.).
• ...Or if you're in an environment that isn't easily scriptable (e.g., Windows), install a peer to peer backup solution to share backups within your office. HiveCache was mentioned here on Slashdot awhile back and it does this. I've also been working on such a solution myself which is currently in alpha testing and runs on Linux and Windows (although I think 3.5 terabytes is a bit more than it could handle at the moment). Even if these solutions don't perfectly match your needs, the cost of having them customized would most likely be less than hiring someone full time. A P2P solution would also address the original problem of growth a little better than the other solutions just because as more machines are added which require backup, those same machines add to a growing pool of storage for where backups are stored (so you don't have to worry about buying a bigger RAID array because you just surpassed X number of machines, for example).

Those are a few ideas on automating the daily plumbing of the backup process. There are also the issues of streamlining the set-up and recovery processes as well, which the original poster hinted at. Perhaps if his company is growing quickly enough, it is the actual introduction of new computers or projects into the backup process and their associated set-up which is dominating the time required for the backup process. It would stand to reason that this would be much harder to automate than the daily maintenance and perhaps this is what the poster wanted to hire somebody for, although if this is the case it would probably be best to just hire temporary help until the need subsides.

Cost analysis

[TheSHAD0W]

Is the total time spent by people in your department on backup operations approaching 40 hours per week? If your department needs more manpower anyway, and the bosses are considering hiring another person, then yes, you might as well dedicate this guy to backup solutions. Specialization is more cost effective than interrupting other jobs to do the work.

Re:Cost analysis

[topham]

I suspect, by reading what has been said, and guessing at what hasn't, that the backups are probably eating everyone elses productive work, causing more than 40hrs worth of interruption in work. (Directly or indirectly, like the extra time it takes to get back to what you were working on before the user asked for spreadsheet #45, from last June, on the 17th, not the 18th.. or whatever).

The problem is of course that it likely doesn't equate to there being 40hrs worth of work if 1 person handles the backups.

Create a part-time postition

[dt23507]

Just a thought: Would your boss would be more willing to hire someone *part-time* to handle the backups?

Re:Create a part-time postition

[sql*kitten]

Just a thought: Would your boss would be more willing to hire someone *part-time* to handle the backups?

Probably not. You would want to share responsibility for backups between the entire team on rotation, so that anyone can do a restore at 3AM without assistance if necessary. Don't forget that the point of the exercise is disaster recovery, no-one sane does backups just for the sheer hell of it.

You could hire a part-timer to come in and change tapes, sure, but not to do anything that requires thinking.

Price out automated backups and charge it back

[toybuilder]

Price out the cost of different automated backup systems, figure out the cost per MB, and then "sell" the different quality levels of storage the teams that are gobbling up disk space.

Oh, oh!

[Associate]

Hire me! Please! I can do mind numbing tasks for hours on end. As long as I can read /. between tapes. I work cheap too. I think $45k sounds good.

Re:Oh, oh!

[Anonymous Coward]

I Bid $35K :) heck... i will work for bandwidth

Re:Oh, oh!

[/dev/trash]

I'll do the job for 32k a year.

Re:Oh, oh!

[Wog]

32k?

Per year?

Kinda gives the term "will work for bandwidth" a whole new meaning, huh?

Re:Oh, oh!

[/dev/trash]

yep per year. And that's an increase from the last job I had.

Re:Oh, oh!

[Anonymous Coward]

Where you used to work your bandwidth was less than 32K/year?

Jeez, that's about 3.74 bits per HOUR!

full-time? give me a break

[anthony_dipierro]

Does anybody have any suggestions as to when data backup becomes a full-time position?

When money grows on trees.

1/2 person

[Dukebytes]

We have about 12 TB and about 45 servers that we maintain in the DC. We have one guy that does data loads and backups.

The backups takes about 1/2 of his time. We have a backup guy for him but hs is only used when the other is away.

We do both tape and "DR" types of backups. We are testing a system right now from Avamar that does snapups, etc... and is a disk to disk system - pretty fast and works good for semi static data. Might want to check them out - www.avamar.com. If you got the cash to build a good backup system - it really should only take about a half a person to maintain it.

Hope that helps.

Duke

Re:1/2 person

[GigsVT]

and is a disk to disk system

Speaking of disk-to-disk, Maxtor's MaxLineII that will be out in a couple months is aimed at the mass archive/backup market.

250 and 320GB ATA hard disks, Rated the same MTTF as SCSI, 3year warantee, $400 MSRP each for the 320GB. 10TB for under $20,000.

For 1 to 10 TB this is a cheap and good solution, combined with rsync/rdiff incremental backup smearshots onto either a Linux NAS with 3ware serial ATA or direct attached storage in the form of something from ACNC or an AXUS ATA-SCSI box.

With the direct attached storage, you could scale it up past 10TB, 4.4TB per 16 disk RAID5 with hot spare, string those together on as many SCSI channels as you need. Each AXUS 16 disk box costs about $6000, 16 of the 320GB disks costs $6400, so 4.4TB will cost about $12,000. Use software RAID0 to tie them into larger volumes if you need to.

Anyway, the potential is there for low maintenence, very cheap, and automated backups using this roll your own solution.

I can't wait until the 320GB disks come out!

Just a couple of quick thoughts

[Gudlyf]

Have you considered outsourcing backups? Granted, I have absolutely no experience in utilizing such services, but it might have a better ring to it than "new hire" to your boss.

Also, how often do you find you're restoring files? If it's more often than normal, and you have the money to spend (as you say), then what about using a snapshot solution? Keep hourly (up to, say, 8 hours) and daily (3 days?) of snapshots on your volumes and the users can go retrieve their lost file very easily on their own.

Re:Just a couple of quick thoughts

[sql*kitten]

Have you considered outsourcing backups? Granted, I have absolutely no experience in utilizing such services, but it might have a better ring to it than "new hire" to your boss.

Ah, I remember the good old days when the easiest way to do a backup was to uuencode a compress'd tar, split it into a thousand parts and post all of them to Usenet! :-)

With Your new ($) Backup Solution

[ReidMaynard]

You should need less manpower on a daily basis, not more. If you need more manpower (after it is integrated) you chose poorly.

Hiring Temp vs. Full-Timer

[moc.tfosorcimgllib]

Another alternative would be to try hiring through a temp service. You would still get a 40 hr/wk employee, but you wouldn't have to give him any of the perks of the company, plus you can fire him a lot easier as well.

Several Problems, Several Solutions

[Crypt0pimP]

If you have 3.5 TB AND the capital to get an autoloader, then by all means!!
Depending on your environment (you made no mention of what OS's, where the capacity is concentrated, etc.) you should be able to call one of the b/u software vendors directly (CA, Veritas, Legato, Bakbone, CommVault, etc) or call one of their channel partners and start sniffing for a deal. I work in this sector and man, everyone is slugging it out and dropping their prices to bottom just to win business.

Also consider some storage management software like SRM- It will let you know EXACTLY what you are backing up, who's hogging space (by user and group) and where your capacity is allocated (an mis-allocated)

The short version is this: With some SRM tools and a good automated backup system, you WILL reduce your workload. I know, I set this stuff up for a living.

Disclaimer: I am a tech for a Computer Associates reseller. I am biased towards their products, but will be the first to admit that they sometimes do suck. Hard. Like turbine of an F-16 hard.

If you'd like to, I will give you a format in which to present your Request For Quote (RFQ) to the various vendors.. It will give you an unbiased measuring tool for all that FUD and MarketingSpeak.

slineyp (at) hotmail (dot) com

To put your problem in proportion.

[jotaeleemeese]

My last 2 jobs have been in big multinational corporations comprising hundreds of servers and thousends of workstations. Several terabytes of data were backed up each day.

Both companies had a backup team. One was formed by two people, the other by three.

If your team is investing close to 40 hours per week on backups (thus mking you consider to hire somebody for that purpose) I would first look at what tools I am using for backups and if those tools fit the job. One thing that would be telling you if the tools or the understanding of them are not good enough would be if you find yourself doing backup work that has little to do with backup tasks (fixing problems with software or hardware, finding tapes, etc).

If you feel you have that you have the right tools, you may want to invest in training first. People using software to its full potential is more productive.

Another thing that may help is to designate one person responsible for backups in a rota basis (weekly, beweekly) so hecan concentrate fully on this task mainly.

Failling all that, then yes, perhaps you may need somebody. The point I am trying to make is that you have to try other things first.