When Does Data Backup Become a Full Time Job?

nasteric asks: "Myself and 5 co-workers assume a number of responsibilities at my current job, including monitoring our tape backup jobs that run nightly. We do a good job of keeping non work related items (mp3's, wav files, etc.) off our servers, but our users keep eating up disk space very quickly. File storage, along with the fact that we have numerous projects 'going live' that each require one or more servers means more investment in our backup solutions. Fortunately, we have the capital to expand our backup solution (media, drives, autoloaders, software, etc.) but my boss cringes when I suggest hiring a full-time person to handle the backups. Ensuring the integrity of our nightly backups is critical, but my teammates are being spread thin due to the rapid expansion of our company. We really feel the best solution would be dedicated backup person. We currently backup approximately 3.5 terabytes of data and our enterprise expands to over 4 states. Does anybody have any suggestions as to when data backup becomes a full-time position? Are there any resources that specify when a full-time person dedicated to data backup should be hired? It would be nice to have some resources to refer to when proving my point to my boss."

Comments

Obvious?

[Perdo]

When the cost of losing data exeeds the salery that would be paid to a dedicated IT guy

-or-

Backup will be taken seriously right after you loose a good chunk of data.

Then your boss will hire two people. One to do backups and one to replace the guy that cheesed the last one.

With all due respect...

[aaarrrgggh]

Is there a business need for the level of backup that is being provided? Specifically, is the backup strategy in-line with the value of the data? No timeframe is given for your 3.5TB, so it is difficult to discern your exact needs.

Maybe the easier battle to fight is to reduce the level of data backup, and show how it can save the company money. Chart the growth in number of tapes in your backup journal, and see if it is getting out of hand.

Of course, if you ever DO have a problem, you will be fired, but... what are the odds? ;)

automated solutions?

[Tumbleweed]

Have you looked into ways of only backing up data that really _needs_ to be backed up? Perhaps a backup solution that is configurable enough to block out mp3's and suchlike? Make the users do some of the work, or at least configure their machines so that the software they use saves files in places that get backed up automatically? (yet is still subject to the non-mp3,etc filters)

There's no way that this kind of thing *can't* be automated with enough work. The amount of billable time spent to get such a system going has to be much less than hiring a full-time backer-upper.

A place I used to work at had some shared drives on the servers - if you wanted your data backed up, you fricking saved it to those shared drives (dedicated folders for each employee & project), or else you didn't get your data backed up. And if something happened to your data that wasn't backed up, it's your problem (and thus, your ass).

Just a few thoughts.

Just a thought.

Re:automated solutions?

[madajb]

Interesting, but it doesn't work like that in The Real World(tm).

In The Real World(tm), users bitch and moan to their boss, who bitches and moans to his boss, who will then bitch and moan to your boss, who will bitch and moan to you.

When this happens, you _will_ drop what you are doing and restore said file.

My solution to this problem in the past has been simple. Start writing a log, simple spreadsheet, whatever, of how many hours per week you spend working on the backups (changing tapes/checking the logs/restoring). Do the same for your coworkers. If you and each of your coworkers spend 6 or so hours a day, that's about 40 hours, or one full time person. If it ends up being only a few hours a day, it might not be worth hiring a full timer.
Of course, this doesn't take into account intangibles, like how productive you are if you keep getting interrupted every hour to restore something. Or how much you know/don't know about optimizing backup solutions.

Another, perhaps more palatable option, is the rotating shift. Just like a rotating on call schedule, one person is responsible for all backup related issues during a given week. Presuming you have an automated system, this should really be limited to changing tapes, the occasional backup, and checking the logs to make sure things didn't break in the night.

===
-ajb

Cost analysis

[TheSHAD0W]

Is the total time spent by people in your department on backup operations approaching 40 hours per week? If your department needs more manpower anyway, and the bosses are considering hiring another person, then yes, you might as well dedicate this guy to backup solutions. Specialization is more cost effective than interrupting other jobs to do the work.

Create a part-time postition

[dt23507]

Just a thought: Would your boss would be more willing to hire someone *part-time* to handle the backups?

Price out automated backups and charge it back

[toybuilder]

Price out the cost of different automated backup systems, figure out the cost per MB, and then "sell" the different quality levels of storage the teams that are gobbling up disk space.

Oh, oh!

[Associate]

Hire me! Please! I can do mind numbing tasks for hours on end. As long as I can read /. between tapes. I work cheap too. I think $45k sounds good.

1/2 person

[Dukebytes]

We have about 12 TB and about 45 servers that we maintain in the DC. We have one guy that does data loads and backups.

The backups takes about 1/2 of his time. We have a backup guy for him but hs is only used when the other is away.

We do both tape and "DR" types of backups. We are testing a system right now from Avamar that does snapups, etc... and is a disk to disk system - pretty fast and works good for semi static data. Might want to check them out - www.avamar.com. If you got the cash to build a good backup system - it really should only take about a half a person to maintain it.

Hope that helps.

Duke

Re:1/2 person

[GigsVT]

and is a disk to disk system

Speaking of disk-to-disk, Maxtor's MaxLineII that will be out in a couple months is aimed at the mass archive/backup market.

250 and 320GB ATA hard disks, Rated the same MTTF as SCSI, 3year warantee, $400 MSRP each for the 320GB. 10TB for under $20,000.

For 1 to 10 TB this is a cheap and good solution, combined with rsync/rdiff incremental backup smearshots onto either a Linux NAS with 3ware serial ATA or direct attached storage in the form of something from ACNC or an AXUS ATA-SCSI box.

With the direct attached storage, you could scale it up past 10TB, 4.4TB per 16 disk RAID5 with hot spare, string those together on as many SCSI channels as you need. Each AXUS 16 disk box costs about $6000, 16 of the 320GB disks costs $6400, so 4.4TB will cost about $12,000. Use software RAID0 to tie them into larger volumes if you need to.

Anyway, the potential is there for low maintenence, very cheap, and automated backups using this roll your own solution.

I can't wait until the 320GB disks come out!