Slashdot Mirror
Best Backup Server Option For University TV Station?
idk07002 writes 'I have been tasked with building an offsite backup server for my university's television station to back up our Final Cut Pro Server and our in-office file server (a Drobo), in case the studio spontaneously combusts. Total capacity between these two systems is ~12TB. Not at all full yet, but we would like the system to have the same capacity so that we can get maximum life out of it. It looks like it would be possible to get rack space somewhere on campus with Gigabit Ethernet and possibly fiber coming into our office. Would a Linux box with rsync work? What is the sweet spot between value and longevity? What solution would you use?'
A couple of details you'd need to fill in before people could give legitimate advice.
What's the rate of change of that 12TB. Is it mostly static or mostly dynamic. I would assume it's mostly write once read rarely video but maybe not.
Do you have a budget ? As cheap as practical or is there leeway for bells/whistles.
Is this just disaster recovery. You say if the station gets slagged you want a backup. How quickly do you want to restore. Minutes, hours, next day ?
Do you need historical dumps ? Will anybody want data as it existed last month ?
Is it just data you're dumping or some windows App complete with Windows registry junk that needs to be restored (don't know anything about Final cut pro)
If you just want to dump data and restore isn't critical, you just need to be able to do it in some time frame then sure rsync'ing to some striped 6 (or 12) TB SATA array is plenty good.
That's all you need. We even use a script to create versioned backups going back six months using perl as a wrapper.
Assuming the same paths, edit to your liking. I've made the scripts available at http://www.secure-computing.net/rsync/ if you're interested. It requires the system you're running the script for have root ssh access to the boxes it's backing up. We use password-less ssh keys for authentication.
The README file has the line I use in my crontab. I didn't write the script, but I've made a few modifications to it over the years.
Does your university have a backup solution you can make use of? The one I work at lets researchers onto their Tivoli system for the cost of the tapes. I think I've got somewhere in the neighborhood of 100TB on the system and ended up being the driving force behind a migration from LTO-2 to LTO-4 this summer. If you are going to go and role your own and use disks, I'd recommend something with ZFS - you can make a snapshot after every backup so you can do point in time restores.
Also, I'd recommend more capacity on backup than you have now to allow versioning. I was the admin for a university film production recently (currently off at I believe Technicolor being put to IMAX) and I've lost track of the number of times I had to dig yesterday's or last week's version off of tape because someone made a mistake that was uncorrectable.
What solution would you use?
First of all, I love linux. Use it for my own file servers, and media machines, and routers, and pretty much everything except desktops.
That said...
For your task, I would probably just build an exact duplicate of the "real" machine and sync them nightly. Always keep in mind that if you have no way to quickly recover from a disaster, you don't actually have a backup.
That said, and if possible, I would also build the "backup" machine with more storage than the "real" machine. As someone else pointed out, you'll probably discover within a few days that your food-chain-superiors have no concept of "redundancy" vs "backup" vs "I can arbitrarily roll my files back to any second in the past 28 years". Having at least nightly snapshotting, unless your entire dataset changes rapidly, won't eat much extra disk space but will make you sleep ever so much better.
Remote storage at a provider like Backblace, Mozy, or Carbonite is a good tertiary level backup, just in case your site goes down, but you are limited by your Internet pipe. A full restore of terabytes of videos through a typical business Internet connection will take a long time, perhaps days. Of course, one could order a hard disk or several from the backup company, but then you are stuck waiting for the data to physically arrive.
Remote storage is one solution, but before that, you have to have local ones in place for a faster recovery should a disaster happen. The first line of defense against hard disk stuff is RAID. The second line of defense would be a decent tape drive, a tape rotation, and offsite capabilities. This way, if you lose everything on your RAID (malware or a blackhat formats the volume), you can stuff in a tape, sit on pins and needles for a couple hours, and get your stuff back, perhaps back a day or two.
For a number of machines, the best thing to have would be a backup server with a large array and D2D2T (disk to disk to tape) capabilities so you can do fast backups through the network (or perhaps through a dedicate backup fabric), then when you can, copy them to the tapes for offline storage and the tub to Iron Mountain.
Of course, virtually all distributed backup utilities support encryption. Use it. Even if it is just movies.
Since we're talking about Final Cut data, it's safe to assume that it's all coming from Macs. The version of cp on Mac OS doesn't take either of those options, so it's a moot point.
Time Machine is probably the way to go. It's integrated into Mac OS, and it's ridiculously easy to set up. I don't know how it scales up, but I'd be very surprised if it couldn't handle 12TB.