Slashdot Mirror
Best Backup Server Option For University TV Station?
idk07002 writes 'I have been tasked with building an offsite backup server for my university's television station to back up our Final Cut Pro Server and our in-office file server (a Drobo), in case the studio spontaneously combusts. Total capacity between these two systems is ~12TB. Not at all full yet, but we would like the system to have the same capacity so that we can get maximum life out of it. It looks like it would be possible to get rack space somewhere on campus with Gigabit Ethernet and possibly fiber coming into our office. Would a Linux box with rsync work? What is the sweet spot between value and longevity? What solution would you use?'
Holy crap we're approaching the need for an Ask Slashdot FAQ. I feel old.
Try one of these babies on for size. 67TB for about $8,000.
There's a full parts list and a Solidworks model so you can get your local sheet metal shop to build cases for you.
Talk to a mechanical engineering student on campus, they can probably help with that.
A couple of details you'd need to fill in before people could give legitimate advice.
What's the rate of change of that 12TB. Is it mostly static or mostly dynamic. I would assume it's mostly write once read rarely video but maybe not.
Do you have a budget ? As cheap as practical or is there leeway for bells/whistles.
Is this just disaster recovery. You say if the station gets slagged you want a backup. How quickly do you want to restore. Minutes, hours, next day ?
Do you need historical dumps ? Will anybody want data as it existed last month ?
Is it just data you're dumping or some windows App complete with Windows registry junk that needs to be restored (don't know anything about Final cut pro)
If you just want to dump data and restore isn't critical, you just need to be able to do it in some time frame then sure rsync'ing to some striped 6 (or 12) TB SATA array is plenty good.
Does your university have a backup solution you can make use of? The one I work at lets researchers onto their Tivoli system for the cost of the tapes. I think I've got somewhere in the neighborhood of 100TB on the system and ended up being the driving force behind a migration from LTO-2 to LTO-4 this summer. If you are going to go and role your own and use disks, I'd recommend something with ZFS - you can make a snapshot after every backup so you can do point in time restores.
Also, I'd recommend more capacity on backup than you have now to allow versioning. I was the admin for a university film production recently (currently off at I believe Technicolor being put to IMAX) and I've lost track of the number of times I had to dig yesterday's or last week's version off of tape because someone made a mistake that was uncorrectable.
Actually, I'd suggest using OpenSolaris so that you can take advantage of ZFS. Managing large filesystems and pools of disks is *stupidly* easy with ZFS.
You could also do it with Linux, but that would require you to use FUSE, which has a considerable performance penalty. I'm not sure about the state of ZFS on FreeBSD, although I imagine that the Solaris implementation is going to be the most stable and complete. (For what it's worth, I've been doing backups via ZFS/FUSE on Ubuntu for about a year without any major problems)
-- If you try to fail and succeed, which have you done? - Uli's moose
You may want to check out rdiff-backup also. It produces a mirror like rsync, and uses a similar algorithm, but keeps reverse binary diffs in a separate directory so you can restore to previous states. However, because it keeps these diffs in addition to the mirror, it's better if you have more space on the backup side.
There are a few different frontends/guis to it but I don't have experience with them.
Everything your TV station broadcasts will automatically be backed up here.
Don't use rsync to make backups. Because you don't just want to backup against spontaneous combustion â" inevitably, there will be accidental deletions and the like occurring in your studio. If you use rsync (with --delete, as any sane person would, otherwise your backup server will fill up in days, not years), then when some n00b runs `rm -rf ~/ReallyImportantVideos`, they'll be deleted from the backup too.
Remember that pro photography website that went down, because their "backup" was a mirroring RAID setup? Yep â" they lost all their data on one fell swoop when somebody accidentally deleted the whole lot. Don't make the same mistake.
Use an incremental backup tool. Three that come to mind are rdiff-backup, Dirvish, and BackupPC.
I would think that rdiff-backup would suit your needs best. I currently use BackupPC at home, which is great for home backups, but I think that it's overkill (and possibly a bit limited) for what you want.
Hope this helps!
Have each student create their "own TV station" as part of their degree requirement - no matter the area of study. Similar to research essays, you'll get the following results: 1) students who completed the assignment with no outside assistance 2) students that copied certain small portions of the data you are backing up and presenting it as their own 3) students that plagiarize everything - yes some students will debate that the same content the TV station has accumulated over the years - all 12 TB - is actually their original work.
As this data appears on the University network, the entire TV station will be backed-up in a local "Cloud". And if these types of assignment become popular at other universities, you can expect to find redundant off-site backups. By this point, the 12 TB will appear on BitTorrent (and probably on Newsgroups and IRC for the dedicated plagiarists). A full restore will only take a few days - as long as the full 12 TB is seeded.
Backups for UNIX, backups for Windows, and backups all across the board almost require different solutions.
For an enterprise "catch all" solution, I'd go with TSM, Backup Exec, or Networker. These programs can pretty much back up anything that has a CPU, although you will be paying for that privilege.
If I were in an AIX environment, I'd use sysback for local machine backups and backups to a remote server.
If I were in a general UNIX environment, I'd use bru (it used to be licensed with IRIX, and has been around so long, it works without issue with any UNIX variant.) Of course, there are other solutions that work just as well, both freeware, and commercial.
If I were in a solidly Windows environment, I'd use Retrospect, or Backup Exec. Both are good utilities and support synthetic full backups so you don't need to worry about a full/differential/incremental schedule.
If I were in a completely mixed environment, I'd consider Retrospect (it can back up a few UNIX variants as well as Macs), Backup Exec, or an enterprise level utility that can back up virtually anything.
Please note, these are all commercial solutions. Bacula, Amanda, tar over ssh, rsync, and many others can work just as well, and likely will be a lot lighter on the pocketbook. However, for a business, some enterprise features like copying media sets, or backing up a database while it is online to tape or other media for offsite storage may be something to consider for maximum protection.
The key is figuring out what you need for restores. A backup system that is ideal for a bare metal restore may be a bit clunky if you have a machine with a stock Ubuntu config and just a few documents in your home directory. However, having 12 terabytes on Mozy, and needing to reinstall box from scratch that has custom apps with funky license keys would be a hair puller. Best thing is to use some method of backups for "oh crap" bare metal stuff, then an offsite service just in case you lose your backups at that location.
Figure out your scenario too. Are multiple Drobos good enough, or do you need offsite storage in case the facility is flooded? Is tape an option? Tape is notoriously expensive per drive, but is very economical once you start using multiple cartridges. Can you get away with plugging in external USB/SATA/IEEE 1394 hard disks, backing to them, then plopping them in the Iron Mountain tub?
The hard drives are desktop class, not designed for 24x7 operation. Not designed for massive write traffic that server backups generates.
Latent defects on disks are a real concern.
You write your data to a disk, but there's a bad sector, or miswrite, and when you go back later (perhaps when you need the backup), there are errors on the data you are reading from the disk.
Moreover, you have no way of detecting it, or deciding which array has recorded the "right value" for that bit...
That is, unless every bit has been copied to 3 arrays.
And every time you read data, you compare all 3. (Or that you have two copies and a checksum)
Well, the complexity of this redundancy reduces the reliability overall, and it has a cost.