Slashdot Mirror
Best Way to Back Up Photos and Video?
jsalbre writes "I do a lot of digital video work, and my wife is a professional photographer. With raw DV from the video camera using up 11GB/hr, and raw images from the digital SLR using 7MB I'm quickly using up a lot of space. I currently back up all my important files each night from one harddrive to another, but I now have over 200GB of irreplaceable data (more than just DV and photos, but those make up the largest chunk) and I'm having to exclude the "less important" irreplaceable files as my backups have started failing. Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive, but with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless. Not to mention that some of my DV files are larger than even a dual-layer DVD, and it would be near impossible to automate backup to DVD. How do other Slashdotters back up their important data? I'd appreciate distinction between methods for frequently accessed files and for infrequently accessed files. Any suggestions will be highly appreciated!"
Why not make two optical backups. Store at least one in a fireproof safe. For the massive files, you might have to invest in one or two hot swappable drives you can use as 'tapes', storing one in your safe. Mirroring might help.
a poor mans .5TB raid?
File -> Print
i think you might want to take a look at tape drives
Seriously. Where have you been for the past 5 years? We've all fucked off optical media for its unreliability. Hard drives have a decent cost per gigabyte storage and onboard RAID makes it easy to just RAID 1 your drives or backup to a RAID array. Don't have drivespace? BUY ONE OF THE THOUSANDS OF FIREWIRE HARD DRIVES OUT THERE.
Sheesh. Talk about your "fucking duh" questions.
Would tape backup work for you? It's archival quality, but you get what you pay for...
And supplement that with LaCie external firewire drives.
If you don't know what AltaVista is (was), get off my lawn.
Well, it's expensive, but maybe you can find a deal on an LTO2 or SAIT tape drive on Ebay. These babies boast 200GB and 500GB of native storage respectively . The transfer rates are nothing to sneeze at either.
And as long as you store the tape properly, it should last a long time.
Have you considered compressing your video using one of the many codecs available? DivX is quite popular, and RMVB offers some of the best quality:size ratio I've seen. I understand how nice it is to be able to store raw mpeg for later use but is it really necessary for your purposes?
... is where it's at. Keep it dry and cool and it will last for a very long time.
I keep my originals (video, images) somewhere on DVD and use a smaller, screen optimized version to show to family and friends.
With that kind of Data load, I'd say good old tape drive is your best best. hard drive for frequently accessed stuff, Tape drive for long term storage. BTW, Seagate has a nice 400 GB HD storage solution for about 300 bucks which should solve your storage needs temporarily until you can get the money together for a tape drive setup.
Official Heretic from the "Church of Global Warming". Proven right thanks to whistle blowers. AGW = Flat Earth Theory
I typically take the edited footage and back it up to another miniDV tape from the computer (using my miniDV camcorder). I then lock the tape to prevent accidental erasure and store the tapes offsite. For photos, I'm taking my chances and burning them off to DVD. I also periodically make digital prints and send them to my parents and sister, who live in two separate locations. Worst comes to worst, at least they have a hard copy available should I lose the original digital version that I have on my computer.
The PC Weenies: 11 Years of Online Tech 'Too
Backup your data to a RAID if you can afford it. preferably either a RAID 1 or RAID 5 (mirrored or checksummed respectively)
I have a similar issue with photos. I have 2x 160GB drives set up in a mirrored raid array on a external FireWire HD. I use MacOS X's software raid. Not the best but cheap and effective. I'm planning on upgrading to 2x 400GB drives in the next year. I also use Flickr to "backup" my best photos.
Of course off-site backups would be best but that's a lot of data to send via wire.
--- http://homepage.mac.com/gregjsmith
You don't really say just how much those irreplaceable files are worth to you. A lot of people have things are irreplaceable until they find out how much it will actually cost them to back it up properly. Then suddenly little Timmy's first steps don't look so hot.
Go pick yourself up a xRAID or the like and back all of your files up to a nice RAID 5 system. Once a year or so do a dump to optical media but just add additional space as necessary.
(i have no experience in the matter) you should place all the un-changing files on a hard-drive which will then sit in a draw and is only plugged in when required. I have been lead to believe that this will reduce the likelyhood of harddrive failure to close to 0. Then you can setup a RAID type setup for you changing files.
Let's face it, one method won't fit all, so I hope your search proves fruitful. That said, here's what I do.
I have a 'cheap' system (sub 500) that acts as my data server. It houses 3 DVDrom drives, and a DVDRW drive, as well 1 200 GB drive. (the processor speed and ram really aren't too important, but for curiousity, it's an athlon 2000+ with 512 meg of ram). It runs gentoo, and I essentially pull the files to burn to DVD over the network weekly, and I keep the stuff I don't access alot on DVD, and the stuff I do access alot on HD -- but I primarily use the HD for holding images waiting to be burned.
War isn't about who's right. It's about who's left.
I don't care what they say, get a blu-ray disc drive!
To a hard disk in a USB enclosure. Better yet, but more expensive, to a NAS box.
http://www.walmart.com/catalog/product.gsp?product _id=3584924
Wal-mart solves all problems
I concurr. For digital media, I would definitely do two of at least two different back up strategies.
First that comes to mind is Tape backup. They store huge about of data, and are very cheap these days, and have been proven to last for a while. Keep a good backup schedule, and keep one copy of the tapes offsite.
Secondly, I'd do optical. Optical's cheaper, but it's also not as long lasting, and takes longer to make the actual back up.
Thirdly, I'd do RAID. Mirror all the files onto a second set of hard drives. If you really want to get paranoid, mirror onto two sets of drives, and once a week swap out a copy of mirrored drives from a fireproof location.
If your data is truely irreplacable, then this is a good regiment. But it's also very expensive.. so you'll have to make up your mind.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
Not from TFA, from TF synopsis:
"Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive"
They've already considered hard drives. Since he's dismissed hard drives and seemingly all forms of optical media, the only thing that I can think of for this article getting posted is that the submitter *really* wants Slashdot to tell him that "Yes, it's ok to mortgage the house to buy that new Network Appliance SAN you've been drooling over."
Seriously, you can put together a nice RAID 5 setup; if a drive fails, replace it and rebuild the array. You're stuff is automatically "backed up" from device failure--maybe not from other issues, but device failure seems to be you're main concern. On top of that, you get the side bonus of fast disk access for your DV work; and you can expand the array pretty much whenever you want for more space.
I have old full height 5.25 hard drives from the IBM XT days that STILL BOOT and still have viable data on them over 20 years later..
200gb drives can be had for under $100 on ebay.
Load them up, remove them and store them in a fire proof safe..
Problem solved..
Well, why exactly are you "backing up" DV video? My idea of DV is that you capture it once before production, then edit/etc it, and turn it into something more manageable such as MPEG2 on DVD, etc. DV is 11Gb/hour is because its hardly compressed (it uses some kind of lossy compression that I dont remember), but once you have teh video and are done editing it etc, there is no point in keeping it as DV - just encode to MPEG2/AVI/your choice of format here. And if stuff is that important to you, well, find a place that will press your DVD-Rs into silver disks, and there you have it, more or less permanent storage of your favorite data.
read here: http://hardware.slashdot.org/article.pl?sid=05/06/ 25/1720258&tid=198&tid=222&tid=137
i would also build a raid rack, however, my collection of, uh, digital videos and uh, high quality pictures... cough, needless to say. It would be very usefull.
This irks me to no end. I think back and cringe at all the data I've lost over the years. I'd kill for a look at the source of some programming projects I did in high school. In retrospect, I should have gotten a cheap black and white laser printer and printed it all out. This won't help archive pictures or DV though.
I think that we're in a strange place right now. We're able to produce digital content that has a real value to us, but we're not able (yet) to back it up on an archival medium.
Your best bet, as it stands right now, might be to use DVD's. Don't expect them to last forever, though. Buy the best (most expensive?) media you can, and store them away from light. Maybe where you keep your family photos, somewhere cool, dry, and dark. Then every couple of years, take them out and inspect them. Recopy if you feel it's necessary.
Sooner than later, someone will develop an archival method that really works. Until then, we have to use what we've got.
"How do other Slashdotters back up their important data?"
I memorize it.
Vincent J. Murphy
Spandex Justice
Lots of gmail accounts
First, there's the need to keep things around long-term. Second, there's the need to have things protected from disaster in the short term.
I once used an external firewire HD for backup, and was reminded of the importance of burning things as well when that HD went tango-uniform on me, destroying months of work.
I'd suggest looking into some sort of RAID - even just a simple mirror - for the short-term protection. That way you don't have quite as much a single point of failure that can wipe out your data, so you can do backups more because you need the space than because you need to sleep well at night.
As for the backups, optical discs are very convenient, but magnetic tape might have a longer lifetime depending on environmental conditions, and although I've seen CD-R comparisons, I've yet to see something similar for DVDs.
There are times where a high-capacity removable hard disk looks very attractive. Shades of the old Bernoulli's or whatever.
(This may not be first post, though there were none when I started. Maybe I'll have to settle for first useful post.)
Village idiot in some extremely smart villages.
Use a RAID 5 array, keep a replacement drive handy, and watch the drive temperature.
My solution (linux-based) has been to buy external firewire drive enclosures, put IDE drives into them, and use the faubackup utility to mirror my files. faubackup uses an extensive system of hard links to perform backups at whatever time interval you desire without demanding huge amounts of space - old files that have not changed from the previous backups are stored as hard links and take up no further space (lots of inodes are required, however). In combination with 1 firewire drive (or set of drives) on-site, 1 off-site, rotated, seems sufficient for my purposes.
-Nathan Siemers
it's not the easiest thing to set up, but it's basically maintenance-free once you've got it working.
for backend storage your best bet would probably be a RAID5 array of SATA drives. Do it right and you could make it growable, so as you need more capacity, you just add more drives. Is this going to be a cheap solution? Probably not, but it's probably the best delta of cost/reliability/usability you'll find.
Supermicro makes 3U boxes that can hold 15 SATA drives. Pair that with an Adaptec 16 channel SATA RAID controller, Linux, and Bacula, and you've got at least 3 Terabytes of storage at your disposal.
For a small fee, of course
It's worth mentioning that Rewritable media doesn't have the shelf-life problem that write-once media has, due to their using metallic material instead of dye.
Each processor would proceed sequentially as if it had been better for them not to rise against Saul.
They offer unlimited storage but you pay to download more than a certain amount a month--but if you have hardware failure that leads you to really need it, you probably won't mind paying, or spacing out your downloads.
As probably half the responses will say, use tape drives. They are affordable and store very large amounts of data.(500GB) Also, LaCie drives can come be about 2TB(or 1TB after formatting ;-) 2TB drive: http://www.lacie.com/products/product.htm?pid=1059 8
Last time I checked, it was about $5,000 for 1TB, so get 2 1TB drives and never have to worry about space again.
Make your computer faster: rm -rf
as i recall the articles on the optical drives,
the problem was with the CD/DVD RW/W drives.
If however you use a proper WORM drive that
burns holes you should be ok
You might try to rent a server somewhere with tons of storage space and then back up all your new files every other day and all your files monthly. But you would be putting them on the internet and you would need a huge upriver bandwidth. Lots of problems with this idea but it could be good for a quick fix and it would allow accesibility because you could download or upload files from anywhere.
With great power also comes HeatVision
If your still images were on Kodachrome they'd be good for over a hundred years.
As an old timer watching a superior media being driven to extinction by a still shitty media that's "convenient" I can't seem to engender any sympathy for your perdicament.
"Sorry", is perhaps the thing to say. Sorry.
If you haven't seen it, used it, or thought about it > 6 months... CHUCK IT.
Why not get a multi-terrabyte raid array? Pricey to get started, but it'll keep your data reasonably safe.
You also could use a tape backup. Any of the results from here could do the trick. At work, we use one of the 200/400GB tape drives for backup and are only using about 10-15% of the space (and that's for a dozen servers). We haven't had to test the lifespan of one of these, but tapes typically have an excellent lifespan compared to hard drives or optical media.
My recommendation is to put a nice RAID array in a different computer. I suggest RAID 5 because it is more efficient ("wastes" less space on redundancy and still gets the job done) than RAID 1 but they have roughly equal reliability. The advantage is that it's an entirely separate computer and you even have redundant backups. Hell, you could even put it off-site, but then the connection speeds drop a bit. If you use a RAID array on your local machine you may be tempted to treat the RAID like it is itself a backup (it isn't!) and while I've never had an issue with a motherboard frying all the hard drives in a system, maybe it could happen, and double failures aren't unheard of, but if you have your originals and redundant backups I think you'll be ok (since that's effectively 3 copies). It also wouldn't hurt to have a firewire drive for backing up pictures and video while you're on the road or something (I don't know if you have a laptop) or are really paranoid about certain irreplacable things. Just my 2 cents.
a lot of people have been recommending hard drives, which i think are great for frequently used files and frequent backups. the problem is that they fail to get your data off site.
you probably want to occasionally back up to something that you can store in a lock box away from your house. i guess you could do this with disk drives, but i'd rather use tape. more reliable than optical media and plenty of capacity.
I'm a solo recording artist, and after losing an entire album in a hard disk crash a few years ago, I decided to do something about it. As hard drive prices started going down, I decided to start buying 200GB plus drives to expand my storage capabilities.
Right now I have a 200gb HDD and a 250gb HDD for backup purposes - both are in USB external enclosures, and are IDE drives.
I wait until prices hit around 35 cents or less per gig, and buy then. Keep an eye out on sites such as Fatwallet and Deal News for deals. My favorite time to pick up a new HDD is black friday - day after thanksgiving. Most stores have really great deals on IDE hard disks. I pick up my external enclosures @ Newegg.
"Better to be vulgar than non-existent" -Bev Henson
One thing good about paper & film is they withstand decades of storage vs. years of normal magnetic storage. Photos and films from the late 1800's/early 1900's are still around whereas you're really gambling with current storage media.
Have you hugged your penguin today?
I had a similar problem a few years ago. At the time, I had a PowerMac G4 so I was limited in my connectors to Firewire (nobody's going to back up 200 GB with USB 1.1). I looked into RAID arrays and tape drives and found they were far too expensive. At the time, I didn't trust software RAID (if it was even available on MacOS X then). Since the majority of my data was video, I thought about getting a DVD-R drive to simply make movies out of everything, but I wanted to be able to edit things together first... and who has time for that?
So my eventual solution was to purchase hard drives of (generally) equal or bigger size. It cost much less per GB (even after the external enclosure was taken into account) than anything else I was able to find. Every few weeks (okay, months) I would make a mirror copy with the backup software I bought at the time. I ended up with about 0.75 TB of spinning storage, split between 3 computers and their external backup drives.
Today I have a new G5 with a 200GB main drive... both it and the older G4 have software RAID set up for automatic mirroring. I'm looking for dual drive enclosures so I can do the same with my external drives. I'm pretty sure this is the best, and maybe the only, way to go, and these days even 200GB drives are pretty cheap. I wouldn't go for tape at this point... though I would consider a Firewire 800 RAID, or Gigibit Ethernet Network Attached Storage device (if it were RAIDed too)... well, I'd consider them if I could afford them.
(other than the continual confusion of "backup" and "archive") is that the same people who talk about how unreliable CDR/DVDR discs are for longterm archival purposes seem to be the same ones who advocate buying a portable firewire drive for every project and putting it on a shelf until the client calls with changes.
Something about that seems horribly backward.
That said, Exabyte still rocks my socks
There's a company called Kanguru that makes a USB-pluggable device holding up to eight 400GB removable drives. That's 3.2 terabytes without changing media.
(Geez, my porn collection is nowhere near that size)
I believe this is what you are looking for.
If I were you I would buy a few large (2-400gb) hard drives and, once filled, put them in a safe location. A USB storage center may also be necessary for easy access. For backup of these proportions, I simply don't think that any other method can be as cost effective as what has proven again and again, a standard hard drive.
Back up to tape.
Back up to optical. Do not leave backup archival disks in sunlight or under flourescent light as this deteriorates them more quickly. Store them in a dark place.
For the MiniDV video, just save the MiniDV tapes.. don't re-use them. What you don't need online access to, you've got on tape.
The digital images should be more manageable on their own. Buy a couple redundant backup hard drives. or, save them to DVDs, etc.
It only costs $250 for 500 GB too!
i believe what the author is refering to is the fact that the layer of the disk where the data is stored deteriates over time
...For the beast had been reborn with its strength renewed, and the followers of Mammon cowered in horror.
Raid is a hardware methodology to increase reliability of disk based storage systems. Backup is an archival strategy to recover data lost for many reasons including inadvertant deletion or modification. rm * or del *.* or delete from table or a fire at the site all will mean your raid system now has faithfully lost all your "backed up" data. Make copies to external media stored off-site and locally so that any catastrophe that occurs will not destroy all copies. Tape is still cheapest for archival.
When I was young, I had to rub sticks together to compute.
I can recommend an Exabyte 270020-1438 tape Library system. This unit can hold 21 Slots and can backup 4.2TB native and 8.4TB Compressed.
If you want to save some money, an AIT 4 tape drive system works quite well. AIT 4 tapes can hold up to 520GB. A used AIT drive can be had for less than $200 on ebay and can eaily backup 260GB
*ducks*
You need to ask yourself a few more questions. How much are your files actually worth? How much is your time worth? What kinds of failures are you trying to prevent?
If your time and your files are both worth a lot, and you are worried about your hard drives failing, then put up the money to get a good RAID setup. If you are worried about accidently deleting something, then you'll need to have actual backups going to something else (maybe another RAID setup in the closet down the hall). If you are worried about a failure that will wipe out everything in your house (fire, flood, tornado), then you need to look into some large tapes and a secure place to put them.
But first, decide how much everything is worth. If it is not worth much, just forget about it. If it is worth a lot, then you'll feel better about putting up the money to protect it.
Rename all of the files so they have filenames like "Teen_Lesbian_fff_Hot!Hot!Hot!.avi". Now make them available through your favorite p2p service. Even better, prepend these files with short snippets of pr0n. You'll find that years later you can kick up just about any p2p client and you'll find your files are still available.
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
I'm amazed that someone has yet to come up with a combination of archival-grade photographic film or paper for storage and an optical 'reader' for truly long-term archiving...
Wouldn't it be ironic if paper backups were to become the way of the future.
A friend of mine recently started a small business to address exactly this need. His product is a Linux based RAID box that plugs in to a home network, and supplies reliable storage via samba.
http://www.permastor-us.com/
I suggest cardboard punchcards. With the right care, those will be in your archive for years to come and have virtual no chance of failure.
You need a FREE iPod Nano
I use RAID to defend against hardware failures trashing my data, and I use logical volume management snapshots to protect against most user errors.
Neither is perfect. Some hardware failure modes could theoretically kill two or more of my four hard drives at once, which would destroy my data. Large power surges are the most likely danger, so I use a high-quality surge protector. I consider the remaining dangers unlikely enough to accept the risk.
Snapshots are also imperfect. When you create a Linux LVM snapshot volume, you have to specify how much storage is allocated to it. If changes on the source volume exceed that snapshot capacity, the snapshot stops storing the deltas and the snapshot becomes effectively useless. However, the most likely way that I might screw up and trash my data is by deleting large numbers of files. Since deleting files only updates the blocks that store the directory and inode data, not the contents of the files, a relatively small snapshot partition would hold the changes from deletion of all the files on the source. Now, if I were to accidentally run "shred" on bunches of files... I'd be screwed. I choose to accept that risk, too.
Although the RAID+LVM combo doesn't do quite as good a job as "real" backups, its failings are pretty minor, and unlikely, and it's advantage is huge: I don't have to think about it. I don't have to mess with lots of removable media and I don't have to remember to do backups.
The one thing I still worry about is some sort of catastrophe that destroys my whole system. Suppose my house burns down, for example. I'd lose it all. So I still need to find some way to get offsite copies of the most important stuff.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
RAID hardware is a method. Tape archive is a method. "Methodology" is the analysis of these methods to determine which better suits the needs.
Next, we'll learn about "functionality".
There was just an article on Slashdot about how to build a cheap RAID 5. Read it, and there's your answer.
/ 25/1720258&tid=198&tid=222&tid=137
http://hardware.slashdot.org/article.pl?sid=05/06
Especially smaller businesses which largely ignore the problem until it becomes serious.
My friend's previous company had to learn the hardway when an unsatisfied employee who left the company also set the office on fire. Talk about burning bridges. Almost all their customer's information and designs were lost and luckily a few harddrives were recoverable. The majority of the stuff that was recovered actually came from employees who brought their work home with them.
Seriously think about hiring a company because once it's gone, that's your livelihood gone.
Not really sure if its the best solution, but just because it'd be so fooking cool, how about a 2.8 TB RAID array? This guy did it: http://groups.google.ca/group/alt.comp.hardware.pc -homebuilt/msg/f8479484a5254f5d?hl=en
And that, my liege, is how we know the Earth to be bannana-shaped.
I run a design studio with similar problems to yourself. I back-up current work to tape and archive completed work to optical media. 2 copies: one for internal use and the other for offsite storage. To help ease your storage burden I suggest lossless compression where possible, eg PSD for your image files and something like Pixlet for your video. As a rule of thumb I buy only 'name brand' optical media with some archival reputation, either stated on the pack or reviewed in the media. I have archive CDs going back to 1995 which still read fine. Tape can shed it's magenetic layer over time, hard drives can fail and need expensive recovery. There are no guarantees, but there will be other solutions in the future, so put together a system you can afford to run at the data rate you need, with a view to it lasting 8-10 years and that you'll rearchive to a better solution then, or at least a fresh set of media.
We were using 200gig external USB drives for backup and were wondering why they would all eventually start failing. It turns out the manufacturer (Maxtor) recommended against using these drive for backup??? Sounds like they're saying that the drives aren't reliable so what good are they? We switched to drives that were "certified" for backup/archival purposes.
I have a mac mini which also deals with backup. Even when hooking up
/home/user/dvds macmini:/Volumes/external/video
a few fire wire drives, the thing is still small and quiet. Then rsync
the data over regularly with a script like
#!/bin/sh
rsync -avzuP -e ssh --delete
and call this by cron. This works reliably also with large files like
vmware workstations or dvd backups with several gig file size. Having the backup
over the network allows having the backup machine in a separate place
which limits the risk (for example of theft). I personally also do not rely on
a single backup and mirror also the backup (again with rsync). Since you are
dealing with video and photos which do not compress well, it would make sense
to leave the "z" flag in rsync away.
A direct network attached storage system would be cheaper. But the mac mini
can double also for many other things like a music box or dvd player in an
other room. With some cheap firewire enclosures and harddrives, the prize is
comparable with network attached storage systems from the shelf and the storage
can grow arbitrarily by adding new firewire drives.
Um, how about on mini-dv tape?
You probably recorded it onto mini-dv tape in the first place, so it's easy. After I do my edits and produce a final output, I lock up my tapes as my backup of last resort. (I could always reconstruct the final output using the tapes.)
I only keep final output on my primary storage (hard disk), but it's generally much smaller anyway.
Actually, most businesses use paper backups. In fact all the businesses I've worked for use more paper now then when they use to have no computers. Paper is much harder to change then a bit is. So paper is already the backup method, at least for critical business purposes.
We have a server with RAID 5+0 and 8 250 GB hard drives. Other than having a backup at a different storage point, that's pretty fail-safe for our budget.
"Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies." - Linus Torvalds
Ethics II Axiom 2. "Man thinks." B. Spinoza
Hey, you could always build yourself a .5 TB RAID array for $250 bucks.
Seriously though, I prefer to use tape for backups in the long run up to about a year, for my own usage anyway.
A non-tape setup is great until you delete or corrupt a file, and your automated nightly mirror to the RAID set hoses your backup as well. You'll suddenly wish you could go back to last week's tapes.
For the dv video just leave it on its tape. The only problem is you can't reuse tapes for other stuff. But as long as you take care of the tapes they should last.
It's not really a difficult problem, just an expensive or annoying one.
Digital data can be reproduced perfectly an infinite number of times. Pick a storage medium and copy them all to a new storage medium/media well before the old one's lifetime is expected to be up. (Given, of course, that storage technologies will almost certainly change, and your massive amounts of data will be getting comparatively smaller and easier to manage, if not the things you're creating in the future.) You should have at least two separate copies of everything (well within the rated lifetimes of the mediums) in case disaster strikes one or the media just doesn't hold up like it's supposed to.
Besides, unless you're periodically transcribing onto popular mediums, you might one day find yourself with perfectly good media that's very difficult (or just expensive) to read with modern technology.
Archive your original footage, and your edited movies to MiniDV. Just like a tape backup. Throw it in a firesafe offsite location and forget about it. No need for anything besides your MiniDV camera and a bunch of blank tapes. That's what I do anyways...
Archive images to a hard disc drive and throw in said firesafe offsite location. Plain old IDE drive. Or invest in a tape drive. Your choice.
-everphilski-
Pros:
Cons:
For DV footage, you will need about 3 4GB DVDs/hour. That isn't a bad price point. Good luck!
I store important family videos onto blank DVD disks. I make 100% par sets and store them onto 2 other DVDs...hopefully even if the DVDs develop bad parts I can still rebuild the avi file from the Pars. I store the DVDs with the pars inside 3 layers of ziplock bags...along with one of those little dehumidifier packets....just in case it is moisture that damages DVDs.
Here's what I do, YMMV. I have a RAID-1 array that I back up my 'important' stuff to (I'm switching to a 4x300GB RAID-5 soon). Every 3 - 4 years I replace the drives. With the natural increase in storage space per dollar, I am able to fit everything from my old array with tons of space left over for the new stuff.
Of course, like everyone else you'll have to look at the content you create and decide if it's worth the money required to back it up. Establish a cost per gigabyte for each solution, and decide 'OK, it costs me $.50 to back up this group of pictures, or $2.50 to back up this video file - is it worth it?'
Help save the critically endangered Blue Iguana
Fire resistant containers protect paper and other common combustibles. However, they do not keep plastics from melting.
That's why you need a media vault or container.
Software RAID 5 (with spare drive already attached to the RAID array) and Linux is a good bang for $$$. The reliability of current IDE hard drives are poor (current SCSI is about 75% better than IDE but still poor in my view - from how many hard drives I have had fail in my SCSI and IDE arrays). I have had a couple RAID 5 arrays become destroyed by having 2 hard drives fail before the RAID array could be rebuilt (IDE RAID5).
:).
So, what is your budget? I would do tape for long term backup and house the tapes in a fireproof medium - even pay for secure off-site housing. Two RAID 5 systems mirroring each other - maybe one housed off site, can be used for daily access (look at rdiff or rsync). Basically, the more mediums you can store your data on and the more locations you have those mediums located at, the better off you will be. I have the view of this; If my city was hit by an atomic weapon, would my company's data survive. All out nuclear warfare, I don't care about my company's data
200gb hard drives are like $cheap
duh
Mea navis aericumbens anguillis abundat
This shouldn't deter you. I made this program to solve that problem:
http://sourceforge.net/projects/filechopper
You should be able to script its usage easily enough.
Use Photo Album 1.0. It's in stable release. Fundamentally, it is a browsable catalog of page objects. The page objects are related to each other via a "spine" object. The spine object determines the page object order.
You can typically store four photographic representations per page object. Latest versions come with these little cardboard holders (also called photographic representation orientation configurators) that hold your photographic representations right on the page objects! The browsing is intuitive. You simply "flick" through the page objects.
Photo Album 1.0 can also be used to backup video. You just need to flick through the page objects really fast.
Just get hundreds of gigs of disk (probably you'll need at least 1,000), then use rsync to mirror them. I've used this system for some years, with an amount of data on the order of what you're talking about.
I have multiple mirrors, and I rotate between the mirrors, so that on a given day, I have a backup which is 1 day old, 2-4 days old, and a week or so old. On top of that, I periodically take mirrors out of circulation, say, every month or so.
Yes, I have alot of hard drives. However, drives are cheap. Losing data is expensive. Manually doing backups is expensive (both because of the time it takes you, and because of the cost of screwing up, which is likely). Tapes are useless -- you can never know if they really work. Same with CD-ROMs/DVDs. Tapes & CD's are also useless because their capacity is way too low -- it's the same as a manual backup.
A vast harddrive array is clearly the way to go. Everything is automated -- hence, very low chance of screwing up. Also, you know right away if a hard drive doesn't work. So, you can be sure of several working backup copies to fall back on.
You'll no doubt find that you want multiple computers to house all these drives...
The NIST report didn't say optical media were inheritantly unreliable at all; it said that there were big differences in media quality, and that storage conditions were important.
Personally I think hard drives are the pits for data reliability. The drives are good for MAYBE 3 years, subject to all sorts of electrical failures, and even if you have a RAID you still can lose the whole thing due to a {virus,controller,power supply,filesystem,usererror}.
I use redundant MAM-A gold stabilized CD-Rs for my data which were the most stable option in the NIST report. That works great for everything I have including digital photos.
DV might be a pain with CD-R so I would probably start with staggered redundant sliver DVD-Rs until I saw some more data on the lifetime of this media.
No way would I consider hard drives an acceptable archival solution.
I personally do not like using RAID as a backup solution. In fact, it is quite useless. All RAID is good for is protecting you against HDD failure. If you accidently delete some files of of your RAID setup, the files are gone. Backups should be done on media that can be moved away from your computer and stored.
is stone tablets of course..
/me whips out sumerian cuneiform tablets and reads the epic of gilgamesh .. still around after a few millenia.
I don't see magnetic or optical drive technology having withstood the test of time.
There are about thousan comment to use raid as your backup technology. If my data s important to me, I would pick a tape drive, and then keep data for as long as 7 to 30 years. Optical media, one scratch and data is gone. nowI should get response for Nearline storage, but people, the guy wants some thng for 200G and retain it for some time. so look into future growth also, setting up raid is NOT backup. if you delete a file by mistake or overwrite a file, RAID does not save for there.
This is the pot calling the kettle black, though. Is there a support group out there?
porn?
Whatever medium you choose, you could combine with a parity archive tool to recover from minor media errors.
no kidding.
all photos and stills from videos are much longer lasting on acid free paper for archiving. they can always be scanned into digital format again.
betamax video tape or laser discs are also better for long term video archiving than the CD/DVD medium. ditto on the reinsertion into digital format when necessary.
all this is old school, low tech, but your chances of keeping your photos and footage are much better than going with current digital media.
regards,
roger born
writing.borngraphics.com
Always drink upstream from the herd.
I have 260GB of digital photos. I incrementally back up over the internet every night using a python utility called link-backup. It creates hard-linked dated backup trees. The great thing about it, is that it maintains hardlinks between files even if they've been moved around or renamed. Also I can tell it to run between fixed time periods, like while I'm asleep.
.DD-HH.MM:SS/log
http://www.scottlu.com/Content/Link-Backup.html
Here's the man page:
Link-Backup
Link-Backup is a backup utility that creates hard links between a series of backed-up trees, and intelligently handles renames, moves, and duplicate files without additional storage or transfer.
Transfer occurs over standard i/o locally or remotely between a client and server instance of this script. Remote backups rely on the secure remote shell program ssh.
Usage:
lb [options] srcdir dstdir
lb [options] user@host:srcdir dstdir
lb [options] srcdir user@host:dstdir
Source or dest can be remote. Backups are dated with the following entries:
dstdir/YYYY.MM.DD-HH.MM:SS/tree/
dstdir/YYYY.MM
Options:
--verify Run rsync with --dry-run to cross-verify
--numeric-ids Keep uid/gid values instead of mapping; requires root
--minutes Only run for . Incremental backup.
--showfiles Don't backup, only list relative path files needing backup
--catalogonly Update catalog only
--filelist Read srcdir relative path files to back up from file
--verbose Show what is happening
Comments:
Link-Backup tracks unique file instances in a tree and creates a backup that while identical in structure, ensures that no file is duplicated unnecessarily. Files that are moved, renamed, or duplicated won't cause additional storage or transfer. dstdir/.catalog is a catalog of all unique file instances; backup trees hard-link to the catalog. If a backup tree would be identical to the previous backup tree, it won't be needlessly created.
How it works:
The src sends a file list to the dst. First dst updates the catalog by checking to see if it knows about each file. If not, the file is retrieved from the src and a new catalog entry is made:
For each file:
1. Check to see if the file path + file stat is present in the last tree.
2. If not, ask for md5sum from the src. See if md5sum+stat is in the catalog.
3. If not, see if md5sum only is in the catalog. If so copy catalog entry, rename
with md5sum+new stat
4. If not, request file from src, make new catalog entry.
Catalog files are named by md5sum+stats and stored in flat directories. Once complete, a tree is created that mirrors the src by hardlinking to the catalog.
Example 1:
python lb.py pictures pictures-backup
Makes a new backup of pictures in pictures-backup.
Example 2:
python lb.py pictures me@fluffy:~/pictures-backup
Backs up on remote machine fluffy instead of locally.
Example 3:
python lb.py --minutes 240 pictures me@remote:~/pictures-backup
Same as above except for 240 minutes only. This is useful if backing up over the internet only during specific times (at night for example). Does what it can in 240 minutes. If the catalog update completes, a tree is created hardlinked to the catalog.
4. python lb.py --showfiles pictures pictures-backup | python lb.py --filelist - pictures pictures-backup
Same as example #1.
Example 5:
1) python lb.py --showfiles pictures me@remote:~/pictures-backup | python lb.py --filelist - pictures me@laptop:~/pictures-transfer
2) python lb.py --catalogonly pictures-transfer me@remote:~/pictures-backup
3) python lb.py pictures me@remote:~/pictures-backup
If the difference between pictures and pictures-backup (for example) is too large f
Your computer's own hard drives should keep only what you are actively working on. Get the rest of the stuff out of your way.
Buy GOOD DVDs ... burn all the files you are not actively working with to these - two separate DVDs for each archive, of two different brands. Check for file integrity, label them well and store them in a convenient, off-site location, cool and dark. Delete the originals from the working drive. Check the archive disks fairly often for degradation and re-burn as needed. They are no more labile than negatives and videotape.
For the large files, buy removable drive bays and holders, and copy them onto large hard drives. REMOVE the drives and store them with the DVDs.
On your working system, continue to back up the data for the active projects. Consider getting a RAID 5 system for data integrity, because if you back up data from one drive to another you risk overwriting a good copy with a bad copy.
I keep reading comments about how CDs/DVDs are unreliable. Here's a great trick i use to make sure my data is safe: i always include 50-100mb of parity files on each DVD. The disc would need to be REALLY messed up to be unrecoverable.
Just get an 8-port serial ARA raid card like the 3wave 9500S series and 8 Hitachi 7K500 drives.
;-)
7 drives in the array, one for parity - boom, 3.5 terabytes and safety too.
Of course, if you don't have $4,000 for that setup, then 2-3 external hard disks would be plenty of redundancy.
I will be buying the 7k500s when they ship, though.
JP
Stiny! Get me a danish!
Backup one HD to another. But I would say you probably want to make sure each "HD" is a RAID 5 array. That's where I'm headed soon. Also... you didn't mention the platform you're on. I use Linux and I've found that rsync is the ultimate backup tool. I use it to back up HD to HD with only files that have changed and network to HD for my web/mail server. There really isn't a better way until someone comes up with reliable removable media that can store at least ten times the largest current HD. That would mean 4 TB of storage on removable media since the largest IDE drives are currently 400 gigs. I don't think that will be happening anytime soon because most removable media companies are way behind the times with offline storage. Also keep in mind that the way a lot of big enterprise outifts are moving is:
;P
Disk --> Disk --> Tape Library
HP also has a virtual tape library that really exists on disks. At this point, I think HDs are still the best backup method. Just make sure you have many copies. The ideal if you're going for just HDs:
Data Disk --> Backup Disk --> Backup Archive Disk
The "Data Disk" is where you store all your data
The "Backup Disk" is where nightly backups go
The "Backup Archive Disk" is where you hold the previous week's backups from the backup disk in tar.gz or Zip format.
I would also keep the "Backup Archive Disk" on a separate machine on a separate network (just a cross over cable from your main machine) on a separate electtical circuit. This way even a power surge is not likely to wipe out all your data. The "Backup Archive Disk" should be at least ten times the size of your main "Data Disk". In my case that would be 5TB. Make sure you pay your electric bill too.
-"...bad old ideas look confusingly fresh when they are packaged as technology" - Jaron Lanier (Digital Maoism on Edge.o
This is my home setup; 2 servers, each with (8) 250GB's HD's. The one server has map drives so the desktops can map the drives as ND1, ND2, ect... The other server as a program called viceversa and I log in by RDP and it sync's the files. Simple and not too expensive (celeron cpu, the 250's where only $100 each after MIR). Just a slight upfront cost to get it running.
I've found more and more friends switching to small-to-medium sized raid arrays, usually based on linux, with software raid.
software raid isn't as fast as hardware raid, but if you have a raid controller fail, and you can't replace it - that's ten times worse.
honestly with the size of files these days, I can't see a viable alternative solution to a good raid 5 setup, or raid 6 if you have a huge amount of disks.
At work we have to archive Broadcast Quality TV Shows (Yes, I work in a TV station). These are 50 Mbit/s for the video plus at least another 4 Mbit/s for the audio. Needless to say this takes up a lot of space. For this, we use LTO-1 tapes that store 100GByte per tage uncompressed (compression gets us zilch with the video and audio). The tapes have error correction that we pay attention to. If there are getting to be too many errors we replace the tape and have the info copied to the new tape. Since we have so many shows, we are moving to LTO-3 tapes that store 400GB per tape. The LTO tapes are expensive. However, as long as you do not do constant reading from them and use them as a true archinve they should be fine. For massive redundancy, put the same files on two different tapes. Also, the reader/writer is a little expensive, but you only need one. Also, LOT-3 drives can read/write to LTO-1 tapes (only as 100GB, not 400GB). Write speed is pretty good to, being above 14Mbyte/s. Shelf life in a temperature/humidity controlled environment is pretty long. A bank vault should be pretty good as well.
Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
First that comes to mind is Tape backup.
So what you'd suggest is that he downloads the video from the MiniDV tape to the computer, then archives it onto backup tapes. Why not just keep the original MiniDV?
"I've got more toys than Teruhisa Kitahara."
paper tape backups.
Setting his threshold to 5, Sparky eliminated most of the trolls on /.
We run an adult video company and have tons of video and pictures that we have to make sure is backed up.. our raid system 2 1.6 terabyte buffalo terastations.. which after raid 5 you get about 1.1 terabyte per box.. we have another raid system that mirrors most of our data and backups anything that changes based upon rsync for windows. we use a 3ware raid care and have 4 400's in the system.. For backups that we take home, we have a hard drive cage in our pc that we have drives we can pull out put up backups on and store them at another loation.. of course this is the poor mans backup but with HD prices to cheap. Tape backups well you never know whats going to happen with them and your data isnt accessable quick.. who knows if your going to have that same backup software for your tape drive 5 years from now.. or if your tape drive fails.. which happens alot.. just my .2
Pocket Girls. Mobile Adult Mini Mags for your Phone.
About a year ago, I built an office file server and a second server for point-in-time snapshots. The two use some relatively cheap CodeGen cases that had 8 5.25" bays in which to mount tons of drives. The file server got 4x200 gig drives in RAID 5, the backup machine got 9x300 gig drives in RAID 5.
The backup machine keeps a regular snapshot of the data for a relatively quick worst-case cold-standby, by using rsync once per night - and after rsync is done, the resultant snapshot gets tar/gzipped and rotated.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
DV Backup is a shareware application for MacOS X 10.2 and 10.3 which lets you backup any data files to a MiniDV or Digital8 camcorder.
With up to 17.5GB of storage on each low cost, high capacity 60 minute MiniDV (or Video8/Hi8) tape, a single one hour tape can store nearly four times as much data as a 4.7GB DVD disc, and more than twice as much as a dual-layer DVD.
- passion
The short DVD life is for the ones that get used. If its in a safe deposit box in a black jewel box I can't understand what might happen to it - chemical breakdown due to air interaction?
I am very easy to get along with, but I don't have time to waste being nice to people who are being stupid. -Theo
If your data is really that important theres a few things you should do. First your desktop that you are using should be a RAID 1 or a RAID 5. That eliminates HD failure from being a problem (unless your dumb and dont pay attention to that flashing message on screen saying "harddrive failed - oh #$$%").
As for "arhive". Get a machine w/ good quality parts - Install a RAID 10 (yes, Raid 1+0) and install linux on it w/ Samba and a gigabit nic. Its a NAS.
And if your really really parinoid - Get a Removable HD Hotswap drive bay (like SATA kind) and a couple 500gig HD's (or what ever size you need). Ever so often (like every month) back up everything on to one of thoes drives and bring it off premisis - you can re-use the drives every few months.
snowulf.com
You are worrying about backing up your DV material? Why? It's already in the digital domain in a nice, compact, reliable format. DV doesn't look like it's going away soon, and IEEE1394 doesn't either.
If you want to do an offsite backup, just clone your tapes and store them off site. A quality 1 hour DV tape is less than 4 bucks and they aren't hard to store.
If you want to save your edited programs you can archive them off to DV as well. There isn't a quality loss since you started in DV, right? If you want to save project files (we used to call them EDL's) you can save them to other media (CD-ROM/RW, DVD, floppy, or paper print-out even), or e-mail them to a Yahoo or G-Mail email box for safe keeping. If you need do a re-edit, you can redigitize from the original source tapes using data from these saved project files to get the material back on your editing system. A batch digitization will only use the material that you originally noted as used, so even for a fairly long project it isn't too time consuming. Shuffling tape is the pain-in-the neck part, but the process is mostly an unattended one. Good media management is key here of course. Name those tapes!
If you must archive, there are some tools that will archive all the media used in a project back to DV tape. Mezzo Technologies http://www.mezzotechnologies.com/index.html is one that you might want to look at.
I guess you could archive stills to network storage like Yahoo mail or G-Mail as well. If this is critically important stuff, then it is probably worth paying for on-line archiving.
I have a 2.5TB RAID 5 NAS that I use to store my data and then I use tape backup (I use a Qualstar TLS-6210 DLT 7000 35/70GB Tape Library that I bought off of E-Bay for about $500) to back up the data and store the data off site.
You don't re-backup everything every time, do you?
...Or at least keep it in a folder that does not get backed up. Use two USB hard drives. Copy your files to both hard drives and give one to a good friend/relative/Bank box or something. once they are archived (backed up for the last time) stop backing up those files unless they change.
1) You should Archive the older less used stuff and remove it from your system.
2) New and modified files will be the only thing you backup on a regular basis. You might backup your entire system (not the archive) every week but backup any new and modified files daily.
good luck
The government which is strong enough to protect you from everything is strong enough to take everything from you.
Sounds like you need Microsoft Data Protection Manager.
Simply yank the HD when it's full, copy important files to a new HD so you'll still have access to them, and use the new HD as your work drive, putting the old drive in storage.
This way you don't have to worry about the drive failing of old age, because you're always using a new drive.
At the same time, keep a USB drive handy for instant backups of important work.
I have a cabinet full of (SCSI, and then IDE) drives going back to '96 stuffed with video and music from when I owned a recording studio (I now run a video production co.) Important data is saved twice to two different drives. Just. In. Case.**
This, alongside your archives to optical, should ensure that your data is useful for years to come. I've never had any problem retrieving old data from drives that are now almost 10 years old.
**I should also store a copy of materials off-site, but if my studio burns down I intend to be inside it.
"This is totally insecure, but very convenient."
You can 1TB of RAID (RAID levels 0, 0+1, 5, and RAID 5+hot spare) for $1500 from . I imagine other manufacturers have similar offerings.
One big thing you may want to consider is that even with a single solution such as a NAS is that if there should happen to be a fire you can still lose everything. I would suggest that you archive each file numerous times to DVD's and distrobute a copy of them to various individuals that you can trust to hold them for you - possibly even a safety deposit box. The only unfortunate would the the data spanning multiple discs. For that I would suggest using some program that can span discs and include a copy of it on each disc. This might be a bit more of a nightmare to keep track of which copy is more uptodate, but it would be a good solution for those things where you are just adding to the archive ( aka photos/videos ) versus things that are rapidly changing and need versioning info - (aka source code docs etc ) Also every 3 months burn a new copy of all your archives. This will give you quite good assurance that you will have access to your data after the fact.
You could always make sure you are taking photographs and making videos that are high enough quality that the entire internet would enjoy them..
:)
Post them on your website.. make sure they are cool.. and just like Usenet, they are now archived all over the world on your behalf.
1. Create a script that archives your important files - if you can set it up to only 'refresh' archives, it will be much quicker thatn a full archive each day. Good formats are RAR, ZIP etc. 2. Run the above script and create the ouput file on a HDD. 3. Back up the file to a Firewire/USB2 connected external drive (200 Gb should suffice) 4. Store said drive in a fireproof safe.
How about you encode your video instead of backing up raw DV? It can't be THAT important that high bitrate xvid won't help you out.
While I love raid, RAID is not a backup - raid is about availability and consistency. So if you delete one item in a RAID it is SUPPOSED to be lost to the entire array.
/. readers, but it's two 1-line scripts and I've seen them on here before :)
/etc belongs in "current"
In everything I've read, the moral definitely seems to be harddrives, lots of harddrives, for price performance. I'm assuming you have a reasonable LAN or can set one up.
Here's the setup I haven't finished implementing yet: PLEASE give me any comments about it to help me improve my setup.
1. Setup a file server using at least one big, inexpensive disk. (This can also be a desktop as long as it can reasonably serve files.) This is your "USE" server.
2. Separate you files (on a per-directory basis) into categories based on how frequently they are changed. The important consideration is: 'If a file is changed/deleted from USE how long should I wait delete a file in the backup' Personally, I only need two categories. "current" = a month or so depending on disk space and "archive" = never (family pics, videos, etc.)
That means that if I delete something in my "current" tree _AND_ I don't notice for a month, my backups will delete it and it's gone forever.
3. Setup a 'backup server' using at least one inexpensive hard disk. Set your backup server to login to your USE server and sync your files.
It should be able to do both "full" (copy everything) and "incremental versioning" = "IV" (if something is changed, keep BOTH copies, marking them appropriately) backups. Neither of these kinds of backups should ever eliminate any information automatically - they should just add information.
4) For me, I'd run:
1) An IV backup of "archive" every night.
2) A full backup of "current" every week.
3) An IV backup of "current" every night.
4) A job that deleted the oldest backups of current every week.
Notice that I'm _never_ running a full backup of "archive" but I'm also _never_ deleting the backup.
Notes:
rsync or rsync over ssh is my preference for doing this kind of backup. It works very nicely, but I'm too tired to get it right just this minute so I'm leaving IV/full backup commands as exercises for other
cron is fine for setting it up automatically.
wget has similar functionality to rsync for a website and you don't need any privileges.
I think most of
Do make sure you log the output of your syncing software. Also make sure you monitor disk usage. If you want to be fancy, it could keep all of the full-backups of "current" until space is short (with a reasonable margin) and then always delete as many of the oldest ones as it needs to to make enough room. This means your number of snapshots will vary with disk space - some people think that's evil.
This system scales reasonably well - for more size add more harddrives per server and/or more servers. For redundancy add more backups per live copy. As long as you can keep it organized and your network handles it, there's also no reason a USE server can't be served by two backup servers or a backup server can't also serve several smaller workstations - or any combination thereof.
Do not add multiple harddrives to a backup server for redundancy. These servers are essentially free and you get much more redundancy (and some scalability) if you use two backup servers. With a setup like this, any server should only have one copy (excepting multiple versions of the same tree)
You could just do a full backup of current every night or whatever, and you could have many possibly more complicated "current" backup schemes. But for me the total size of "current" is massively smaller than "archive" so it's really not important. Remember, having more of these isn't more redundant - they're all on the same drive.
This backup server should generally run no services except possibly ssh and certainly shouldn'
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Could be a number of reasons to do the seemingly illogical double-hop method.
First, the new data may have been processed (edits, color correction, etc).
Second, the backup media may be better rated for long term storage. I'm not familiar with MiniDV, the stuff I work with is all DLT and HCART2 under Veritas Netbackup, at 200GB raw/400GB compressed per tape.
Third, it may be helpful to have the indexing done for him by a good backup program.
However, as I say, I work with Netbackup. To say it's pricey is an understatement... but it's changed my views on what's a "workable" backup system to only liking enterprise grade stuff.
This might sound obvious.. I'm a photographer.. I have a 2 bay firewire drive set I use for "HD" backup of my photos and video. Its 700 gig. I also burn DVDs.
When I backup my stills onto dvd I use jpeg 2000, its lossy but really not that bad once the image is in a good state.. I did some tests in college on jpeg/jpeg2000 vs tiff (uncompressed) of the smae image to see how much is lost. Not a lot it turns out. I love uncompressed images, but the loss when storing as jpeg isn't so great to matter unless you do a lot more manipulation. I'm also still shooting film which can always be rescanned at a later date.
However, you shouldn't backup all the DV (raw video) you dump on the computer. The original tap e can act as the backup. its still on the tape even after you dump it into the computer. Label it and set the right protect notch. Voili, instant backup footage.
I'm assuming you edit this down and give the client a dvd/video. Just keep a DVD copy for yourself. Thats all they can really ask you for. If they come back at a later date, because the dvd is bad try yours. If that doesn't work you have to go back to the tape and redit and recharge.
Why not just keep the original MiniDV?
Probably because mini-DV holds about 13GB and an LTO has a capacity of 400GB. Get a 4 tape autochanger and you've got 1.2TB, or about 92 Mini-DV tapes.
LaCie Bigger Disk6 2
http://www.lacie.com/products/product.htm?pid=104
What if he downloads the file, does some processing and modifications to the video, then encodes it into his preferred format. I don't know how many cameras would support a completely foreign file being copied to the tape, in fact, I don't know how many cameras even support writing from the firewire connection.
Or perhaps he wants to keep the files separate, instead of tarring them together.
In that case, any tape would be kinda lousy because or insanely long seek times for arbitrary data.
Late night posting strike again... make that: 1.6TB is 123 Mini-DV tapes.
Back up to multiple destinations, say to multiple media, maybe 2 DVD and 1 Tape. After a certain amount of time, 1-2 years or so, restore the data which serves to prove the data integrity. Then backup the restored data to new media. This can also help to stave off the fear/risk of technology becoming "unusable" in 10 years.
One drive for backup on day 1, the next drive for backukp of day 2, third drive for day 3. Each day, 1 drive is taken off the premisis. Maybe bank safe deposit box? Or archival service.
Drives are aged out after a year or so.
So, what's 3 200+ GByte SCSI drives cost these days?
TODO: create/find/steal funny sig.
Unless you can afford to backup to off site spindles (and few small businesses can,) off site tape is still the way to go.
Older DLT Drives seem to be the best value right now, but if you need more capacity you could always go with Super DLT or LTO.
It might be a paranoid verison of backups, but I've found that the best backups are redundant backups. Tape fail, CD's fail, hard drives fail, but unless they're all in the same place when it catches on fire, they probably won't all fail at the same time. My backup strategy for the 80+gig of photos I have is:
1) Simply keep all of the online. I have one server dedicated to nothing but photos, and it's mounted as a share to all the other PC's.
2) Periodically copy over the entire directory structure of another server that has the space.
3) Regular tape backups of the main photo server. I bought a DLT tape drive for that. The tapes go off-site to my office.
4) Annually I take a CDR backup of the past year's photos, and put them in a safe-deposit box at the bank. I first go to the bank to get the old CD's, take them home, try them out to make sure they're still OK, and then take all the CD's back. If any CD's fail, re-burn them from the master server. Take everything back to the safe-deposit box and repeat next year.
This method, albeit tedious and time consuming assures me that I have a good copy of the photos in several places at all times.
I have 1000 Gmail accounts, 2TB of backup!
First, the biggest problem is that your backups are 'failing' - why? It's imperative that you figure that out first. Continuing past that point is useless, and as you don't really say why they're failing, answering further is pointless.
However, asuming you can get a backup situation to work:
I would recommend a large RAID5 array dedicated to backup on an independent system. Set up a server that will accomidate data growth for 2 to 3 years at the given rate of accumilation.
I could get as complex as you want, but that's as the terse version, and pretty much as much data as can be given without getting sidetracked from your goal, given the information you've given slashdot.
What the hell.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
DVD/CDs die.
Tapes die.
Hard drives die.
All but hard drives are a pain the ass. This is open to debate.
Put it on a hard drive. Duplicate the drive. Put it on a shelf. Buy a used computer with an interface that matches the drive for cheap. Put the computer on a shelf.
In twenty years you should be able to retieve your data. By then someone will have come up with something to preserve your data for all eternity.
With any luck it will be me =)
Google,
Come in Google. Are you listening to this? You need to use your massive stores to get into the DVD video and picture backup business. You already have the business model, and the know how. DO IT! I want to see DVD Picture Beta in two weeks (One week for the idea to make it to the website, and 1 weeek to start getting some harddrives set asside.)
According to the article itself, some CDs stand up well to prolonged high light exposure. The best is silver + gold coating with a phthalocyanine dye. According to http://www.silverace.com/dottyspotty/issue12.html, the only disc like this is Kodak's Ultima Silver+Gold CD-R. Other discs that do very well use the phthalocyanine dye. According to Roxio.com, the "phthalocyanine dye is pale green, appearing yellow-green on a gold-backed disc."
Also, keep your discs in the dark at controlled temp and humidity if possible.
Maxtor has their OneTouch line of external drives. It ships with their Dantz Retrospect software, which can launch a full backup just by pushing a button.
The retrospect software is pretty powerful, not necessarily the easiest interface but it can do a lot.
You can also customize the button on the front to launch whatever application/script you would like. Personally I launch the backup manually, but I use the button on the front to open the calculator.
I've had a OneTouch 2 for about 2 months now, and should the internal drive ever fail or I want more storage I can just put in another PATA drive.
I was also drawn to it because of the solid cast aluminum shell. It makes it VERY sturdy and doesn't need one of those tiny little fans that last about a month before they start whining and rattling.
I currently use an Xserve hooked up to a Xserve raid, when a drive fails, it is striped, so the data can be recreated from the other drives. If a drive fails, i replace it immediatly, and send the faulty one back to Apple.
Currently i have over 3,000 GB of DV film stored. Besides that, i have over 800 GB of standard pictures from digital SLR. I do amateur photography (Mostly nude). There is also around 400 GB of songs on it.
It has been good to me. Only one drive has failed so far, and the data has been safe. It is connected over Fiber to my G5. Dual 2.7 Ghz water cooled.
Okay, so the above was all made up. It is what the school i work for has though, and they use it as a backup solution, and it works perfectly. They use it for student files though, not nude photography. They do have a ton fo DV stuff though, as we have a Video Production class.
cat
and yes, I should upgrade but money's tight right now.
1) Download from card to 1st HD
2) MD5 checksum every file.
3) Copy to 2nd HD (manual mirror)
4) Confirm MD5 is correct for copy
5) Burn 2 DVDs on different media types.
6) Catalog disc with thumbnails (ThumbsPlus)
7) Catalog files on disc
8) Put 1 DVD in the Century CD DVD unit jukebox
9) Put 1 DVD offsite.
Repeat.
For larger images, use nero's "Back-it'up' to make a larger copy.
Occasionally buy a 250gb or 160gb HD and copy every file over, then take it offline.
Seems to work for now, but as you can imagine it's starting to get aggressive in terms of power usage.
This probably isn't very practical due to cast. But I can tell you what you my company uses for backups of stuff like patent records and stuff.
We use what can best be described as a computer with a cd jukebox hooked to it. When a patient record gets stored. The jukebox burns the images onto two DVD's and a third one that uses some kind of error correction that can rebuild the images even if it can only get fractions of each. Headers and written to each DVD, and to the computers database (so they can be retrieved quickly, or have the database rebuilt even if you only have the DVD's). And finally a robot arm moves the DVD's to their storage place.
The only problem, this system has the word medical in front of it, so you can imagine the cost.
If you are reaching volumes that large, why not just stick with hard drives? Hitachi offers a 500GB hard drive now. Put two in a RAID 0 (striped) array and you have an extra 800GB of space. Or skip the RAID and just have two new drive letters...
Since you probably have a high quality video camera and your wife probably has a top-of-the-line SLR camera, you can probably afford two new drives. Burning to DVD or CD will be cheaper in cash value, but how much money is your time worth?
Since you're wanting an archive that will age well, may I suggest engraved/embossed titanium plates? Simply carve all of your pertinent ones and zeros onto the plates which are specially fabricated for that purpose. I'd say at least a few mm thick, and any size you want. Hell, size apparently isn't the issue, it's longevity of storage... and baby, have I got your shelf life right here. For extra-long-term solutions, invest in a vast underground bunker to store your porn^H^H^H^Hpaleological studies. You didn't mention whether cost was an issue, so I took the well-heeled route.
The Chronic *WHAT* les of Narnia!
I have been in the same position the Author discussed, and I have come to ONLY negative conclusions. In a few words, and I hate to say this, but buddy:
WE'RE FUCKED.
Digital is a loser's proposition. backing up to analogue or even digital data on analogic substrates (such as DV tape) fail. Simply nad purely.
The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.
Also, a RAID assumes a continuity of several things that are not likely to be continuous:
With Video:
Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables.
With Audio:
sample rate, compression format, bit depth, file format, etc.
Basically all of it points to very bad places.
I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper, our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.
Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work. Archiving the 21st century will be impossible. To the 25th century, the 21st century will be seen as a dark age - not only for the holocaust of the die caused by the failure of the petroleum based economy, but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on /.
His problem of saving personal video is just the tip ofthe iceberg. His problem is the problem of our very civilisation, writ small.
That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.
RS
Shoes for Industry. Shoes for the Dead.
What I usually do is put it in silicon storage. I have a very large glass grid with pits in it about every 3 mm. I then put a single grains of sand in the pits. A pit with sand is a 1, and a pit with no sand is a 0. As long as I don't breathe too hard on it, it works great! And its all solid state, so it'll last for ages!
SIGSEGV caught, terminating
wait... not that kind of sig.
Rsync ( http://rsync.samba.org/ is really great for backup of Unix-like systems. The ability to hardlink identical files allows me to store hundreds of daily full images of 100GB of sources to a single target 250GB hard disk. Rsync is very smart about moving only changed data over the network, resulting in speedups of 10x to 100x. This allows me to do full backup on my offsite colo without using a lot of bandwidth. Note that Rsync is great for Mac/Unix/Linux, but it does sometimes have problems with windoze clients. But then, so do I ...
Dirvish (originally written by jw schultz) is a Perl wrapper around Rsync. It facilitates the scheduling and management of Rsync based backups. We have a fairly active mailing list and contributions from around the world (open source is so cool!).
Backups should be safe against:
Backups should be automatic (or they will not get done) and cheap (hard disks are cheaper than tape, and much cheaper when you use hard linking). Rsync stores the data in a file system closely approximating the original, which facilitates restores.
If a cheap electrolytic filter capacitor dries out in your power supply, and the 5V output decides to start making a 15V squarewave instead, everything in your computer case will get fried. Including every one of the RAID disks. External USB enclosures (or airgaps!) protect against host and power supply failure.
If I was really paranoid about protecting my data, I would run a long ethernet cable to a nerdly neighbor a few houses away, and put a second dirvish server there. While I do rotate my drives into ziplok bags in a fire-resistant safe, the maximum credible accident (a furnace explosion) would tear open the firesafe. If I was paranoid and rich, I would use a high bandwidth VPN connection to a big disk in a colo machine in a different city.
The best backup is server-pull, frequent, automated backup onto multiple R/W media in multiple places, and frequent checking of that data. The closer you can approximate this, the more secure your data will be.
Keith
Keith Lofstrom server-sky.com
If he recorded on MiniDV, then GREAT, there's never a reason to tape over the master copy of anything!!
But most of the time, it's a digital camera, where it's flash ram it's recording to. For digital video, it's most likely in need of some editing which is the whole reason to bring it into the computer anyways, which is when you need to start the backup proceedure.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
http://backupmyinfo.com/
it's easy. it works.
This
Just today I saw a new device Ridata Topy at http://www.ritekusa.com/, it uses 500Mb-1Gb disks
that supposedly live longer than regular CDs.
Writing device is about $130.
LaCie has terabyte USB/Firewire drives available
that you can use to do rotating backups of your data.
The company I work for has 5 terabytes of removable
hard disk media that we use for daily, weekly and monthly rotating backups.
With a couple of these drives, rotating backups, you could
not only make your data backups but
also store one copy offsite in case your
house/office burns down or something else unforseen happens.
These are definately a great value for doing backups and aren't stupid slow like tape drives.
If it was 100% absolutely necessary, I would backup to multiple harddrives with the same data and place them in multiple locations. I would also think of backing it up to DVD-R, which is annoying, but I suppose if something is important enough, you have to do what you have to do.
Don't buy a tape drive. I have one, on a shelf. I gave it up.
I bought a raid controller and 4 sata drives. 400 gigs of parity. It's not perfect, a comet could hit my house, but it's pretty good. And it's live, all the time.
Put the RAID in a NAS system on your net and copy things to it. Buy a battery backup and there ya go. Good enough for government work!
...or maybe not.
Do what google does:
1. Buy the cheapest $/GB ratio disks you can find (250GB today)
2. Attach as many of them as you can to any machine(s) you own.
3. Don't use any kind of raid. Google uses GFS (replicating file system), but you can do a poor man's GFS using rsync.
My setup: DELL SC400 with 9 hard drives hanging off of it: 3 IDE inside, 4 SATA + 2 USB outside. The SATA drives run on a separate ATX power supply. USB have their own power. 7 of the drives are 250GB. I organize my data in sets (photos, video, music, books, software, projects, notebook backups, etc). I wrapped a script around rsync which allows me to assign a replication factor to each of the sets separately (depending on importance and storage requirements), but most sets are replicated 3 times. There is one master copy which I mount remotely. The others are replicated.
Advantages: each disk is independent (you don't rely on raid hardware or presence of any other disks to get to your data); you can grow it incrementally; SIMPLICITY (fewer things to break); you can dedicate one or two disk as emergency copy (in case of fire just grab two extenral USB disks)
Disadvantages: slighly more expensive than more cost effective RAID 5, but at under $0.50/GB it hardly matters.
If your going to continue to add to your archive, then how about d-Space. Its an open source project over at http://www.dspace.org/
s ue/feature_mit.asp
Might be overkill maybe not. Theres a good article over at MIT Technology Review this month.
http://www.technologyreview.com/articles/05/07/is
Whats a good hardware/software tape drive solution for OS X?
I have about 700GB of data on my G4 tower and disks as well as a XServe with about 600GB of storage.
.
Stand back for a moment and ponder the following: What kind of person would keep thousands of images, which are never going to be edited or manipulted again, which are simply records of your travels and life - in the form of RAW bmp files?
Nobody. Absolutely nobody. Yes maybe if you're intending to do things with them sometime in the next 6 months, you should shoot and keep raw or very high quality versions. But if it's all for posterity and "maybe's", perhaps it's time to do some encoding.
Please do NOT tell me you're storing tons and tons of raw uncompressed video that you are unlikely to ever edit or retouch ever again. Yes, maybe you'll have to choose your compression format carefully, just in case you decide to edit/splice something different together some other day in life, and perhaps you'll have to use high bitrates.
But there's no way in hell I'm keeping around GB sized 30 minute home videos for the rest of eternity that could easily be converted into high quality xvids with keyframes every 30 frames.
.
If you are going to compromise for something cheaper and simpler I suggest rebackuping everything every 2-5 years and not destroying the old media and checking the condition of the older storage medias. This way if the new storage media is unreliable, there is high likelyhood that the older ones still work.
with that much data once you have a crash you won't remember half of the stuff you had anyway. Ignorance is bliss.
Buy a bunch of miniDV tapes from ebay or a stock clearance and just use them? Or if you can compromise on quality then even VHS?
If you want a higher level of home/small business office fireproofing, look at gun safes. They have to be more heat resistant to contain a cookoff.
Example - I don't work for Cabelas but I order alot of shit from them so thats where I went to off the bat.
"Stack-On Fire Resistant Personal Safe
Perfect for storing valuables, documents and more. Both of these fire resistant safes are ETL verified to manufacturer's fire protection specifications. Up to 1,700 F. for one hour with the interior temperature remaining below 350 F. Solid steel, pry resistant door with four-number combination and key lock provides greater security. Holes for mounting to the floor are pre-drilled. Fastening hardware is included."
http://www.championsafe.com/tech/fire.asp
DVD life studies: "DVD Rot" / DVD Longevity and Reliability (9/2003).
Inbound pictures are stored on the work machine.
Every now and then these are moved to a RAID5 array.
Any totally critical stuff is also backed up onto DVDR.
Not optimum, but unless you want to spend a ton of money......
r.
While RAID was originally meant for data security, availability, and consistancy, it has a lot of other applications that weren't in it's original design.
First of all, disks are *so cheap* these days, hard drives are a more than acceptable backup medium. As disks tend to be identical in size and construction if you buy in batches, disk-to-disk backup is quite the good system, just as long as you don't always keep the disks in the same location (aka, not even on the same controller! *gasp*)
Secondly, you went into a lot of specifics that I didn't care to; a lot of backup systems are custom tailored to the situation.. so while this kind of system might work great for you, I doubt if it would work so well in this case, especially. Digital media tends to be very non-compressible, very volatile media. That being said, operations like MD5 are very crucial to insure the data from one location matches another, which means even more precautious MD5 storing measures. You're also dealing with larger files which means rdiff is almost entirely out of the question.. I could go on and on about different, application specific schemes, but I feel I did good enough with suggesting three different mediums and to have at least two copies of two of them, preferably in 4 different locations.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
<ikkenai> i don't have hard drives. i just keep 30 chinese teenagers in my basement and force them to memorize numbers
These are the only ones I can trust to be around in 100 years or more.
*All* digital images get written to CD-Rs are are stored in a commercial document-control facility. But the ones I really want to keep get written to film.
Best Buy can have you arrested
For my photos, whenever I've taken enough to fill a CD, I burn four copies, two each of two different brands of media. I then mail one of each type of media to a family member in another state, I put one of my two remaining copies in a cool dry closet stored in a plastic jewel case, and keep the other at my desk for use. (My family member sends me discs of their photos to store too.) I swap the copies from the desk and the copies from the closet occasionally.
If I find that one copy has gone bad, hopefully one of the other three copies will still be readable. If one brand of media goes bad first, there's a copy on another brand of media. If my house burns down, I ask my family member to please give me their copies of my photos.
Next year I plan to also put a 2 terrabyte RAID 5 server in the house and keep all my media on that. I still want to send backups to my family for offsite storage, but I won't have to back up the whole thing: I can do incrementals mostly, and do a whole system backup only once in a great while.
I save DV as 4:2:2 MPEG2 created with TMPGEnc Plus with 2 passes. It's almost indistinbuishable from the original and one hour will fit on a DVDR.
If you're only getting 11G/hr on your miniDV, something's bad wrong. miniDV is 13G/hr.
Considering how paper has proven itself as a storage medium over millenia, why hasn't somebody come up with a mechanism for storing digital data on paper to be read later by a scanner?
If you consider an 8x11" page and a 1200dpi printer, you could store 253MB on a single double sided page, 126GB in a 500 page 'book'. You would probably need to develop a special printer and scanner system, it would be expensive, and the 'access time' would probably be slow, but it would last for ages, unless barbarians burn down your library.
What was the last law that benefited people but not corporations?
Other than a disk array, I would suggest buying some huge SATA drives and a tape drive. Maybe setting up a home file server to store your data.
Just my two cents worth.
Umm... they do. It's called the film scanner.
About 3 posts ago was info on a .5TB raid for $250, it doesn't get much better than that..
I concur, but why bother with the fireproof safe? I really doubt that a DVD-R could survive the heat that transfers through the safe given that they can't survive direct sunlight very well, either. Best to make sure that the really important stuff is in two different physical locations and hope that they aren't both burned down at the same time.
Also, you should periodically make new copies of both DVDs. I have some CD-Rs that haven't lasted three years, and I doubt that DVDs last as long, given the higher data density.
I have about 200GB of pr0n. What the best way to store it?
See? That was easier.. no need to come up with some elaborate story about raw DV files and SLR cameras.
--
Stay tuned for some shock and awe coming right up after this messages!
As a video editor I can tell you how editors back things up when it comes to editing. When doing a video project (in your case DV video) the vast majority of your media sources come from the original video tape. You use your NLE (Non Linear Editor ie. Final Cut Pro, Premiere, Avid, iMovie, Vegas etc) to log the tape and then acquire the media to a hard drive. Once that media is on the hard drive you can then edit it. When you log the tapes you are telling the software to suck in the media on the tape starting at one timecode point and ending at another. This information is saved in your NLE's project file (the one you double click to open the project.) This means you can edit a show, make a master tape and then delete the media off the drives. If in the future you ever need to restore the project again to work on it some more all you need is that tiny project file and the original tapes which of course you have saved. You simply need to reacquire the media off the tapes again. Any other media used in the project such as music, pictures or graphics, as well as the project file itself can easily be saved in multiple redundant locations as these are far smaller files then the actual media files off the tapes. This is the time tested method used by editors the world over. Few editors ever mess with backing video up to DLT tape or anything like that. Thats what the video tape is for. Good luck!
If you think about it, there's no reason to do a tape backup as long as you keep your master tape. If you get a tape backup drive, then copy your dv files to it, you've just made a copy of what came out of your camera. Sadly enough, if this video is something you want to be sure you have years down the road, your best bet will be to find a good quality analog tape in a format you like and backup raw footage to that. Sure, you'll lose quality with analog, but it can withstand aging and oxidation better than digital tape. With analog, you just get short dropouts in your video, with digital, you may get huge chunks of video that just go away. Keep all your master tapes, stripe them with timecode, then when you're done editing your project, dub it out to whatever tape format you're using, save the masters, and save the project edit file. If you need to recreate the project, you can just batch digitize everything from tape again.
Why not get a "cheap" USB/1394 hard drive at about 300GB or an enclosure, a copy of Drive Snapshot and a Sony AIT-E tape and back that sucker up right?
Your original DV tapes are, of course, your best backups and writing your intermediate cuts back out to DV preserve a lot. DV tapes are cheap. I use Final Cut Pro and save all the project files. These are relatively small and along with your original field masters can reconstruct your project. Sound masters can be saved in the same way.
I backup all my photography - about 300 GB/year at this point - to hard drives that are in trays for a FireWire enclosure. So far this works very well. Film, of course, is its own backup but for digital it helps to have two copies on different media.
The problem with tape as a backup media for photography is the difficulty in access. Mostly you want to be able to use a resource tool such as Extensis Portfolio to catalog your work and then point the way to where the image(s) you want are stored. On tape it can be a timely matter in finding it. On HD or DVD it's a few minutes max.
ab
. . .if you delete one item in a RAID it is SUPPOSED to be lost to the entire array.
Deletion, however, is not the same thing as corruption.
KFG
Actually, I was wrong.
I take back what I said before. I could have sworn it was the other way around.
Never mind.
Couldn't you stick the CD's inside a thick book or wad of paper for insulation?
The same principle as making the whole plane out of the stuff they make black boxes with.
What do you mean "you foo'"?
Do what I do, use paper tape...
I work for a local photography company and all data saved to our systems must last 4 years before it is moved to an alternate location.
I have setup the system where two 250gb drives are raided as one 500gb drive. There are two external 500gb drives, backup A and backup B. Backup A is always connected to the server, and automatic backup software syncronizes it with the main drive every 2 hours (new or modified files, if newer). Backup B is kept off-site and brought in weekly to obtain the latest data.
I also have another interal drive whos main function is the recovery drive; when files are deleted off the main drive, one copy of each deleted file is kept on the internal drive for quick recovery.
If files are deleted off the main drive, the deletions are syncronized on the backup drives but the files are still accessable on the recovery drive.
Everything is fine and automated, in fact I'm getting ready to write a server side script to automatically email a copy of the log file of backups every day to my boss.
You just HAVE to make sure you disk cleanup and defrag the main drive at least once a month, or expect performance problems while working on images off the drive over the network.
There is also that Iomega backup software that backs up as each file is modified or created. The only problem with that is any small mistakes that wouldn't be recoverable (unless you have a recovery drive)
Wow, I should be sleeping now, but how can I sleep if I haven't read slashdot all day??
In my experience DVD-R media is worthless for backups, because the quality of the manufacturing is even worse than CD-R media. Even my expensive TDK disks have failed after six months. I don't know what to recommend besides DLT, but stay the hell away from DVD!
For critical/irreplaceable data, DLT is good.
:)
You can pickup DLT 2000XT (15gb) off ebay for a song, or a DLT 7000 (35gb) for a bit more.
DLT tapes are incredibly tough, you can drive a car over them and they won't get damaged. You can drop them down flights of stairs without worrying about them breaking. And the DLT tape path isn't a convoluted helical scan design like most drives, so you don't run into the chronic tape munching problems that eg DAT and other helical scan systems have.
DLT is all about reliability and durability. They aren't _enormous_ storage, but they're pretty cheap for peace of mind.
Forget about random access though
so the little monster in me pops out and expresses bewilderment at a certain kind of posting that has occurred on Slashdot lately:
"Hi I'm Bob. And I have a problem. I have this...I have that...I have that also...I've been doing this...my wife has been doing that...and now we've found ourselves with the following situation. I don't want to do X. I also don't won't to do Z. I heard about Z not being so reliable afterall...so I don't want to do that either. I want my cake and I wan't to eat it too.
So...
Here I am...I've painted myself into a corner so friggin tight...so tight that I'm controlling how people help me...so much so that I'm holding a gun to my head with my finger on the trigger already half way squeezed.
What to do?"
If you don't know what I'm refering to...then move along.
Night!
=8-)
www.backuptrauma.com
Watch the video
Host localhost (127.0.0.1) appears to be up
Fire proof safes aren't all heat proof. I was told by our insurance company recently that the FP safes and cabinets are only rated that for durable goods such as paper (which can withstand a lot of heat in some situations), metal, etc. but CD/DVD substrates will melt or distort rather quickly because they're so thin.
You just need the right kind of fire safe, one that is rated for class 150 or class 125 (tape & disk) instead of class 350 (paper).
I'd sooner put them in a safe deposit box at a bank where the vaults are much safer in most scenarios.
While I'm sure the bank has better physical security, they are also a bigger target. And safe deposit boxes have no protection from fire (although the bank probably has a sprinkler system).
For the past five or six years, I've been taking my data, applying steganography techniques to encrypt it into the background of porn images, and then distributing those images via usenet and a few porn sites I've whipped together (ok, ok, the bangbus videos.)
At any time when I need to recover the data, I just use google to find someone with a copy of my data, download, decrypt, and voila!
This is my cheapskate's Network Storage Device!
For when the big EMP hits.
"EMP - it's the only defense we have against the machines" - Morpheus
"EMP - it's the only defense we have against the infidels" - some future Ayatollah
How about a stack of punch cards? A really big stack...
To get something done, a committee should consist of no more than three persons, two of them absent.
In most cases optical media failures affect only a small fraction of saved information (DVDR dye flaking, etc).
Archiving repair information based on reed solomon codes, or similar, will significantly improve the chance of full restoration from partially failed media.
Check out PAR2, commonly used on binary USENET to repair partially corrupt or missing posts. It works great with unreliable media too. You can recover N bytes of original data using the remaining data and N bytes of PAR2 recovery info, regardless of which N bytes are corrupted or missing.
If you are worried about future availibility, also backup the PAR2 spec and portable C source.
If the volume you need to back up is manageable -- that is, you don't back up simply everything -- and have other necessary infrastructure, here's what I do.
Our active file server uses a RAID-5, and is backed up over a high speed (gigabit) line to another "backup" RAID-5 in the next building. The expense is manageable if the volume you need to back up is static; if you're continually producing new data that needs to be stored indefinitely, not so much. I figure the risk of total loss is acceptable because if both these buildings go at once, I'm not going to be around to care whether my backup survived or not.
If it matters, the production system runs Windows Server 2003 and the backup system runs Fedora Core. Production RAID-5 is an external Promise array that uses IDE drives and appears like a SCSI device to the Adaptec controller; backup RAID-5 is a 3ware 4-drive bay with IDE drives connected to a 3ware controller. Both of them work like a charm.
"There is no night so forlorn, no mood so bleak, that it cannot be infused with pleasure by tender meat..." - R.W. Apple
MORE ON USING JUST HARD DRIVES
I like and agree with arete's backup scheme. Hard drives are a flexible fast and powerful method of backing up. With enough redundancy you should, theoretically, be able to extend the life of the data fairly indefinitely. Here are a few methodologies which may compliment this backup scheme:
1. Local backup
It may lower costs and simplify your backup strategies if you backup to a second drive in your working system. I would recommend sticking with automated scripts as recommended and I would recommend still having one (or more) backup servers.
2. Offsite backup
A few thoughts in this area. It might make sense to find a friend who lives reasonably far away who would like to swap offsite backups. If you setup FTP or a VPN connection you could automate backups to a backup server you bring to your friends (going along with arete's suggestion of using a very inexpensive system with the appropriate amount of hard drive space). If bandwidth is not sufficient you could store a hdd at the friends house once a week etc...
3. Local Redundancy
I would recommend using a RAID locally if your current data is difficult and/or expensive to replace if lost. The most recommended RAID solution currently is 1+0 or 10 (same thing). This is a non-redundant array of disks which is mirrored. It is fast but very redundant (unlike stright mirroring) The reason for backups is to avoid the loss of data. This type of RAID (as well as normal striping or RAID 5 and mirroring which is either RAID 1 or 0[I always forget which is which]), is designed to avoid the loss of data. Backups are great but never having to use them is even better. Good Luck!
It seems to me that an HD sitting on a shelf would probably last longer than a DVD or CD, is way faster to access, and of course holds a lot more in the same amount of space. It's also better sealed and protected.
However I'd advocate at least two of those drives...
"There is more worth loving than we have strength to love." - Brian Jay Stanley
I think you are keeping too much stuff; weed it down some more and get rid of the dreck. With the cheapness of digital film, the urge to over-shoot, achive and catalog everything is overwhelming, but most of it isn't going to sell.
For archival / long lasting, flash is probably the best way to go. Get some large flash drives (Toshiba is even selling 2.5" ones that plugs in as HD now), and write to those, and put it in a ziplock bag with some dehumitifier and toss it in a freezer. In even normal conditions the electrons in the floating gate takes years to quantum tunnel out, and lowering the temperature would make that even slower.
then I imagine you wouldn't be opposed to spending some money on a good solution to make sure you are covered in a disaster. 1. Get a NAS device and replicate all data there for disk to disk backup. Make copies of your pictures and videos to DVD and keep them offsite (most likely in a safety deposit box that is relatively far enough away in case of a tragedy like earthquake, wild fires, tornado, etc.) 2. Get 2 NAS devices, replicated data from location a to location b. (these locations could be in your basement and your friend's basement). Still do the DVD backups in step 1. I think that is about as foolproof as you can get. Then again, it is an expense that you might not be willing to pay for if indeed the pictures and video isn't really invaluable.
I have not a professional, so this may not deal directly with this subject. But here it goes. After buying so many cameras (video and still) to document life, I have run into this same issue. What do I do with all these photos? Do I even need to keep them backed up? Then I had a revelation. Why not use the gray-matter between my ears to "remember" my life. Yes, I know that these events happened in my life without proper digital documentation. Give a shot some time.
Do it the Torvald way! ;D
Just create nightly torrents of all your important data and let us "archive" it for you =)
A "media" safe, say the Sentry 6720, with about 4x8x8 inches capacity, runs about $300. I recall the manual saying it was some kind of water buffer in a sort of gelatin like state, that slowly burns off, keeping film and CD's safe for the required time. It is noticably heavier than the much larger letter-size fire safe for paper.
Also, I wouldn't really worry much over fancy locks. I remember going into a safe shop where the owner pointed at a menacing 3 ft tall safe with the whole mechanism ripped apart. He said it was an easy 15 minute job with a crowbar. I hang the keys to my safes in plain sight. I'd rather a thief pop it open and decide it's not worth hauling 100 lbs for simple home videos or pre-school art projects.
My method:
1. Raid 1
2. Nightly auto rdiff-backup
3. Occasional manual rdiff-backup, elsewhere
4. DVD-R DL
Get 2 computers, each of them with a harddisk beeing able to hold all your data. Now use rsync to copy all your data from one computer to the other regularly. Together with hardlinking, there are even simple ways to even make "incremental" backups.
:)
Now the really great thing here is, if you use both computers, you will immediately know if your normal harddisk or the new one failed. When it failed, you can immediately act on it, replacing it.
The system also scales, as you will probably replace your computers from time to time anyhow. Then you just move the data from the old one to the new one. In fact, you can even buidt some kind of "chain" of computer upgrades. You buy a new computer and use the next older as a backup computer.
This solution is not only cheap compaired to tape or optical disk backups, but it also scales, and, if you are carefull, you can keep your data far longer than the tape recorder manufacturer exists.
I guess this is ideal for home backups. In my installation, the backup computer actually is about 100km away from the main one. As you use rsync, encryption is no problem at all, and you only transmitt changed data.
It was called the IBM 1360 Photo-Digital Storage System [wikipedia.org].
Damn it I was getting ready to right a patent and cash in if anyone else came up with one....
I've had problems recovering data from tape (dds), old CD-Roms, Iomega Zip disks (no shock there), and normal hard disks (in my case, a 52MB Quantum disk that used to hang off my Amiga 500, was completely blank after a couple of years on the bookshelf)
For backups at home, I now use Fujitsu "DynaMO" 640MB magneto-optical disks and a couple of drives for my home backups, that I aquired from a local uni, along with a bunch of disks.
From what I've read, Magneto-Optical is pretty much unbeatable for long-term archival storage, in terms of media reliability and longevity. The manufacturers claim 100 years plus archival life.
As the media is in a cartridge, it's hard to accidentally scratch. Due to the way it works, the disk must be heated and subjected to a strong magnetic field to accidentially erase or change the information contained on then disk, so it's reasonably safe from magnetic interference.
More importantly, all brands using this technology I know of (Sony, including MiniDisk, Fujitsu, Hitachi, HP) are generally backwards-compatible; ie I can record to and play back 230 and 60MB disks on my 640MB drive, and I can read my 6240MB disks in the latest model 3.5in drive from fujitsu also.
so, in 10 years time you can probably find a working drive that can read back the media. This is a big problem with tape storage formats (can anyone sell me a reel-to-reel quarter inch tape drive?)
Recording speed is pretty slow on my setup ie maybe 300-400Kbps but playback is about as fast as DVD-Rom. I know the technology has advanced to hold more per disk, and performs better.
i've had no complaints so far, and absolutely no cartridges i cound't read back (yet.)
Send it RF out into space and when we as humans learn to go faster than the light speed, u can go out far enough and retrive it.
I heard I love lucy original telecausts are just leavin our solar system.
Brad W.
I have half of your problem, namely the digital photo storage problem. My solution has been to copy the really important photos to negative film. It's usually expensive, but luckily a friend of mine has a digital film recorder, so I only have to pay for the film.
You can get a "decent" film recorder for 35 mm film for under $1000, but if you want better results you'll need to spend more...
Here are a few you can check out if you like:
http://www.ctcsouth.com/
There are companies that will transfer digital video to analogue film stock for you, but they are REALLY expensive. If the video is really important, you can check them out. Example: http://www.cinebyte.com/
On the off chance that a robber does rob the bank, is able to rob the safe deposit boxes and for some reason deems what's in yours important enough to take, you could just burn another copy from your original which you still have on your computer.
Remember, this is a backup, not an archive.
sounds like you need BackupPC
Your hair look like poop, Bob! - Wanker.
The answer is to have redundancy at distributed locations. Have a backup server at a hosting provider and copy the data over the network. The hosting provider can be around half of the globe, and you can tightly secure a dedicated backup server. It should even be possible to do a continuous backup over the network.
For a personal archive, you can also consider to carry a copy of your backup media to a bank vault.
I see the future techniques of digital archaelogy as analogous to the tricks present-day archaelogist and paleontologist resort to to piece together the past.
I think it's necessary to make a distinction between the types of digital data you're storing. There are types of data where a damaged part does not equate to a damaged whole. Take for example, an mpeg video. Just because the first five minutes of the video are damaged doesn't mean you can't watch the rest of the two hour movie. Of course if your movie is in some encrypted format where every bit counts, you're screwed.
It's just like having a Greek statue where vital parts are missing, perhaps because of vandalism. Provided enough of the statue remains standing, an archaelogist can have a fair idea of what the whole statue must have looked like from the missing pieces.
I'm a sci-fi vegan: I don't want the aliens to think we have as much right to live as the fried chickens we eat.
In Linus' imortal words:
"Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it."
"goodbye and hello, as always" ~Prince Corwin, from Zelazny's Amber series
Infomation now is much more perminant than it was in the past, and digital has improved this a great deal. The amount of information we generate these days is enormous, far more than ever before the digital age. Thus it's not supprising much of it gets destroyed. For that matter, most of it isn't worth saving anyhow.
Books are not such a perminant media as you might think. They wear out, and can be destoryed. A good example is the Mayan Codices. Records seem to indicate there were thousands, however Spanish priests burned them as "works of the devil" during the European conquest of the Americas. Today only 4 remain.
Digital data can be so perminant because it is so easily copied. Perminance of data does not come form trying to make a single, eternal copy, but from having many copies all over the world. Digital data can be copied for essentially zero cost very easily. Thus it's easy to give it a great deal of robustness. Also, as new formats come out, you simply copy and convert the data. I have data on my harddrive today that orignally existed on 5.25" floppy for the Apple II. It has simply been copied and converted a number of times.
Finally, it's not like book are going away. On the contrary we publish millions of works a year amounting to billions of books.
You seem to have a false sense of perminance, as though in the past things were archived forever. That's not the case, actually, most data was lost, that's one of teh reasons we have such an incomplete picutre of history. You don't even know all that was lost, because the record of it even existing, if there was one, is also lost. What has survived is by chance, or by effort, not because we had some wonderful archival system.
You don't have to have something on an immutable, indestructable medium for it to survive. The Nordic Legends weren't written down for centuries, yet today we still have them. They were passed down, as an oral traditon for generations. There was no perminance to them other than stories in people's minds, yet they've durvived thousands of years.
This works for photos at least:
Step 1: Find out when the next Google Maps satellite image will be taken in your area
Step 2: Stand outside your house holding the photo facing up towards the sky from before the satellite image until after it. Alternatively you could lay out the photos on your roof.
Step 3: Do this every time Google updates their maps.
Step 4: ???
Step 5: Backup!
Art is not eternal.
Provide UPS can get it there without loosing it.
Firstly, please notice I'm an IBM employee working with IBM Tivoli Storage Manager (TSM) for eight years now.
;-)
Secondly I am a little disappointed in the answers you have gotten so far. Both in seriousness and in quality.
And as a remembrance to the first point, whenever you read TSM in my reply below, keep in mind there might be other software or solutions that could be of help...but I think myself TSM might be the best solution for you.
At first a small setting of the environment: you and your wife work with large files, and with a fair amount of total data. So in the end, you need high quality software and hardware that enables you to move all your data on all your computers offsite and/or to newer storage hardware. This hardware will be costly, but as you and your wife are professionals, you should take into account any extra costs for backing up and archiving as occupational costs.
If you do not have the amount of money needed...then you will always end up with solutions that are not answering all the needs you pointed out.
TSM will manage the data of multiple computers. These will need a network connection to the TSM server, which might be located on any one of your computers (it doesn't uses much CPU, but it will eat up 1 GB of memory though). So likely you could use any available old/spare computer you likely possess, as a DV professional I expect you to buy faster computers on a regular basis.
You likely want to keep track of your backups which is about storing different versions (that is, when editing a file you create a new version) of the same file over a certain period. A restore should likely be possible for say some version of some time ago. And sometimes you would like to be able to group some files together for backup and versioning, as the one without the other maybe pointless. And whenever a files becomes obsolete (say after ten versions or any old version after a defined period), it should be removed from your precious and expensive storage
You also definitely want to archive which is storing a partical version of a file for a certain period (maybe forever).
This is all possible with TSM and the right hardware. Have TSM perform backups regularly (scheduled daily or moreoften if you want) or manually started in between whenever you need one. You can chose what files to backup, and what not. You can also create archives with TSM, which will be stored for any period you define.
But then it becomes serious: what will you be using for hardware? With TSM you need a storage library which can store all versions in your backups and archives online, so you can access it without hassles. And likely you want to use removable media so TSM can make duplicates of your data which then may be brought offsite (any vault or shelf in another location).
I think you have two possibilities: either a tape library or a optical library. There is really not much against optical libraries if only that tapes like LTO3 may store 400 GB uncompressed (800 GB compressed), which exceeds any optical device by far. But do not compare filesize to device storage as TSM will split any large file over any amount of opticals or tapes if needed.
Lastly, you and your wife are a DV/photo professionals, not specialists into business continuity. So you likely want a one-stop solution that is easy to work with, uses TSM and a library, and is pre-setup. You then might take a look at www.storserver.com. They have solutions with disk-based storage only, or additional tpae libraries etc. etc.
There might be others I do not know of, please consider an IBM Business Partner for more help in this.
In the end: such storage is much more expensive compared to USB-HDs. But these won't last forever, do not keep track of different versions in your backup, manage your archives and it won't keep all your backup and archive versions online.
Still, you could use maybe a single USB disk for each of the seven days for backup, and use additional USB-HDs for archives. As soon as your daily data exceeds the USB-HD capacity, you come into trouble again.
What a bogus answer this is. Just buying hardware like a NAS-box does not manage any of the data nor will it bring the data offsite by way of thought. You need SOFTWARE too and removable media to accomplish this in a sound way. Please read my own reply for a more serious answer.
We can create permanent archives of digital data, it just requires the entire resources of the planet.
First, back up on stone tablet punch cards. Those egyptians knew what they were doing, that's why their newspapers/murals are still around.
The downside is that an archive of a video like "One Night in Paris" would require the amount of stone contained in Mt Everest in order to ensure the punch cards/holes are large enough to withstand the ravages of time.
Also, we'd pretty much have to get every human on earth involved in creating such a video archive. This would almost eliminate the food shortage entirely, as it would largely lead to a huge drop in the world's birthrate, as everyone's busy chiselling, and getting sexually excited over a stack of stone tablets with holes is not very likely.
I'm sure with a proper study, we could find arguments for and against this proposal, but the fact that Bob Geldof doesn't have to put out another Live Aid concert is enough reason to at least try this.
OK - point for point:
Thus it's not supprising much of it gets destroyed. For that matter, most of it isn't worth saving anyhow.
That's not the argument - the problem is the evanescence of digital media itself. It's not a question of most - it's a question of ALL.
Books are not such a perminant media as you might think. They wear out, and can be destoryed.
I didn't say they were - they are merely MORE permanent if they are made properly. furthermore, the *context* of their information is much lower - all it takes is paper and pen and you can (carefully) copy the data *with no loss* of the "original* message. This is how the Bible and other "important" works were maintained over the centuries.
DIgital data requires a very high context situation for its copying: it MUST be copied to another digital (drive) or digital supporting substrate (tape). Tape breaks down (I occassionally work in tape restoration - tape SUCKS for storage. Sticky shed gets you sooner or later...) and drives die and corrupt (I found that out the hard way last month when my main computer AND my back up both died within 2 weeks of each other. I lost a LOT of data...)
No one can sit and copy out trillions of ones and zeros - there isn't enough paper. Digital requires a huge and wasteful industrial system, which has been proven over and over to be unsustainable. Something's going to go, and I would submit that video and digital audio will be among the first to go.
The Nordic Legends weren't written down for centuries, yet today we still have them. They were passed down, as an oral traditon for generations. There was no perminance to them other than stories in people's minds, yet they've durvived thousands of years.
Then I suggest you learn all your favourite slashdot posts by heart so you can pass them down to your grandchildren, assuming we all don't starve to death with our kids in a refugee camp in Oregon in 2032.
RS
Shoes for Industry. Shoes for the Dead.
Dont punch cards pose the solution to this problem, coding peice of data in binary, then storing them as punch cards, that should preserve them aswell as any painting!
The video files that your NLE uses are exact duplicates of the data your camera writes to DV tape. Take a hint from that and just save your DV tapes. All modern NLEs work with EDLs (edit decision lists), so save your session files, overlays, transition parameters, etc to a CDR and push the lock-tab on your master tapes. Keep your tapes labelled and organized so you wont have a problem finding them again. It's trivial to recreate your project at that point, and it thankfully isn't MPEG-compressed on a video DVD.
Alternately, all modern NLEs have 'export to tape' functions. Just record your final product back out to your DV deck or DV camera and make a master archive on tape.
Someone please mod Ralph's post up.. it's the best commentary on DRM I've ever seen.
--- Grow a pair, liberals... stop letting the Republicans bully you!
Perhaps the questionner could clarify exactly what he is trying to protect against:
1. house burning down
2. disk crashing
3. PC being stolen
4. accidentally deleting files
etc.
Personally, I think RAID-1 is great for (2) and use rsync backup to an offsite machine to guard against (1) and (3). I don't care much about (4). But I'm dealing with much smaller amounts of data than the questionner.
No you're going on with the load, the load of shit that all digital data is important, that it all needs to survive. No, not so much. Most of what we generate will be lost, and that's just fine. Been that way for centuries. If it is important, precations can and will be taken to preserve it.
/. posts, as the amount of data increases, the amount that doesn't matter also increases. I post on /. for my own amusement but I do not delude myself in to taking history will have lost something of any importance should it all be deleted.
You like to talk about the bible but realise the immense effort that went in to each copy prior to the printing press. It was an amazing amount of effort to copy all of it and attempt to do it without error. These days, you spend 1/10th the effort and you'll have something far more perminant.
If I have DV that's important to me, $50 and a bit of time will allow me to archive it in several stable formats that I can then place in controlled settings.
Generally, however shit that's important isn't that high priority. I have "important" data, in that I don't want to lose it. However it's not important enough to make any serious backup attempt of. Having it on 2 seperate servers with redundant drives is good enough for me. The world won't stop turning if it's lost.
This concept that because there is data that has little perminance somehow manes all data, espically important data, has no perminance is fucking stupid. Ya, your home movie of your kid's 8th birthday party that no one cares about and that you'll never watch on DV tape will probably not survive. Important data, like weather information, will because it's copied thousands of times over to stable formats and stored.
You seem to forget that, much like
Anyone know of a good barebones online storage service? There are companies like xdrive that provide fancy interfaces to 5GB of storage for ~$10/month, but I'm wondering if there's a company catering to geeks that sells a bit more storage for a bit less with some minimal interface (ssh?). I've had no luck digging up any such company on my own; anyone else find something interesting?
This site was recently featured here on /. for another mod. Now he has come up with the Poor Man's RAID Array. I would check this out for a cheap RAID 5 option.
I must agree, rsync kicks ass for doing backups. I'd be wary of raid though, I've had too many problems with raid systems, and besides, the simpler you keep your setup the more reliable it will be.
../ | tail -2 | head -1) myuser@myipaddress::mydir/ ./
Here's my system:
1. I have an extra server laying on the ground, under my 500W subwoofer, in my parents basement.
2. I turn the backup server on and mount an encrypted drive with all my backups.
3. I cd to my directory and do something like "mkdir $(date +'%F-%H') && cd $(date +'%F-%H')".
4. I type rsync -vax --stats --progress --link-dest=$(ls -1
And $(some foreign french exclamation)!, you only download the files that have been modified since last time you backed up.
The overhead of hard linking is something like 10 percent, in my experience.
Apart from the question of how long various forms of media last, don't forget that hardware becomes obsolete too.
I still have the odd (analogue) 1" video tape, but I don't know of a machine to play them on outside the local TV station.
Whatever method you use now, you're going to have to be prepared to re-invest in new technology periodically and be prepared to copy the whole lot while you still have working hardware...
If it's your work as in job, then I'd suggest to get with the program and get yourself some serious storage space, maybe a dedicated server that you can easily add HD's to. Because any AV studio (even if only run by 1 or a few people) needs storage. If it's not your job, then I wonder how important it is to store everytinh, and store it in DV. Be realistic about it. Do you only watch it on the computer, then encode to DivX, or if you watch it on TV as well, make DVD's. Yeas, the rot, so you'll have to copy the discs once every 10 years. If you insist on keeping it all in DV format, then buy a tape backup solution. Tapes don't take much space and can hold a lot.
You need to store a lot of data. You also don't need all the complicated overhead of configuring RAID, or dealing with tape backups or anything else. You want to spend your time on your work, not on the technical solution.
The best solution for you is to use USB or Fireware HDD drives. Simply buy the drives as needed (in a capacity that gives a good price-point, e.g. about 120GB now), and you don't need to buy performance, e.g. choose 2MB cache drives, not 8MB. You then need a USB/Firewire HDD caddy.
You then need somewhere to store them - buy a fireproof drawer/cabinet/etc to put somewhere in your house. As you make your backups, store the hardware in this place.
You then need some way to make the backups.
I'm not sure about the solution here, because I'm not sure whether you want backups, or you want archival storage.
If you want backup, you want an off-the-shelf desktop backup solution, that'll just allow you to make periodic incremental backups using the USB/Fireware HDD as the destination. This will be a monthly chore, for example.
If you want archival, then you just need to store your media on the drives, but retain thumbnails or an index, so that you can search and choose which archival drive to pull out of storage. I'm not sure of good software here either.
From what I'm hearing about your situation, you probably want archival, because you're generating a lot of data that you don't always need to have fully online.
If that is your business, props for starting a business that lets you take pics of hot chicks. ;)
A witty saying proves you are wittier than the next guy.
>> One thing good about paper & film is they withstand decades of storage vs. years of normal magnetic storage. Photos and films from the late 1800's/early 1900's are still around whereas you're really gambling with current storage media.
So, all you really need is a good laser printer and lots of paper. Oh, and some Chinese kids who can type 1's and 0's quickly should you lose something...
PocketGamer.org - For the gamer on the go!
I do backups with dar http://dar.linux.free.fr/ on my sever system and on my workstation. The server copy I send to my workstation with scp, and the workstation copy I send to my server. Both are checked with an md5sum when they arrive to ensure they came to the other side without corruption or lost bytes.
The chanses for both systems to break at the same time is kind of slim. Unless of course a case of fire occurs. But then I've got bigger problems then lost data to care about.
Hey! That's my sig you're smoking there!
I feel a lot more comfortable having a full backup occasionally. Every quater is enough, just something so that if your initial one is faulty you can restore rather than relying on every single incremental record in a long sequence (and having to restore it all incrementally)
I am trolling
It is demonstrated here that CD-R and DVD-R media can be very stable (sample S4 for CD-R and sample D2 for DVD-R). Results suggest that these media types will ensure data is available for several tens of years and therefore may be suitable for archival uses.
So that doesn't sound like a bad medium!
The trick, as the paper also mentions, is knowing which brand/manufacturers are good and which are crap. So maybe the DVD-Rs in the bargain bin aren't really a bargain...
Am I part of the core demographic for Swedish Fish?
Uhhh.. you think tape backups are very cheap these days? Where are you looking. I see 20/40gb drives for like $500. There is some new iomega deal that is 35g for $299. Anyway I wouldn't call that "very cheap". Not as bad as 5 years ago but still ridiculous considering. I can buy 10 40g hard drives for the cost of a damn tape drive. Quite annoying. I really hope someone replies to this an proves me wrong. I'd love to find a reasonably priced 100-200gb backup solution.
Personally I just prefix all my files with "porn " then share my entire drive on Kazaa.
Works for me.
This is probably not an answer to your question but interesting to other readers with less big photos. FLickr is a nice place to backup data because it is relatively cheap (35$/2 years or so) and you can upload 2GB/month. This works wuite well for me and saves me with the trouble of further HDs and DVDs.
I use dvdisaster to protect my DVD backups. At its default settings it creates a Reed-Solomon correction image that is 14.3% the size of each DVD I write. I archive these to DVDs, which are also processed by dvdisaster. If a DVD goes bad, I can use the dvdisaster recovery image to recover many, but not all, errors. If the recover image is bad I can use its recovery image to recover it .... See the dvdisaster webpage for more information: www.dvdisaster.com.
Use an almost lossless codec like Apple's quicktime pixlet that you can easily access and edit into other distribution ready formats.
If you also want more space savings at ALMOST H.264 quality without the immense time to compress, try exporting to 3ivx quicktime at qp1.
Once you have a smaller archival format like either of these, then focus on your backup media approach.
Cheers,
- Zav - Imagine a Beowulf cluster of insensitive clods...
Oh, the irony that this was posted only a few hours before...
Just use this baby as a backup machine and keep the originals on your computer somewhere. Then you've got two copies which will likely survive any data loss not related to disasters like fire or flood.
"No problem. I have the capacity to do infinite work so long as you don't mind that my quality approaches zero."-Dilbert
yeah that's also about $10,000. In this guys case, that's a waste of money.
Storing it in a fireproof safe will not save it from a fire as "fireproof" simply means that no flames will get in...but if it heats up to 200+ degrees (F or C) the disc will be damaged.
Fireproof safes were designed for protecting *paper*, which has a fairly high ignition point as long as they are not in contact with a flame, a fact which Mr. Bradbury used once as a title.
ttyl
Farrell
CAN-CON 2019 - Ottawa's only book oriented Science Fiction Convention! October 18-20, Sheraton Hotel, Ottawa, Canada h
The last thing you want is to need your backup media that is stored offsite and finding out that your friend spilled his Cheerios all over it.
A bank safe deposit box would be much better.
Do you have a dollar amount that describes the criticality of this data? Without knowning that, how can I possibly design a system of backups that isn't too expensive, yet be considered by you, my customer, to be a reasonable effort/attempt? Heck, any backup strategy should include the required time to recovery and offsite storage plans too.
In short, if you don't want to spend 1 penny more than necessary and are willing to take risks, listen to the advise you get here.
OTOH, if you are serious, we can discuss LTO-2, STK-9840C, and EMC. If you are really serious, we can talk, SRDF/A and GigE links across town (50km).
My design rate is $250/hr, but it will work and meet your needs. You'll need $500K, minimum - probably $1.5M to make this happen. Reply to my message, and I'll contact you if interested.
Again, how critical is your data?
No.
They are a minimally acceptable backup media for short-term storage.
Consider the fact that with tapes, you really just have to worry about tape errors. If the tape drive fails, you can use another.
With hard drives, you have to worry not only about errors on the drive, but about hardware failures in the electronics as well.
In 10 years, that hard drive will probably be dead no matter what you do. But a properly stored tape backup would still be around.
I'm confused why someone who believes their data is so important, and obviously spend a good chunk of money on their equipment, would be so cheap when it comes to their choice of backup medium. DAT and DLT are decently fast, very reliable, and much larger than DVD, and can hold as much as a hard drive, or even more if you purchase a loader. These can be expensive as hell, but weight that cost out with replacing all of your data and see which one sounds better.
If your data is that important, spend the cash on good backup.
"Well you're not Fiona Apple, and if you're not Fionna Apple, I don't give a rat's ass."
The key to art and thus the main tool of every artist is selection. The affordability of digital media and ease of use mean much more material is being shot and the selection point has only been moved down the line from production to post-production. Be more strick for yourself: don't shoot everything and don't have the illusion that you to keep everything (unless you are an historian).
Also: DV footage is already on a tape. It almost seems to me that poster is digitizing media files to computer and then seeking a new way to get it on backup media. Why not backup your original recordings, work with reel name/timecodes and make meticulous backups of your projects only?
And: learn to finish a project. Accept it is over. Keep the end product (prints, dvds) and maybe the project file for reference and accept you have no longer a need for the rest. If it work, you've done the job, the client has paid the bill then it's finished.
OK I've seen six variations of the "Linus Torvalds system" proposed as jokes. But could this work seriously? Some combination of public key cryptography, along with strong antileeching facilities?
The idea is you donate 100G of drive space, and in return you get some smaller amount of replicated storage on other peoples' computer's.
I wouldn't sell this as an enterprise solution, but for home users, maybe it could work. Maybe it could talk SMB and NFS.
OK, why not? I see upload speed and security as the two obvious worries.
> Up to 1,700 F. for one hour with the interior temperature
0 .shtml#SEC759-SUBSEC3
> remaining below 350 F.
Fine for paper (Fahrenheit 451).
Not so fine for magnetic tape (125 F) or CD/DVD (248 F) media, both would be damaged long before 350 F.
There are a number of data media rated fire resistant safes that will keep under 125 F for an hour for a 1800 F fire.
From
http://www.kodak.com/global/en/service/tib/tib430
The glass transition temperature for polycarbonate is approximately 140 degrees Celsius. If the temperature gets within 20 degrees Celsius of the glass transition temperature, there is a likelihood of significant disc deformation.
From
http://vsg.cape.com/~pbaum/magtape.htm
Other than a fire, the real danger of high temperatures (above 80 degrees Fahrenheit) is an increase in tape pack tightness caused by wound in debris, tape distortion caused by this pressure, and possible layer to layer adhesion. Print-through is increased by approximately 1.4 dB for every 10 degrees Celsius (18 degrees Fahrenheit).
-- Dan Jenkins, Rastech Inc.
That's more or less the setup I'm using too. I have the big router/home server with a shared 120GB harddisk, and a backup server controlled by a timer. At 2AM the timer turns on the computer and it makes it's backup using rdiff-backup. As it only copies differences in changed files, I don't really worry about the backup size, at the moment it's merely 1.6GB. After the backup, it generates a log in html and copies it back to the main server.
All this I wrote in bash in one afternoon.
"It's too bad that stupidity isn't painful." - Anton LaVey
I've never used Netbackup, but at work, we have a 2TB Raid 5 HD Array, and thats the first level... Using Comvaults Galaxy package($$$$+), it Backs up to the Magnetic array, and then during lag periods (usually a few minutes later) backs that up to the DLT AutoLoad Tape Drives...
:)
:)
Before, it was Arcserve (it bite$!) and that would take time to merge a tape, find the file, and then tell it to retrieve the file... sometimes as much as 3-4 hours just on the merge tape if it was pruned because it was a week old...)
Now, with the new setup, and Comvault... about 5 minutes assuming the Tape is in the drive... 10 if I have to call over to the offsite and have them load a particular tape/date range...
Offsite? Yes, the Magnetic Drive array is the first line of defense, it is local to our server farm... The Second line of defense is the Autoloading Tape Drive unit, it is off site... Connected over "Dark" (ie, GB) Fiber.. RIGHT to the tape array... It's really bitchen that we don't have to "SHIP" to the offsite, it goes there on it's own!
(ie, MC and all those folks that lost data while shipping tapes via UPS... Think about it... )
Before, I used to hate users that called and asked for a week or older file.... Now, I still hate them, but I'm glad to do it because it is just a point & click.... Dump into thier folders during the restore... Done... another happy Luser.. And I can keep on reading my slashdot.
Oh... So I guess I should ask... What is NetBackup like?
--- Relax, that mass muderer is just trying to reduce our carbon footprint, one fetus at a time...
Try http://backuppc.sourceforge.net/
I currently backup 43 machines on to one 300GB HDD (which is then backed up to tape, but you could raid disks if that was your final target).
I am storing ~1.3TB worth of backups in this thanks to the pooling of common files and compression.
BackupPC can use rsync over ssh to do it's backups so even a "full" backup is quick and has less load than our previous branded tape-based backup and I am more confident about the security of it too.
With blackout periods and the ability to ignore such periods when a machine has been off for too long, backups happen when is good for you and can ensure that you get a backup if a machine has been off (or off-network) for a while.
BackupPC has proved itself to be a lot more reliable and faster than the several commercial tape based systems I have tried. (Not to mention cheaper - and that's including the cost of a dedicated server to run it on!).
Back in the old days they used stone tablets for that sort of thing. With the same drive technology applied to this like they used in zip drives. You could have some great fun when you press eject. Although if the disk gets stuck in the drive a paper clip won't cut it. Finally your backups will be around for quite some time baring erosion.
~~ Behold the flying cow with a rail gun! ~~
My solution to the backup problem is quite simple, two SATA drives in my work computer in a RAID 1 configuration, and two external 250 GBytes harddrives which I use as backup drives, for redundancy I have two, not one. The ideal would be to take one or both of the external drives off-site, but I haven't gotten that far yet.
It is very unlikely that all of my drives will fail at the same time, unless there's a fire or something of that order - and obviously, off-site drives would remedy that.
When new technology arrives, or I need greater capacity, it's a simple and fast procedure to move data to the new technology, at worst I'll have to add a new interface to the PC.
And most importantly for me, I can afford it; as this solution is quite inexpensive. You don't need any special hardware or software, just regular standard drives which are around 200 each, and that's it, you're about as safe as you can be.
Summa summarum, in the digital age you shouldn't count on archiving anything, instead focus on keeping 'live' backups, store data on media whence it can quickly and easily be copied to new formats or media, as they arrive or become cost efficient.
Oh, and for the DSLR RAW images, you might consider looking into the longetivity of the RAW formats, what guarantee do you have that there's going to be any software to read or process those proprietary files in 10 years time?
See http://www.openraw.org/ for more information. The same applies to any digital format, but the more specialized the format, like e.g. a particular model DSLR from a particular manufacturer, the less likely it is that you're going to have software to read it in the future.
You're entirely right. As a DR consultant, I come across many companies which keep their backups in the "fire" safe. Whilst this is fine for protecting against theft, it is just not suitable for storing magnetic or optical media. Magnetic is particularly bad as the tape tends to have a very low melting point.
There arew fireproof safes around which are "Media" friendly and have been designed specifically to keep tapes cool in a fire and they are usually rated to show how long they will work for and at what temperatures.
A fireproof safe is NO replacement for off-site backups.
Alex
Fry's here in Silicon Valley has a NAS (Network Attached Storage) box called a Buffalo that provides 1 TB of storage. It goes on sale periodically for under $700. The company also have a range of sizes including 1.6 TB version as well. I would get two of these units, periodically mirroring one to the other and placing the second one offsite.
i l.php?productid=99&categoryid=10) and to the 1.6 TB version (http://www.buffalotech.com/products/product-detai l.php?productid=100&categoryid=10).
N AS_RAID_1_0_5_Back_up/q/loc/10995/10396259.html)
Here is the link the 1 TB version (http://www.buffalotech.com/products/product-deta
A quick Google search shows that buy.com has the 1 TB units for a shade under $1K. (http://www.buy.com/prod/Buffalo_TeraStation_1TB_
-- Mache
First that comes to mind is Tape backup. They store huge about of data, and are very cheap these days,
I take it you haven't dealt with many tape backups systems... I strongly encourage the world to let those useless piles of crap die, once and for all!
First, they do NOT come cheap. Using AIT3, the bese size match to what the parent mentioned at 100GB raw (DV will not compress even a little bit, so don't count on the inflated "after our magic compression algorithm runs" sizes), the drive alone will set you back $1700. Then add in a neat $50 per tape - And he needs two per full backup, just as his current size. Now compare that to buying a few 200GB IDE drives - $86 each, and you could probably shave 10-20% off that if you buy a ten pack.
Second, performance. That particular drive, the AIT3, gets 12MB/s. Compare that to 30-40MB/s for a modern HDD, then thow in a seek time best measured in seconds (or even over a minute in the worst case)...
Third, durability. A lot of people will say that you need to treat a HDD more carefully than a passive blob-o-plastic like a tape. That holds true to a point, but you can't just toss a data tape in a box in the attic like old family photos. I've seen falls from waist-high (ie, on the corner of a desk) break them. Leaving them in a hot car while you get lunch will destroy them. Overall, if you really want to ever recover data from a tape, you need to treat it just as delicately as you would a hard drive.
And speaking of resiliance - True story. At my workplace recently, we had a rather important system (one that we really couldn't afford a full day of downtime on) fail critically (as in, people coming in that morning complained of the smell of burning electronics). Not a problem, because it ran on PC hardware and we had complete backups of the system. One problem - Guess the source of the problem? The tape drive itself cooked, taking out the power supply which in turn took out everything else (judging by the degree of damage down that chain). So, we had great backups, alternate hardware, and no way to restore the backup onto the new machine. We had a new drive overnighted, but still, it cost us a day of downtime that really hurt us. And for those who will say "Well, duh, you didn't truly have a backup if you didn't have a backup drive, as well - At a small company, you try to convince your boss to spend part of your already tight budget on a $9k drive that will, in all likelhood, never even leave its original packaging. BTW, yes, we use tapes at my workplace. I loathe them, for all the reasons I've mentioned here.
Overall, tapes just suck as a form of backup. Get yourself a pile of external USB drives (a little more pricey than EIDE, but you can hot-swap them), and backup to those. You'll pay a LOT less up-front, have a backup solution IMO far superior to tapes, and "the drive" contains its own reading hardware, so no chance to have the media but lose the reader. Get 10, do nightly backups on a rotating set of them, and take one offsite every Sunday rotated on a monthly schedule. And the entire collection of them will cost about half as much as a single drive and pair of tapes.
If the problem with optical backup is degredation due to air and light, then the solution seems easy enough. Put the discs in a vacuum sealed, acid-free plastic bag using one of these and store in the dark in a cool, dry location.
And burn a second set of DVD's for actual use, so you don't have to break the seal on the others.
Also, as history shows, storage media will continue to grow in size and decline in price. In five years, he will probably want to re-archive everything, anyway, to condense it down.
This solution may not be perfect, but it's inexpensive and IMO stands as a good chance of working as any practical solns I've heard.
Use layers. Buy a BIG gun safe (40+cubic feet) with a 2-hour fire rating (meaning, in an average house fire, paper inside will not char for two hours). Then purchase a number of smaller (1 cu. ft) "file folder" safes. Put the important docs in the small safes, and put the small safes inside the large safe.
The 2-hour-rating will ensure that the inside temp of the large safe will not exceed the char temperature of paper. So outside temp of big safe = ~1700F, inside temp = ~350 (guessing here?). The smaller safes, if similarly rated, will further resist temperature increases. IF the outside temp is "only" 350F, the inside temp will be much, much lower, likely well below 200F
Oh, and install this near an exterior wall over a concrete floor; for two reasons. First, in the event of a fire this is likely the lowest temp spot, and second that setup is going to be damned heavy -- too heavy for a wood-frame floor. A corner of a heated garage would be perfect.
First that comes to mind is Tape backup. They store huge about of data...
You've got a cold in your fingers or sth? :-)
Used 35mm film to archive data. Went the way of all projects at Kodak: Too soon, too expensive, too complicated.
A good example is the Mayan Codices. Records seem to indicate there were thousands, however Spanish priests burned them as "works of the devil" during the European conquest of the Americas. Today only 4 remain.
If the destruction of our civilization is going to be as sudden and complete as this one, the few surviving texts will definitely include works of questionable value like The Mayan Calendar and the Transformation of Consciousness. Later civilizations may indeed think of this age as a dark one.
You don't have to have something on an immutable, indestructable medium for it to survive.
The question is whether and how later generations want to remember us. The church and dark age kings did pass on some of the greater works of Greeks and Romans because the value of those works was at least recognized by some throughout the two millenia that have passed since their time.
History has been far less kind with the losers of history like the Egyptians, Sumerians, Mayas, Incas, etc.
The Nordic Legends weren't written down for centuries, yet today we still have them.
Germanic oral traditions are far less reliable as history. Note that there are also many reproductions of apparently classical and dark age works that are not taken seriously as a historical source. There is a variety of reasons to forge historical works.
Engravings on headstones, or big stone buildings in inobtrusive places like deserts seem to last very long. Unfortunately the texts people carve in stone (sentimental and religious drivel) are usually not the knowledge one would want to preserve for posterity.
The Longnow Foundation has the right ideas for making sure that posterity will rank us with the Eqyptians and Greeks. I think it's a good idea to add some nuclear fallout near to the monuments as a disincentive (or 'curse') for reusing materials, which has completely messed up the archeological record in densely inhabited places like Europe, India, and coastal China.
Odd that nobody mentioned Magneto Optical drives. Fujitsu still makes and sells them, and they are only a couple of hundred bucks. For those unfamiliar, read a description of how they function and you'll quickly see why they are so secure.
We seriously need to find a way to utilize our brains as a storage medium. Installable, pluggable chips that will integrate with the brain and become a mass storage medium. It needs to be indexed and/or refreshed often so that it is not "forgotten". Matrix-type shit ya know. We can make movies about these things but we can't seem to actually create them. Oh yeah, don't forget to open source it! Self-rating...Insightful, -2 long-time reader, first-time poster...forgive me if it was a bust.
I'm a (digital centric) professional photographer as well, but don't do or know much about the DV side.
:{)
:{D
I've started the following for my workflow and it works for me.
1) move all shots for an "event" to the system.
2) create CD-R's of the originals
3) process and create output for delivery and on-line,
not touching the originals except as input. I
usually have three stages of production
original -> psd -> final jpg (may be multiples)
If using RAW theres an interrum input stage to PS.
4) create DVDs of all work (too big for CD-R)
5) clean up intermediate files, and remove originals from system.
In addition - once a year I duplicate the CD-R's and the DVDs, keep the original-originals and toss
any in between then and now. I do this as a weekly
process, select a batch and copy them.
I figure that
a) two different kinds of media, less chance of loss
b) different stages of production so multiple backups built-in (yep I'm paranoid as well
c) when I do a compare of the first CD-Rs to the
latest I can tell (hopefully) what the longevity curve looks like.
I've not seen a failure yet, but it's only been 3 years.
d) the originals should be offsite for storage.
e) when a better technology comes along
(like I did going from all CD-R to DVD then to DL-DVD)
I can incorporate it into the flow and upgrade on the fly.
The biggest problem is forcing the regiment on myself to do this!!
With all this I've got around 200 CD-Rs and 250 DVDs.
I spend about 1 hour per job creating backups, and
1 hour a week duplicating old.
Lastly - don't worry about your paranoia.. you're
no crazier than the rest of us.. just deal with the realities
hope that helps,
Watchin oWo
One of his questions was backing up DV files over the size of a DVD. I use Winrar (you could use Winzip) to break the big files down into smaller chunks. I set it to 1,125,000,000 and four files fit nicely on a DVD. It takes 3 DVDs to back up one hour of DV video.
There is concern about how long DVDs last. I assume name brand DVDs should last at least four years. By then technology should have changed and I can move them to something larger and faster. If things aren't better copy them to new media and get more years.
For the truly paraniod make multiple copies, verify them in another drive and send a copy offsite.
While I love raid, RAID is not a backup - raid is about availability and consistency.
Nope, but when you add snapshot volumes to the mix, you get pretty darned close to the same effect as a backup. The main difference, in practice, is that backup tapes or whatever can be moved offsite. If you're not doing that, then, the difference between RAID+snapshots and real backups is basically zero.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
USB enclosed hard drives.
USB2 is fast and everywhere. As soon as SATA drives get cheaper switch to those.
Also look for drives enclosures with fans. They might be noisy, but not as noisy as the fan you might start putting next to it to keep it cool if you buy the ones without fans.
Few people need data retention for 10 years. For most of us, if we can reconstruct the current state of our drive after a crash, and restore files we recently accidently deleted or altered, we've got all we need.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
I nest one fireproof safe inside another. I have a little one (about the size of a briefcase) that I put all my computer media, camera film negatives, old Hi-8 camcorder cassets and other heat sensitive media into. That one then gets put into my larger fireproof safe in the garage.
If you already have a large fireproof safe, buying the smaller one isn't much of an expense. The advantage of this is that you have truely safe media storage that's still conveniently located at home. I take a new disk out to the safe every week. There's no way I'd do that if I had to take it to a safe deposit box.
You might consider getting a pile of EIDE drives and a just a handful of USB enclosures/adaptors - it's easy to swap drives into the enclosures.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
It's true that RAID is only for redundancy, and my data isn't as critical/irreplaceable as family photos/videos. I've been researching recently on a good RAID solution, and I'm thinking about going with a Promise VTrak 15110. It has 15 hot-swapable SATA hard drive bays. Add some big hard drives (I'm looking at 400GB Seagate Barracudas because I've found Barracudas to be reliable and they have 5 year warranties). All you need for a system to connect it to is a SCSI-160 controller. If you do the math, 15 drives x 400GB/drive = 6.0TB of raw data storage max. It'll get pricey (the VTrak alone is just under 4000USD on Newegg), but you'll have plenty of storage. At least that's what I'm looking at...
Is your CPU so horribly slow that you can't even fathom the concept of compressing your stuff ? You don't _need_ a 7mb picture of someone else's kids. You can chomp that bugger down to 1mb easily and still retain ridiculously decadent image detail. Even better for video, run it through a decent encoder and chop it into high-bitrate Mpeg-2 or something, or maybe something like DivX/Xvid in the 4-5 mbit/sec range if you're ambitious.
Just quit it with your uncompressed DV. The camera probably induces more video noise than the compression algorithm ever will, such is the nature of pro-sumer equipment.
But hey, I just happen to run an OEM store so feel free to order as many 200gb drives as you like. You can lead a horse to water, but.......
-Billco, Fnarg.com
You were calculating data per FRAME not per second, and at only 8 bits per channel, no less. Multiply by 30 for the 8 bit/channel total bit/sec, or replace the 24 with a 30 and do the same for the 10 bit/channel rate.
-- It only takes 20 minutes for a liberal to become a conservative thanks to our new outpatient surgical procedure!
I just built a RAID-5 array for storage space. I found a deal on Seagate 300 "GB" drives for $120 each. So, for a total of $488.48, plus a controller card, I am the proud new owner of 838GB of redundant storage space.
For protection from physical calamity, I use a box in a friend's room. Our unique situation means that, while physically seperated by over a half mile, we have a 11+MB/s connection between our rooms. I also don't back up all of the 838GB of data, however, and it's unlikely that a network system would be fesible for you. Another choice might be a RAID 0+1 array with a second mirror set, which would net you better performance. For the same price, you could get 556GB of formatted, fast data storage, and add $240 for a full backup nightly that you could store off-site (or at least with you, so it can be removed in case of fire). If you want to keep the originals around on your camera for one day, you could drop the first live mirror set and save some money.
It wouldn't be too hard to scale such a system up to the terabyte-level sizes that EIT-5 tape gives you while remaining well under the $10k cost of such a system. A lot more maintainance is required -- I'd definitely want to run thorough disk checks of the backup volumes every month or so -- but disks certainly aren't getting any more expensive.
I just picked up a pair of 300gb seagates (5 yrs warranty) for under $145 ea. (and I was in a hurry, could have done better at Circuit City if I'd have been patient!) That's less than 50 cents/gb. That works out to what, $5.50 for that 11gb of data your digicam captures per hour. If you want to make sure you keep the data even barring drive failures, get drives in pairs like I do and mirror them. It's not the world's most efficient use of space, but it's very easy to maintain and highly resistant to failure. It also lets you store the data in two places, preventing total loss by fire or flood such as you might risk with a single raid-5. That would drive your cost up to about a buck a gig, but for "irreplaceable" data, that's not really out of line. Also, IDE HDs are much more likely to be technologically accessible 10 yrs from now than (name your backup tape media). It's already getting hard to find replacement tape drives for tapes that were made 6-8 yrs ago. Try to find a drive to read a low density DAT tape sometime - you can find them, but not in a hurry and not cheap.
Take one set of your hard drives and drop them in a safe deposit box. ($15/yr) Keep the others at home in case you need access to the data. Physically label your drives, and keep an index spreadsheet so you can find the clippings you need quickly. Record the drive ID, the filename, size, and a short description so you can find it quickly later. Keep two copies of this index - burn one to CD each time you drop off a new HD to the safe deposit box - losing this index file would make for a mound of work to rebuild.
DVDs are nice for price when taken one at a time, but at (only) 4.7gb each, the time you spend backing up 290gb of data (60 discs!) that could have just been a drag-and-drop to a mirror and go to the park with the kids for the afternoon, the savings in your own personal time add up and break the cost-savings of optical media. Then there's the issue of having to segment files > 4.7gb, and that just makes matters all the more complicated and wastes more of your valuable time. Never forget, your time is worth something.
I work for the Department of Redundancy Department.
I have a similar situation as yours and used a homemade solution.
I purchased a fireproof safe and then put a rubber seal around the lid. I used a large drill bit (1 1/2 inch) to drill a hole in the safe. A friend of mine helped retrofit some sort of air pump to it so it sucks the air out and makes it a near vacuum inside.
I did this because I don't want oxygen or anything else to encourage cd/dvd-rot.
The studies have shown that CD's rot because of being exposed to oxygen, so I just tried to remove that component as best as I could.
Another solution would be to use some of that sealing plastic wrap and seal the DVD/CD in that, getting as much air as you can out of it.
Just a suggestion!
It was called the IBM 1360 Photo-Digital Storage System [wikipedia.org].
Damn it I was getting ready to right a patent and cash in if anyone else came up with one....
Not that that stops anyone these days.
There is really a very simple solution. http://www.newegg.com/Product/Product.asp?Item=N82 E16822144365
Default installation backs up every local drive automatically Full system restore to an earlier version Saves historical backup versions Simple diagnostics check the health of your drive with a single click
It stores 300 GB of data and even more compressed.
It has a 1 Touch feature so you dont have to copy paste or deal with any obscure backup software.
It does CRC checking to make sure everything it in tact after the backup.
Tapes are OK in a business environment where you need 3-7 copies of your backup and a swap schedule. But for home backups I would say Burn them to your favorite Optical media and stick that in a secure offsite location the utilize and external drive backup for daily checks every month or 2 you can run of another optical backup and put that in your offsite location.
With so much of the data being generated being mostly "worthless" - flash ads, me-too blogs, today's online edition of any newspaper, and so on, survival of the most "valuable" information will be akin to the discoveries in the pyramids and burial chambers of the world.
What 1845 newspaper are you referencing on a daily basis? How about that depression-era film projector in the media room? I don't know about you, but I don't play any Chubby Checker 45's anymore.
Who is going to look at - let alone attribute any sociological value to all this stuff we generate everyday? There's a giant digital "pump" spewing gigs and gigs of this junk every day.
The important things rise to the top and through achieving a place within popular culture, they achieve permanence.
Here's a theory - the lack of everlasting digital preservation will bring Western Earthlings' oral tradition out of atrophy. We may not have all the gems, but then, the longing for something can be potentially more energizing than the actual "thing" itself.
Listen to a romantic yarn about a freight train hopping adventure.
Ask your Grandfather about how much he loved your Grandmother when they first met (or vice versa).
Hours and hours of home movies are a drag - no matter what millennium.
When future civilizations unearth your casket, look in your inside coat pocket, and examine the little stack of Mini BluRay discs, what will matter is the whole timeline of it all, not just an isolated event. That good cell-cam pic of the chick with amazing gams - a blink of an eye. Your Mexico vacation where you discovered your latent cinematic genius - boiled down to "human away from home village".
Do what you can, spread it out, put the eggs in more than one basket, and take it with you when you go.
Mod Parent -1: Sales Pitch
I backup to DVD, but I burn at a slower rate than the maximum, usually 2.4X rather than 4X or 8X. Some studies I've read have correlated higher burn rates with less longevity. I also burn a second identical DVD every time and label them 'a' and 'b'. I then take one over to my parent's house so that my second set is off-site. Its not a perfect system, but I don't trust important content to a HD even for a day. And to me, pictures & video of my daughter's birthday are important. I do the same with the FLAC backups of my CDs. I have a plan to re-archive the content in 4-5 years with whatever the best technology seems to be at that time. I'll cross that bridge in 2010 or so. Cheers.
From the limited number of replies that I have read, I conclude that unless you are prepared to shell out in the near deka Kilobuck range, and beocme familiar with backup software, which in my experience truly sucks, there are no good solutions.
The paper printout seems the best - at 2200 dpi, you have ~ 1e8 bit per 8 by 11 sheet of paper. If you can get 8 bits per dot with color, then you have ( ithink) ~ 1e8 bytes
Here is the solution I have. On my server there are 6 250gig drive in a RAID 5 configuration, giving me just over 1 TB of redundant storage. On this partition are 4 folders, Waiting, Active, Finished, and Archived.
The waiting folder is were all new projects are ripped to, DV-stream, pictures, audio, basically anything unedited. The contents of this folder are also copied to a hard disk on my editing station. I don't always back up my waiting projects to the server as all I have done is rip the DV-stream, scanned images, or pulled them off CF.
The Active folder contains all of the projects that I am currently working on. These projects are on my working drive on the editing station and then when I am done with my editing session I copy and replace files on the server.
The finished folder is where I put all my projects that are completed but are not yet archived to disc. There is a copy of this folder on my desktop as well.
Archiving the all critical conclusion. I take the project folder and use winRAR to make 4gig archives, and for every 5-10 of those parts I have winRAR create a recovery segment, acts just like RAID for ever recovery file I have I can loose or corrupt one of the parts and it will recreate it. I then take those 4gig files and run them through WinRAR again, this time dividing them into 200meg chucks with two recovery files.
For example, I take a 60gig project and divide it into 15 chucks plus 2 recovery then divide those 17 chucks into the 20 200meg chunks plus 2 recovery ones. I burn these all to DVD (single layer) and now have each disc RAID 7 and then the set of disks RAID 7ed. So I can have two discs now blow up and loose have 2 files on each disc go bad and still have my data.
The archiving of the data to disc is simply a backup of the RAID 5 on the server. These disc should be kept somewhere other then where the server is. This is basically incase the server burns down. There is a way (lengthy) to get the data back.
This is alot of work, but it has worked good for me, and keeps my work organized and I know exactly the state of the work.
--
So who is hotter? Ali or Ali's Sister?
One thing I'd like to really stress: whatever system you choose, test it before you need it. I recently graduated from an art school and made sure that I had copies of all my course work (animation frames, composited video, etc) and class demos on rewriteable DVD. Six months later I found that at least five of my disks had become coasters; apparently the erasable CD labels I affixed caused the disk layers to separate over time. (And yes, the disks were stored in cool/dry places.)
In a separate incident, I bought some backup software and found that the instruction manual glossed over certain items. I did a test on a spare drive and found that one of the backup options made a snapshot of the data on the logical drive but also the partition information for the physical drive, instead of just the logical drive: any partition changes made after the image was taken (even if it did not affect the drive being mirrored) would be erased/undone if you tried to restore from that snapshot.
Many people have also noted that the original poster did not specify what kinds of things he wanted his backup/archive to protect against: Acts of God (fire, flood), Acts of Child (virus, "Daddy, what does this button do?"), or Acts of Accident ("rm -f...WAIT, NOT THAT FOLDER!!"). Obviously, each will require a different solution. I'm not sure if this is is practical, but: if you're trying to archive edited work, could you maybe put a secondary copy on (S)VHS? At least half the /. questions about long-term storage have a reply that points out that TV stations have tape libraries going back >>20 years, and as a last-last-chance backup, it might not be too bad as far as image quality goes. And you can always put more than one project on a tape, with 5-10 seconds of black in between. (What's the line? "Never underestimate the bandwidth of a station wagon full of quarter-inch tapes") I have no idea how much data is on a DV tape, so it might not be possible to do this...but what the heck, even dumb questions can lead to new ideas sometimes, right? ;-)
There have also been the suggestion to use some kind of parity-generating software; I think that's a good idea, so I'm parroting it. ("Arrk! Polly wants a cracker.")
. . . my knee-jerk reaction is to say:
We should have NOT allowed the industry to consolidate the way it did. Too many competitors bought eachother out, and now there's no incentive for competition, and the field is littered with crappy offerings. Just absolute shit compared to what my first employer was working on 12 years ago.
On the other hand, tape backup hardware has lagged cost/performance-wise, to dasd technology. I'd say, just shell out some bucks on some external firewire hard drives, and clone your disk out every weekend or so.
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
Copying the Bible on CD is "1/10th the effort" of hand-copying it?
Wow, are you using a laser pointer to write your CD-R's?
I am currently a system administrator at a photography studio. We go thru about 1.5terabytes every six months. Backing up is a very hard thing to do with that size of data I am constantly fighting with that myself (since we keep photos for seven years) and the only 2 things that I can think of is 1. Raid array 2. Tape drive The first of the two by far is the best
I've been watching Blu-Ray off & on for years. They've been available in Japan for months now, and this *IS* the intended purpose for this tech. The battle for the format is still raging (vs. HD-DVD) but I'd stick to Blu-Ray... I'm a bit of a capacity freak.
Just keep buying hard drives. They have almost everything else beat in every respect:
1) $180 gets you 320GB right now, and that will probably keep dropping.
2) The media has its own controller. There's no need to make a distinction between a tape drive and the tape media. So you won't run into compatibility problems later.
3) Seek times. Scanning a tape for the file you want can take a LONG time, compared to the 10ms or so you can get on a hard drive. And the through-put you can get now (on hard drives) is incredible.
4) If something happens to your "main" machine, you can immediately plug in the back-up hard drives into another machine, and all your data is immediately accessible. You don't have to worry about installing the right driver for the tape back-up, or finding the right back-up application, etc.
5) If you don't like handling individual hard drives, just put them in $30 FireWire/USB shells. These are cheap, and "reusable" in the sense that it's not too replace the drive that is in one (say, when 320GB is tiny and you want to put your new 4TB drive) with a new one.
6) It's really easy to set up back-up tasks that run every night (or whatever) that copy diffs to a hard drive. Extremely fast. When a back-up drive fills, unplug it, label it (or just edit a text file in its root directory), and plug in the next one.
Go with tapes. Now that LTO/Ultrium is out, DLT-IV is dirt cheap for both tapes and drives. You can fit up to 40GB compressed on a single cartridge, it's been tested to last for decades if kept well, and it can take a whole lot more abuse than other tape technologies.
Sounds like a vacuum food sealer would be perfect for that. You could remove all of the air and seal individual disks or stacks of them.
First of all, use retrospect, that way you don't have to worry about chaning files. Also, it will keep older "versions" of your files, so if you get some kind of data corruption you can go back to an earlier, clean, version.
Second, you need to decide how important this data is to you. I see 3 levels of security.
Level 1, you have one backup. This is not very secure because if the original gets corrupted as well as the backup, you have lost the data.
Level 2 you have 2 independent backups, or one backup that is redundant. This is pretty secure if you also keep the original, because you now have 3 copies of the data.
Level 3. You have 2 independent backups, plus the original, and one of the independent backups gets archived off state or off site, depending on your paranoia.
I personally feel that if my house burns down, family photos will be the least of my worries. So I go with level 2, and the easiest way to implement this is with a Raid. You want to use a parity raid that allows you to add disks, so as your backup grows you can add space. Raid Level 5 is ideal. If you lose a disk you can just replace it, so the backup is redundant. However, I would NOT use this without something like retrospect, because if you simply copy the files, overwriting previous versions, you will have data loss for sure. You need to keep the older versions.
For added security you can make copies of some files and store them off site. For example, if you have a buddy with another digital video camera, you can each store a copy of the others stuff. Just hook them together and copy the material from one tape onto another. Digital photos can be burned to DVD's.
I've been archiving my porn collection for years like this.
Thor, the Pizza Delivery God is a favorite of my kids, as is the Bang Barge Sagas.
- Organize your data into a reasonable heirarchy.
- Create metadata records descibing the content and the format it is stored into. (file type, compression format, internal organization, etc.)
- Set up an online/nearline/offline workflow. Work files, files needed often or used recently are stored locally on RAID-1 or better devices. Files that might be needed soon are stored to readily-accessible media like DVD-R and local tape. Archive data goes to multiple long-term tapes or optical media, stored alongside paper records containing metadata and placed offsite in secure, climate-controlled storage. (For home and small businesses, nearline=offline.)
- Test your backups for recovery at least once or twice every six months.
- Stick to your workflow.
- Whatever your chosen archive data format is, keep a known working drive of that format near or with the archived material. Even better, get a cheap/cheap/cheap PC minimally configured for the drive and the backup software, and keep that with the data.
- Evaluate your long term storage every ten years, and be prepared to spend the time and money to migrate your archived data to new storage formats and media.
Procedures, workflow, maintaining metadata and thinking long-term are all at least as important as the specific drive types you might select.-- Gary Goldberg KA3ZYW 301/249-6501 AIM:OgGreeb Digital Marketing Inc., Bowie, MD
You missed my point fairly entirely. I have a hard time believing you could have read my post and not figured that out, so perhaps you're trolling - but in case you've managed to confuse anybody...
_hard drives_ are a perfectly acceptable backup medium. I went into great detail about that.
RAID is NOT a backup medium. Backup to harddrive != RAID. !!! RAID is explicitly about consistency - so if you (or a hacker) delete a file from a functioning RAID it immediately gets deleted from everything. This is not a backup.
Also, the linux-HA guys say you should NOT buy the same kind of disks because it increases the chance they fail at the same time.
Finally, you don't need to diff the files you only need to gather the timestamps. (Your way is more space efficient, but I think the gains are marginal)
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
I'm ok with snapshotted volumes (on or off RAID) except for:
1. Single point of failure at the PS/MB/controller level.
2. If that machine is owned or the OS goes corrupt it may delete your snapshotted data also.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Lifespan is an important HD consideration. And furthermore, it's one that I forgot to mention. I have a fair number of 7 year old HDs running and at least 1 at 9. But I agree you can't count on that kind of lifespan.
At some intervals you should definitely add new backup servers with new harddrives and let them sync up. I really paranoid person might have 3 or so backup servers and might add a new one each year...
The great advantage of HD based backups is that adding an entirely new setup to the mix is very easy because you don't need to swap tapes to get _all_ of the data.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Lots of gmail accounts. Lots, lots of them.
Yeah, but in three weeks if your system dies, you can completely recover with RAID; with tape it can be done, but it takes much longer to restore, and check for data integrity. With RAID, the integrity is built in.
People trying to argue RAID isn't a backup medium obviously don't understand backup. RAID is just a means of getting two copies of the same data on different hard disks. *You* are the one responsible for making sure that data is safe past that (which goes to my suggestion of storing the set of parity drives in a firesafe location).
Quality hard drives are built to last. Especially if they sit around, not moving, in a static free environment. Sure, they have more failure points, but that's not what I'm arguing, and that's not what backing up to harddrive is for. And for the last time, RAID is a means to an end.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
Using an inexpensive case like this:
f or-multiple-drives-4x.html
http://www.cooldrives.com/usb-external-enclosure-
you could have 4 250GB+ drives in your backup system. Using the slide-out trays, it is easy to move one or more drives to an offsite location. Then you'd have to decide whether to do full backups, incrementals, or a combination of both. In any case, backups are fast and it's easy to move the drives around or to replace a dead drive.
Many people do not seem to understand about backup and its role in the original poster's dilemma. These data are photos and videos that are irreplaceable. Several horror stories have been posted about going back to online disk data years later and finding it unreadable. So the short answer is ALL rotating storage fails, period. Yes even ups backed up raid-N; Pick any N you like. If the data are the irreplaceable photos of your children then think: hurricane; flood; fire; lighting; nasty power failure; power brownout; virus; accidental corruption or destruction; memory failure; etc... The rule is: If the machine, the power company and the Net can see it can be hurt! Therefore the First Step in protecting data is get it copied, get it OFF-Line and then get it away from site. Getting it copied is the point of this thread and the OP is right: - DVD is not there yet though Sony's Blu-Ray looks promising. The Blu-Ray disc is encased to protect it from scratches and UV damage. - Removable and RAID disk at first blush looks good but they are very susceptible to physical damage. Ask yourself this if your backup disk fell out of your hand and hit the floor what would be your first thoughts? To test it to see if it had survived. Obviously not a good offsite backup candidate. - And then there's Tape. Which is the only ...
Well for the time being, yes. It is still the least expensive $/GB. Which makes it a little less painful to follow the Cardinal Rule of Backup:
Backup often and Test/Read/Verify every backup. That way you have snapshots of your data at various points in time. (This can be very important when data is being slowly corrupted by flakey hardware or malicious viruses).
As far as what to use for a home system, the Sony AIT1/2/3 units are some of the lowest cost and are the most reliable in the industry. (Real world data here; Period.)
For business LTO2/3 is the weapon of choice at the high end. Though Sony AIT3/4/S is an excellent choice in the mid to high price sensitive (SMB) market.
Try and avoid the temptation of going too cheap. After all we prefaced this thread on the fact that the data is priceless.
For example DAT drives are an industry joke. The manufacturers have gotten together several times trying to all agree to kill the damn thing. If it were not for its low cost to the system OEMs it would be long gone. Definitely not reliable/large/fast enough for backup! In fact DAT drives are the only drives worse in the field then the Exabyte drives.
And make sure to use a software package that uses a format that will be supported in the future. (think tar, dump, Amanda, windows backup, etc) As a rule of thumb no package with less than ten percent market share is likely to survive long term or be supported by their larger competitors.
The last part (of the First Step) is easy; your local Bank will rent you a safety deposit box. And really, you should have one anyway for the low tech irreplaceable items that you need to protect. Rent one and keep at least every other backup tape in the Bank.
The Second Step in protecting your data is keep a log of what is in the safety deposit box and what media/format it is in.
Why? A case in point. In recent years it has been virtually impossible to get certain Exabyte tape drives repaired. So much so that dealers were buying up all the used ones. Finally with no more availability, used ones were selling for more than new because new ones were in short supply as well (tape head problems) This happened repeatedly with more than one model of drive from them, over several years.
The point is if your backup data is on one of those Exabyte tapes and your tape drive fails, you cannot read your data. Nobody can fix it and you cannot get a replacement. So in reality you are just as bad off as you would be if you had NEVER BACKED UP. Exabyte is a good example here because they are slowly going out of business. When they do, in short order everyone who owns their drives is screwed when it fails. And ask anyone who knows; they always
Few people are professional photographers or videographers. This whole article is about an exception to your rule.
Under capitalism man exploits man. Under communism it's the other way around.
Any second-tier engineering school has enough information catalogged on paper to recreate most of the engineering accomplishments since the 18th century. There are thousands of these schools on this continent alone; we've come a long way from the days of the library of alexandria.
Digital media is much more robust than it's analog equivilant because it can be copied instantly. That's what makes your palm better than a black book; the sync cradle.
Regarding your observations on the petroleum age, you might be at least partially correct there - suburbs are not a sustainable model. There's enough coal to turn the sky black on the planet, so I wouldn't worry too much just yet.
More likely, you are overestimating the value of your experiences and contribution to the future. I'm pretty sure the 25th century isn't going to be too concerned with pictures of my car. Pr0n, otoh..
..don't panic
Your system sounds pretty good, I love rsync as well. But if you are dealing with very large amounts of data that 100M network is a bad bottleneck. So I would go with Gigabit, just so your backups don't start taking half a day. You also should have the fastest CPU you can get for the machine that is actualy running rsync.
Repost. Sorry this should be easier to read... Many people do not seem to understand about backup and its role in the original poster's dilemma. These data are photos and videos that are irreplaceable. Several horror stories have been posted about going back to online disk data years later and finding it unreadable. So the short answer is ALL rotating storage fails, period. Yes even ups backed up raid-N; Pick any N you like. If the data are the irreplaceable photos of your children then think: hurricane; flood; fire; lighting; nasty power failure; power brownout; virus; accidental corruption or destruction; memory failure; etc... The rule is: If the machine, the power company and the Net can see it can be hurt! Therefore the First Step in protecting data is get it copied, get it OFF-Line and then get it away from site. Getting it copied is the point of this thread and the OP is right: - DVD is not there yet though Sony's Blu-Ray looks promising. The Blu-Ray disc is encased to protect it from scratches and UV damage. - Removable and RAID disk at first blush looks good but they are very susceptible to physical damage. Ask yourself this if your backup disk fell out of your hand and hit the floor what would be your first thoughts? To test it to see if it had survived. Obviously not a good offsite backup candidate. - And then there's Tape. Which is the only ...
Well for the time being, yes. It is still the least expensive $/GB. Which makes it a little less painful to follow the Cardinal Rule of Backup:
Backup often and Test/Read/Verify every backup. That way you have snapshots of your data at various points in time. (This can be very important when data is being slowly corrupted by flakey hardware or malicious viruses).
As far as what to use for a home system, the Sony AIT1/2/3 units are some of the lowest cost and are the most reliable in the industry. (Real world data here; Period.)
For business LTO2/3 is the weapon of choice at the high end. Though Sony AIT3/4/S is an excellent choice in the mid to high price sensitive (SMB) market.
Try and avoid the temptation of going too cheap. After all we prefaced this thread on the fact that the data is priceless.
For example DAT drives are an industry joke. The manufacturers have gotten together several times trying to all agree to kill the damn thing. If it were not for its low cost to the system OEMs it would be long gone. Definitely not reliable/large/fast enough for backup! In fact DAT drives are the only drives worse in the field then the Exabyte drives.
And make sure to use a software package that uses a format that will be supported in the future. (think tar, dump, Amanda, windows backup, etc) As a rule of thumb no package with less than ten percent market share is likely to survive long term or be supported by their larger competitors.
The last part (of the First Step) is easy; your local Bank will rent you a safety deposit box. And really, you should have one anyway for the low tech irreplaceable items that you need to protect. Rent one and keep at least every other backup tape in the Bank.
The Second Step in protecting your data is keep a log of what is in the safety deposit box and what media/format it is in.
Why? A case in point. In recent years it has been virtually impossible to get certain Exabyte tape drives repaired. So much so that dealers were buying up all the used ones. Finally with no more availability, used ones were selling for more than new because new ones were in short supply as well (tape head problems) This happened repeatedly with more than one model of drive from them, over several years.
The point is if your backup data is on one of those Exabyte tapes and your tape drive fails, you cannot read your data. Nobody can fix it and you cannot get a replacement. So in reality you are just as bad off as you would be if you had NEVER BACKED UP. Exabyte is a good example here because they are slowly going out of business. When they do, in short order everyone who owns their drives is screwed when it fails. And ask anyone
1. As I just mentioned in response to another post, I very much encourage your backups to actually be on another machine - if your server is own3d, or your OS/RAM/MB freaks out you have no idea what it'll do to your backup drive. If you're posting this on /., I figure you can get a machine out of the garbage and put this together... Again, it's my opinion that having more than one copy of the data per computer is a waste of HD. (other than for high availability, a la RAID)
2. I'm all about using your friend's internet connection to do this. Furthermore, in response to someone else - if you think your friend is spilling Cheerios on it.. a) get better friends and b) get MORE friends/backups. I'll take redundancy over perfection any day.
3. RAID is great in those situations where your intra-backup loss (ie, from a day) is very great.
I agree with you that a lot of people recommend RAID 10, but I think they are quite wrong OR they are using crap systems - my lengthy explanation follows.
Good RAID controllers use battery-backed write cache - that means they "accept" your write immediately and use a battery to actually put it on the harddrive LATER, even if the power goes out. This is a HUGE speed improvement for multiple small write situations, even with just ONE disk. I ignore this effect in the below discussion.
I'm going to assume a system where you have two similar drives on different buses on the same machine. I'm also going to assume that you're HD I/O bound (ie, the harddrive platters/heads are what's causing the slowdown, not your CPU) I'm also going to assume you do more reading than writing - at least more files if not more bits (which is pretty typical)
--- First, why RAID 0 is stupid (unless you're using very large files AND not using them at the same time) I'm going to compare RAID0 to just putting different stuff on different drives (for instance, OS/swap/apps on drive 0 and data on drive 1.) I'm calling this setup "noRAID"
RAID 0 is straight striping - it writes half of every file to each disk. This means that the _write_ time (time from the time it starts to the time it finished writing) is twice as fast, but the _seek_ time (time to get the head to the right place to write) is exactly the same as a single disk. For writing very large files this is almost twice as fast. For writing smaller files it is not faster at all because the seek time (time to find where to write it) totally overwhelms the time to actually do the writing. For reading the same thing is true. The "bulk" of reading a file is exactly twice as fast but the seek is not changed at all. So most of the time it really isn't faster except for really big files.
The short answer is that RAID0 is stupid because it has no benefits when seeking.
Compare this to just using 2 drives: if you try to read or write simultaneous small files that are on different disks, noRAID is absolutely _twice as fast_ If you try to write a single very large file RAID0 approaches being twice as fast as the write time becomes much larger than the seek time. Of course, the weak point in this argument is that sometimes you want two things on the same disk - then noRAID is only the same speed as RAID0 for small files. So noRAID doesn't average being actually twice as fast.
In addition, RAID0 is half as redundant because either disk failing destroys everything.
--- Second, why RAID1 is good.
RAID 1 is straight mirroring. On a modern RAID system (like Linux's SoftRAID) this gives you performance that - compared to a single disk - is exactly identical on write to a single disk (Assuming your CPU can always keep up) For multiple file reading, though, it peforms better than any other setup, even _better than noRAID_ because it only needs to read from 1 disk and it reads from whichever disk has a head in a convenient spot to do THAT read.
It doesn't have the disadvantage of noRAID, because it ALWAYS has a copy of the data it needs on the
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
A lightning hit our house 2 weeks ago. All our network cards were rendered inoperative, along with one hub-switch and a router. My motherboard and/or my CPU were affected too and I've had to change them (did not yet test which was broken). Our 5 other computers were intact and, interestingly, no HD was damaged. No data loss at all.
You are more than the sum of what you consume. Desire is not an occupation.
Many people do not seem to understand about backup and its role in the original poster's dilemma.
...
These data are photos and videos that are irreplaceable. Several horror stories have been posted about going back to online disk data years later and finding it unreadable.
So the short answer is ALL rotating storage fails, period. Yes even ups backed up raid-N; Pick any N you like. If the data are the irreplaceable photos of your children then think: hurricane; flood; fire; lighting; nasty power failure; power brownout; virus; accidental corruption or destruction; memory failure; etc...
The rule is: If the machine, the power company and the Net can see it can be hurt!
Therefore the First Step in protecting data is get it copied, get it OFF-Line and then get it away from site.
Getting it copied is the point of this thread and the OP is right:
- DVD is not there yet though Sony's Blu-Ray looks promising. The Blu-Ray disc is encased to protect it from scratches and UV damage.
- Removable disk at first blush looks good but they are very susceptible to physical damage. Ask yourself this if your backup disk fell out of your hand and hit the floor what would be your first thoughts? To test it to see if it had survived. Obviously not a good offsite backup candidate.
- And then there's Tape. Which is the only
Well for the time being yes. It is still the least expensive $/GB. Which makes it a little less painful to follow the Cardinal Rule of Backup:
Backup often and Test/Read/Verify every backup. That way you have snapshots of your data at various points in time. (This can be very important when data is being slowly corrupted by flakey hardware or malicious viruses).
As far as what to use for a home system, the Sony AIT1/2/3 units are some of the lowest cost and are the most reliable in the industry. (Real world data here; Period.)
For business LTO2/3 is the weapon of choice at the high end. Though Sony AIT3/4/S is an excellent choice in the mid to high price sensitive (SMB) market.
Try and avoid the temptation of going too cheap. After all we prefaced this thread on the fact that the data is priceless.
For example DAT drives are an industry joke. The manufacturers have gotten together several times trying to all agree to kill the damn thing. If it were not for its low cost to the system OEMs it would be long gone. Definitely not reliable/large/fast enough for backup! In fact DAT drives are the only drives worse in the field then the Exabyte drives.
And make sure to use a software package that uses a format that will be supported in the future. (think tar, dump, Amanda, windows backup, etc) As a rule of thumb no package with less than ten percent market share is likely to survive long term or be supported by their larger competitors.
The last part (of the First Step) is easy; your local Bank will rent you a safety deposit box. And really, you should have one anyway for the low tech irreplaceable items that you need to protect. Rent one and keep at least every other backup tape in the Bank.
The Second Step in protecting your data is keep a log of what is in the safety deposit box and what media/format it is in.
Why? A case in point. In recent years it has been virtually impossible to get certain Exabyte tape drives repaired. So much so that dealers were buying up all the used ones. Finally with no more availability, used ones were selling for more than new because new ones were in short supply as well (tape head problems) This happened repeatedly with more than one model of drive from them, over several years.
The point is if your backup data is on one of those Exabyte tapes and your tape drive fails, you cannot read your data. Nobody can fix it and you cannot get a replacement. So in reality you are just as bad off as you would be if you had NEVER BACKED UP. Exabyte is a good example here because they are slowly going out of business. When they do, in short order
With a normal incremental backup I would agree with you. But the "IV" system I discussed doesn't need to be reconstructed like that - which is pretty much why I set it up like that.
1) It's already writing to a filesystem. So it is already "reconstructed" on a system level.
2) It's not saving the diffs of individual files, it's saving full backups of the individual files - if they actually change.
If you have big disks and small data, I wouldn't object to writing everything to 2 parts of the same disk in addition to everything else. But really I'd rather put that effort towards more backup machines.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Get yourself a digital interface to a VCR and make two copies to VHS. They hold a large amount of data and are cheap.
I didn't mean it quite as simplistically as "accidental deletion" I basically mean that there are too many points of failure that I don't like: users operating the machine. Applications writing to the file. bad RAM. OS. FS drivers. drive controllers. viruses or malicious hacking. If any of these things go wrong, your data is toast, and it's likely toast on ALL of your mirrors.
Normally your RAID array is a read-write server, so a virus on a _client_ machine can wipe out big (enough to be important) sections of data. And it's more vulnerable to hacking because it's providing "public" services (at least on your intranet)
So I'm going to give two examples of where I think you're reasonably right:
1) A very well-secured RAID fileserver that doesn't actually give client machines permission to change/delete files. Using snapshots is a reasonable example - but it isn't the RAID that makes it a backup, it's the snapshots. And some FS have snapshots without requiring RAID...
2) Increasing redundancy by adding 2 drives to a backup server that is already operating behind the kind of protections I discussed originally.
In both of these cases I still think it's better to use a different backup machine because it's more redundant at a pretty marginal cost. [If you scale this up enough it's at NO additional cost, because you "fill" every backup server HDs and with one copy of as much data as possible. If you don't scale it up then it's a couple of lowend machines.]
I've definitely heard reports of multiple same-type HDs going bad at the same time... which makes sense. If they're made at the same time and subjected to the same environment. Reducing the likelyhood of more than one of my drives failing at the same time seems like common sense to me.
timestamps are usually reliable with regard to whether the file has changed on the same machine - the chances of the clock being off such that the changed file has the identical timestamp are fairly low. But I certainly agree that also MD5 checking is even more safe.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Your insurance company didn't give you all the details. You _can_ get media rated fireproof safes that can protect tape/optical/etc, but they are specifically designed for that purpose.
The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.
/.
Why does RAID work? Because digital data can be copied at virtually no cost. Ergo, the survivability of digital data depends on it's redundancy, which can be increased any time that it can be read.
With Video:
Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables. With Audio:
sample rate, compression format, bit depth, file format, etc.
And all of those things can be reverse engineered, with or without ISO standard or patent documentation, original source code, or anything else that might be available. Any future historian who gives a damn about our pirated XviDs is going to have the patience and technology to discover how they can be decoded.
I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper,
Few? Huge numbers of books are pressed on acid-free paper. It's very common for the first edition of a book to be on acid-free, but not subsequent or paperback editions.
our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.
No photographs? No film? No microfilm? No analog audio or video tapes?
Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work.
You're making several flawed assumptions here:
1) that people of the future will have no way of generating large amounts of electricity. That they will have expended all fossil fuels, all nuclear fuels, and will not have any renewable electricity infrastructure.
2) that they will balk at the few hundred watts necessary to power an early 21st-century device
3) that they will care about early 21st-century devices at all when they don't have enough electricity to power a few light bulbs
4) that they will have any reason to use early 21st-century devices at all to access the data stored on their discs. (That is, why wouldn't they build their own?)
Archiving the 21st century will be impossible.
No more impossible than archiving any other century. That is to say, the survivability of any single bit of data is pretty grim, but given the massive quantity of stuff produced, a sufficient chunk of it will survive. Who cares if the 25th century doesn't have pictures of every 21st-century person's uneventful vacations?
To the 25th century, the 21st century will be seen as a dark age but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on
Who cares? We don't have everything ever written by everyone ever to have existed, and yet, we don't consider the entire history of civilization a dark age. Just because it could be archived doesn't mean it needs to be.
That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.
I hate to break it to you, but not every painting from 500 years ago has survived. There is nothing inherent in the medium itself that lends it ultimate survivability. Most that survive do so either by chance or because people actively wanted them to survive.
If you want anything you create today to survive 500 years, you just have to make sure that 500 years of your decendents (either family or worldly) will want to preserve it.
Don't become a regular here -- you will become retarded.
In considering whether you need a fireproof safe, and whether such a safe would be OK for media, it's helpful to go back to first principles:
If what you are trying to protect is truly a backup, and not an archive (that is, it is a spare copy, not the only copy), then it is not necessary to protect both the original and the backup from fire (or flood, etc.). It is necessary only to ensure that the hazard doesn't affect both at the same time.
If you want to store both in the same room, or even in the same house, then indeed some sort of fireproof safe would be needed. But it you can store one offsite, then no fireproofing is needed. There's defnitely no need for a safe-deposit box at a bank!
Once I simply rented an additional locker at my athletic club (in the hallway, where they were cheaper, not in the humid locker room). Another time I stored my home backups at the office, and vice versa.
At the offsite location, you may need theft protection. Hiding is the best way, but there are many safes that are very good at this, even if they're not fireproof.
Another reason for not using a safe-deposit box is that you want the storage location to be easily accessible so you will use it often. You don't want your backups sitting on the hall table for two weeks waiting for your next trip to the bank!
Archiving digital video and audio can be a daunting challenge. Many groups in the television broadcast industry have grappled with this issue for years. Some have settled on expensive spinning disk-based storage such as NAS and SAN solutions. Others have found value in tape-based robotics libraries for massive archival arrays. Recent advances in optical storage technologies have led to the development of Sony's Professional Disk for Data. The optical disks used in this technology have the same physical dimensions of a typical CD or DVD disk, but can store 23.3GB (about 2 hours at DV-25) for 50 years or more. Sony markets these disks in a cartridge-based format (primarily because these new disks don't yet have a protective coating like CD's and DVD's, making them more susceptible to damage from particulate matter). Bare media is available for use in specialty applications such as Asaca's AM750PD robotics library. For the typical user, I would recommend cartridge-based media for practical day-to-day use. More information is available at: http://www.sony.net/Products/MO-Drive/ProDATA/
Can't stress this enough - redundancy. Having a single backup is having a single failure point. Not good.
Also, without sufficient research I'd suggest not trusting *anything* for more than a few years. Optical media, magnetic media, etc, all needs to be refreshed every once in a while. Continual rotation onto the newest medium ensures that you'll have a working copy available when needed. And that the medium will be compatible with current technology (try listening to 8-tracks lately?).
The simplest DV storage medium is the dv tape itself. Mini-dv tapes are fairly cheap in bulk. Having a 200GB external drive to make a direct copy is also a good option.
Finally, don't bother with DL DVD medium. Buy a boatload of DVD blanks, and use proper backup software (Retrospect comes to mind). The backup software makes a spanned backup set on your DVDs, so it doesn't matter if the individual files fit on a disc or not.
[Serious mode on] I once heard in a conference that the most reliable filesystem ever was the ed2k network. [Serious mode off]
So, here goes a suggestion:
1) Zip/Rar it with a password.
2) Divide it into downloadable chunks.
3) Rename it to "Lesbian chicks hot stuff!!!!", or similar, you get the idea...
Et voilà.
Has anyone ever considered setting up several high-latency, error-correcting communications links to keep the bits in motion? Just pull the bits off when you need them.
Don't fool yourself. All of those books that were "painstakingly" copied were not really done like you said. The Bible itself has at least 10 different translations....probably more. In english there are:
NIV - New International Version
ASV - American Standard Version
KJV - King James Version
Those are just the ones I remember and there are definitely more. Plus there's things that get lost in the translation from Hebrew to English as well as from Aramaic. My point is the Bible of today had been changed over time to suit the differen Christian sects. There's no Bible I can point to and with any sort of confidence say that it is how the original writers wrote it. Heck, many of the scriptures in the Bible were wrote to stand alone and not to be incorporated in any book.
Back to the subject at hand...in order to make sure your digital photos outlive you, the thing that must be done is copying. Digital mode is MADE for copying and distributing data. You can easily move data to a CD-ROM, a DVD and whatever other technlogy there is. You need to move it fro one data media to another....from cd's to a dvd to a backup tape to whatever other technology there is. I think what frustrates most home users is there's no real economical way to back these up at home. The size of our hard disks has gone up and up yet the size of CHEAP back up has not. The best we can likely do is dual layer DVD and you even need multiples of those to backup 200+ Gig of data. The best option now is another hard disk and even that is suspect.
Gorkman
I'd mod you down for being unable to figure out how to spell "permanent".
I see it as just the opposite. Those who argue RAID is a backup medium don't really understand backup. Backup intrinsically involves making one or more complete separate copies and storing them in a safe location.
RAID 5 is used to reduce the probability of needing to restore your data from a backup. It does not remove the need of doing a backup and it is not the backup method itself.
Single point of failure at the PS/MB/controller level.
But how often does a failure of one of those components actually destroy data? It does happen, but it's extremely rare. It's more likely that your house will burn down.
Oh, and you can minimize the risk of controller failure (with PATA, anyway) by putting all of the drives on separate controllers. Another advantage to software RAID over hardware RAID.
If that machine is owned or the OS goes corrupt it may delete your snapshotted data also.
Even in the case where the machine is owned, it's fairly unlikely that the attacker is going to bother removing snapshot volumes. Actually, it's pretty rare that an attacker bothers deleting data at all. In addition, this is a risk that's fairly easy to defend against, unless there is some specific reason that a particularly competent attacker is highly interested in your specific box.
OS corruption could screw you. But unless you're mucking around with experimental kernels, the odds that Linux is going to lose your data is vanishingly unlikely. The same holds for Windows and the BSDs, actually.
In short, all of the risks that on-site backups defend against but snapshots don't are extremely unlikely. Given how much easier snapshots are to make and manage, this tells me that for most people snapshots are a better choice. "Real" backups may be better, but they're difficult enough that they tend not to happen. An easy solution that is 99.9% as reliable is likely to be *better* once you include the human factor.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Sure they're prohibitively expensive but if you truly want a high capacity permanent archival storage medium, DVD glass masters stored in their special cases can survive extreme light, heat, and evil spirits: http://www.disctronics.co.uk/technology/manuf/rep_ master.htm#Glass%20Master%20Preparation
You can now print your RAW files directly to negatives these days. And you know how good negatives are for longitivity and reliablity.
Analogue (will always be) > Digital.
Yeah, but in three weeks if your system dies, you can completely recover with RAID
Not always, but yeah, most of the time. RAID-1 should be safe, but with any of the others you're looking at varying degrees of trade-off.
Quality hard drives are built to last. Especially if they sit around, not moving, in a static free environment.
What do you mean by "sit around, not moving"? A hard drive left sitting on a shelf will eventually sieze up. I'm not saying they don't fail while running, but then you'll at least have the opportunity to know it's happening and take appropriate action.
Realisticly, it's startup and shutdown that cause the most stress on the drive. A drive that's kept spinning will last a remarkably long time; even beyond the point where, if it stops, you won't be able to get it going again.
And yeah, most new SCSI drives are rated for an MTBF of 1.2 million hours (about 137 years, which makes me question their methodology).
FWIW, I used to stress test hard drives for a living.
I do think hard drives are a good option for this purpose. Tape is just too expensive for most people, and being a magnetic media also I don't believe it has any significant advantages over a hard drive for long term storage. And what if your tape drive fails? Will you be able to find another one? I think it'll be quite a while before working IDE controllers are hard to find. Tape mechanisms may be more reliable individually (I don't know), but hard drives have a massive installed base advantage.
BTW, if you plan on keeping a hard drive on a shelf, keep in mind that they're designed to sit "on edge". Any place that buys drives in bulk should have some nice packing material to hold them safely in the proper position, or you can make your own without too much trouble. I do recommend a static bag, though, unless you're absolutely sure you're dealing with anti-static materials.
Under capitalism man exploits man. Under communism it's the other way around.
Sure they're prohibitively expensive but if you truly want a high capacity permanent archival storage medium, DVD glass masters stored in their special cases can survive extreme light, heat, and evil spirits: http://www.disctronics.co.uk/technology/manuf/rep_ master.htm#Glass%20Master%20Preparation
The safe deposit boxes should be in the vault, which sure as HELL had better be fireproof (vs the rest of the building).
And with the two-key systems for opening each individual box (the bank can't open the box without your key), the boxes would have to be forced, which takes time, which is something a bank robber doesn't have.
I'd say the local bank is safer than my firesafe in my house, rated for paper for 2 hours. I definitely need something better (both larger rated for better insulation (negatives/optical media), and longer burn times (wooden structure in a rural setting).
I think you mean "media safe"; these are fire safes designed to protect digital media, such as tapes, from being affected by the heat of the fire. Regular fire safes only keep the contents below the flash point of paper, which means they can still get hot enough inside to ruin or scramble magnetic or optical media.
Media safes are significantly more expensive than the equivalent-volume fire safe.
If you could read, you'd understand what I said. Pictures from workstation replicated to NAS. Backup NAS to DVD/Tape. If you want geo-backup, but NAS in another place, replicate NAS devices. Of course their is software. I thought you might be smart enough to understand that replication would need software. There's tons of it out there. Take your pick.
I have a SCSI HD I used to use 10 years ago that still works OK when I plug it in. The HD's I use for storage I do not have plugged in every day. If I were to be using even DVD's for backup I would have a few hundred to concern myself with, way to many to keep shuffling. At least when it comes time to backup a HD it can all be done in one operation.
I am lookign forward to Blu-Ray computer drives fro backup, I think that's large enough it makes for a pratcial alternative to hard drives.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Tape, tape, tape, tape...
My fault, I wasn't clear. I agree with your reply completely. Let me rephrase:
I'm OK with snapshots. (snapshots are != RAID, of course)
I have the following tiny issues...
(rest of post)
In short, I think that snapshots are basically fine but personally I'd rather have my backup on a machine that wasn't a server in the "accepts incoming filesharing connections" sense.
I welcome the day when my mainstream Linux distribution comes with "install snapshot filesystem over RAID1" is a standard simple installation option.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
I hadn't heard this before - which edge are they supposed to be stored on?
Are they better to _run_ "on edge" or flat?
If different, does this mean I should turn my machines sideways when I turn them off?
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
I like it so far. We've got a bunch of multi-server installs (8 total, scattered around the country in various datacenters), some with a few terabytes of disk staging, some without, some with multiple media servers, some with only one server that does everything. It all depends on the requirements at that site.
:)
:)
We also do the offsite backup-via-offsite server. Handy as heck.
Right now I think we're doing something on the order of 25TB of data backups a night across all the datacenters. Netbackup handles the scheduling well, but we do also have an eight person team that is our enterprise backup team, and they've put lots of work into taming the beast.
I still get between 20 and 200 backup related tickets a night, but when you compare that to the tens of thousands of jobs being done, it's practically set-and-forget.
You'll have to pardon me for the vagueness in the posting, people would be annoyed if I named things specifically.
I'm OK with snapshots. (snapshots are != RAID, of course)
But without RAID, obviously, you need some real backups, because your data is vulnerable to a single disk failure.
I'm sure you understand that. I'm just clarifying for anyone else: RAID+snapshots is nearly as good as backups. Either one alone is useful, but inadequate.
I welcome the day when my mainstream Linux distribution comes with "install snapshot filesystem over RAID1" is a standard simple installation option.
The new Debian installer (used also by Ubuntu) comes close. You can do the initial install with RAID and LVM, though you do have to walk through the steps of defining the partitions, selecting them for RAID and then creating a volume group and logical volume.
Okay, so maybe that's not so close :-)
Also missing is a package with appropriate cron jobs for creating and managing the snapshots.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
Shouldn't you be comparing the price of the tapes not the drives? How much does a terabyte of tape and a drive compare to a terabyte of hard disks ?
Photographers need a storage medium that is good for write once read occasionally once a year or so that is cheap and reliable. What's the aphorism? cheap, fast, reliable? When you generate a few gig of data a day (on average, a few gig an hour isn't trying very hard), you can do without fast.
If you read the original post, it sounds like what he wants IS an archive not a backup.
I am surprise that no one mentioned microfiche.
Advantages:
- Lasts for thousands of years
- Pretty much fire proof
- Nuclear proof ( unless they are in an area that gets over a few thousand degrees)
- Resistant to scratches
- Always readable
- Still very common
- Analog medium, which may mean a lot better resolution
- Cheap. Compared with DVDs, it is still dirt cheap.
Disadvantages:
- Analog medium, which sometimes means worse resolution
- As a small customer, you will need to get someone else to do it.
- Restoring images takes time, and unless you buy a lot of equipment, involves physical access to the backup copies.
However, machines are getting cheap as companies are trying to get rid of most (but not all!) of there microfiche machines.
I'm a little late on this one, but I worked for a photography studio for a while and we were all digital. Our best way of backing up photos was to make use of external harddrives and keep a pretty basic filing system for customers pictures. It works very well for pictures, and it's not too expensive in the long run. Never worked in video. No idea about the needs there.
You'll have to pardon me for the vagueness in the posting, people would be annoyed if I named things specifically. :)
:)
:(
No Problem... 25TB... Thats cool.. Ours is only for a medium sized school district... Not too much to really back up, about 15 Novell Servers... User & Apps Data mostly...
but we do also have an eight person team that is our enterprise backup team, and they've put lots of work into taming the beast.
I wish... 4 people for entire district... (15+ Buildings!) - "Enterprise Backup Team" is just one of my MANY hats that I wear on day-in/out basis...
BTW, No... WE (our IT staff) are 12 month employees, not like teachers... No summers off..
Sorry but this sort of loss has already happened.
About 20 years (ie. in 1986), the BBC ran a country-wide project to create a modern-day Domesday Book. School kids from all over the country gathered data and entered it using the then available BBC micro-computer that was standard in schools.
The BBC then consolidated all the data and cut a few CD's for each school (in non-ISO9660 format - but playable on the BBC micro)
Guess what? Although many of the CD's themselves have survived the format didn't, and there isn't
even a single functioning BBC micro around anymore. There are a few simulators, but simulators can't read non-standard discs.
Every single one of those CD's is completely unreadable as *no* working hardware has survived that understands them.
That's how (im)permanent digital data is - it's not permanent at all.
Far from being alarmist, the parent is almost certainly right.
And yes it is something to worry about. Take a small thing - propertly ownership. How do we know who owns what?
Through paper records that go back (in some cases) hundreds of years and through dozens of generations.
If this stuff gets computerized and is lost or damaged, our property rights are affected.
Alarmist? No, it has already happened. Several years ago the State of Victoria in Australia digitised their land titles - using Black And White digitization.
Many of those paper titles had colored notations and areas (which were then referred to in the text as deliniating someone or anothers rights)
Since this happened (the paper copies have been destroyed), there have been several court cases which hinged somewhat on the judges interpretation of which shade of grey corresponds to "red" and which to "yellow"
I bet there's at least one case where the judgement went the "wrong" way and someone has lost something that would've been clearly theirs if the paper copy was still available.
Imagine that in 10 years time, you have 50+ tapes - how are you going to realistically use this 'archive'? How will you browse it? How will you search it? What happend when you switched OS - did your backup sw vendor move with you or did you find an alternative - can you now read the oldest tapes? Do you still have the index without first loading the tape?
Sounds more like we need a personal document management system or personal content management system or similar.
I am going to buy an external HD nad put it in a fire safe!!!
Think about it!
Just need to drill a hole for the cable.
Use one of those bus-powered jammies and you're all set.
Now, what to fill the cable hole with so the fire safe stays fire proof?
Hmmm...
"I'm sure you understand that. I'm just clarifying for anyone else: RAID+snapshots is nearly as good as backups. Either one alone is useful, but inadequate."
Exactly.
SuSE (novel) has had a RAID installer like you describe - a very nice one, actually, for at least years; I believe I first used SuSE in 7.3 and it was there.
I believe that some of the newest FS versions automatically support snapshots in the FS driver. So theoretically installing RAID under a FS with the snapshots on should do what we want. But I still definitely want the installer to go a step further and say.
"YO! this is a big pretty button that's one step to reasonable data safety. Buy two harddrives, then click here! "
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Dude, I'll backup your data. I have a stack of floppies here and nothing but time.
The real silver bullet to good programs is caffeine; lots and lots of caffeine! *twitch, twitch*
You should check out the rsnapshot utility. It stores multiple snapshots of your data based on times that you specify (I store a week's worth of daily snapshots, a month's worth of weekly snapshots, and then 6 months of monthly snapshots). But it stores using hardlinks, so if a file doesn't have any changes, it doesn't take up any more space in the other snapshots.
Also, it is automated, so you don't have to worry about LVM silently discarding your changes when your snapshot fills up.
For offsite backups, you can do remore rsnapshot, but what I do is every so often tar up my snapshot directory and pipe the tar through gpg which encrypts and compresses very well. Make sure to use gpg -c unless you also have your gpg key backed up somewhere! Anyhow, I burn the resultant encrypted file to CD. (You can use DVD if you have more data... or multiple DVDs).
Oh, rsnapshot is free/open source, of course. :-)
"Avoid employing unlucky people - throw half of the pile of CVs in the bin without reading them." -- David Brent
Just wanted to provide a link to another great tool.
"Avoid employing unlucky people - throw half of the pile of CVs in the bin without reading them." -- David Brent
What you need to do is have every user on Slashdot to donate their 3.5" floppies and back up everything to the floppies...
Monday laughs...
200GB is not good enough. Get a seperate server that you can set up a RAID with some hotswap capabilities. Then in the server get a backup tape or external SCSI tape device to handle daily incremental and weekly full backups. This will cost you about $9K for a minimum setup using name brand hardware and software.
Not that I recommend Dell, but it is easy to configure and price a system like a poweredge 1800 with RAID 5, and 5 146GB drives with 2GB of RAM will give you a very good redundant system and then add a PowerVault 110 with 400/800GB tape backup.
This will give you almost 600GB of reliable data storage and then the tape backup will keep it safe. Last thing you need is a Media Safe that will be fireproof. Don't look at the cheap models at Office Max because those are just document safes and will not prevent your media from melting.
Using this as a guide you can start to price things out with other vendors, but you better be prepared so spend some bucks for your professional setup
Like those before me, long-term archives/backups to tape, the working stuff in a RAID array.
Just a bit of advice, watch out who you buy your HDDs from. I know that both Maxtor and WD have lowered their warranties to one year, and from what I've seen, they tend to crap out rather easily. My reccomendation if this data is important to you and you are going to be storing some on a HDD based system would be to spend the extra money and get a quality HDD by a reputable manufacturer. I'd personally reccomend Seagate (the warranties are for 6 years, and the HDDs last forever).
If you're looking for the ultimate in storage quality, then you could go SCSI, but in reality, you could just throw together a decent system with a 3Ware RAID card and some 250/300 GB HDDs (there are some places you can get a 250GB drive for $125.00 or $0.50/GB).
I'd go for filling the safe w/ nitrogen.... pretty cheap, and you'd be able to displace all the oxygen. Just make sure you open it in a well ventilated area!
So, I bought into the idea that I wanted to consider setting up snapshots on a new server we're about to setup.
.1n (where n is the "number of nights of backups) multiplier depending on the situation, or possibly much less.
.1n or so for each cycle.
But then I realized that my idea of a snapshot and the practice are very different - I think I heard "versioning filesystem" when you said "snapshot"
First, for anyone not following my myriad posts on this thread, I like something I'm calling an "Incremental Versioning" or IV backup - it's an additive incremental backup leaving both versions available in the case of any change. To me this is key functionality, because I want to be able to run a small backup often and catch small file changes without using enough disk space for a full backup.
So, I like the idea of a versioning filesystem over RAID. Ideally the filesystem would manage a COW (copy on write) layer, so that it would keep both versions of changed files but only need one copy of unchanged files. So, used with RAID, you get the complete versioning of backups at a perhaps 1 +
The Wayback filesystem does this, but it seems a little fringe for the filesystem in a production server.
Alternatively you can setup a cron job to rsync the data to a different, protected directory. Full backups use 1 + 1n space. But a full backup along with IV backups uses 1 + 1 +
So, based on a little googling, I'm guessing that by snapshots you meant LVM snapshots. LVM does not seem to support versioning, and only seems to support "full backup" snapshots.
I understand that in cases of very high disk usage locking the drive momentary to create a snapshot may be valuable. But I see no other general advantages to using snapshotting over rsync on the same machine for the purposes of backup.
I think that if I were to use LVM snapshots I'd feel compelled to do rsync IV updates more frequently anyway, so I can fit more versions in less space.
Incidentally, this seems to justify my original idea that RAID as backup is bad, because this only works by giving up more than 50% of your available space without any disk-level redundancy.
So in the fringe case where you have two big disks and you only need less than 1/4 the space you have, that's great.
But given at least 3 disks and needing 1/4 the space you have I'd MUCH rather do a round robin backup and have no more than 1 complete copy per HD, I think having multiple _full_ copies on the same drive of most of your data is generally wasteful, they should be spread around.
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
... and then one day, when IBM knew the RAID card was dodgy, they added another hard drive and the whole lot went *poof*
Now comes the fun part.
It turned out that the last backup done by IBM was more than 6 months ago. Since then they had only done 'inremental backups' (only backup changes to files) and it 'was time to do another backup' a couple of days before they added the new drive.
We are talking here about a Gov dept with more than 5000 employees, each with a virtual drive, with 3 servers for these virtual drives, and also rhe 'group drives' for the Central Office.
Lovely.
It took them more than 3 months to 'restore' what they could. Even then not everything we had was restored. They found that there was 'a fault' in the partial archiving system they used (sorry, I don't know which product they use to archive/restore) and in the end they were restoring by hand (again, I don't know what they did only that it was extremely time consuming).
ALWAYS create a full backup before making changes to SAN box. Always.
My first option would be to get a spare 250GB HDD and put a usb 2.0 cover to it. I would also make a general backup of the entire 200GB to DVD's. Sure it takes some time but the files are worth it. This already should have quite a bit of redundancy. Redundancy is the name of the game with backup. Now any optical media is good for at least 10 years, but you can buy specialized media that has a 100 year life span. So I really am confused at why optical media is considered inferior to magnetic tape media. Magnetic tape media is much more delicate and prone to data loss than optical media. Also with optical media even if it is not readable due to deterioration, it can always be read with specialized more sensitive equipment. The future of optical media looks very bright. Blu Ray players are already available. They can handle around 30gb now and should be able to handle up to 60gb quite soon. So 3 discs should easily cover all your present needs.
.... would become like glue.
Or would dirp.
Or who knows what. The manufacturers are telling how reliable the media is by means of their guarantees.
IANAL but write like a drunk one.
Honestly, that will take all the anguish of your digital life.
One can snap easily 200 pictures in a weekend break, plus nonsense in the way to work etc.
Whay wold you want to backup all that rubbish?
DO an exercise of reduction: choose 10 pictures a months and back those up. THat is plenty.
Any people really interested in your pictures will have plenty of material with that.
As for the rubbish one produces make them public: that may be the best antidote against oblivion: send as many CDs/DVDs of your holidays and memorable occasions to friends and family, publish them on the internet, etc. THat way you harness the popwer of people that may be interested in keeping copies of your pictures.
Otherwise, let them die, you will not miss them (like if you could find them: cataloguing personal collections is becoming tha nightmare of every household with a digital camera).
IANAL but write like a drunk one.
I can get Hard drives now from http://www.pricewatch.com/ for $0.38 Cents a gig.
I just get Regular Parallel ATA drives, or put them in a USB drive case.
I just bought 4 x 160Gb Hard Drives at $61 each! for this very purpose.
Backup to the drive then unplug and put pack in the Anti Static bags. I don't bother bolting the drive down, I just leave the computer open with the drive dangling out.
The reason this is better any other solution I have tried, is because I have data dating back to 1976 on media, and after going through 7 or more itteration where the old tape drive hardware is lost or stops working and is no longer available.
Or the tapes get gummy and sticky.
It's a nightmare. I spent almost a month backing out 9 track tape at differnet BPI's and 1/2 and 1/4 tape formats, 4mm Dat drives, 8mm, etc.
So far the best media has been CD, and DVD-Roms, although today they are too small to be useful.
The new Optical disks are not a popular standard like CD ensuring you will be able to read the data back at some point in the future.
Next has been the SCSI and IDE/ATA drives. These have been excelllent in the respect of backward compatablity and being able to read data from computers I had from 15 or more years ago. Heck I have even managed in fire up old SCSI drives from 20 years ago and read the data off it reciently. A 10 Mb SCSI drive I remember thinking $60 a meg was a great deal.
The next largest issue is, will the drive even still spin up.
I recomend keeping multiple copys for that. Luckily my constant upgraging and copy the data to the new drive solved of that.
There are also data recovery services that are excellent with dead drives at about $500 to $1500.
Somehow I have managed to keep almost every bit since 1980 still in my possession. And readily accessable.
Heck I can even run some of my old TRS80 Model1 code from 1979 and C64 stuff on emulators.
I am always doing that which I can not do, in order that I may learn how to do it. - Pablo Picasso
If you read the fine print in your normal fire safe documentation, most will expressly say that they are not intended for protecting computer media. What you want is a media rated fire safe.