Slashdot Mirror
Best Way to Back Up Photos and Video?
jsalbre writes "I do a lot of digital video work, and my wife is a professional photographer. With raw DV from the video camera using up 11GB/hr, and raw images from the digital SLR using 7MB I'm quickly using up a lot of space. I currently back up all my important files each night from one harddrive to another, but I now have over 200GB of irreplaceable data (more than just DV and photos, but those make up the largest chunk) and I'm having to exclude the "less important" irreplaceable files as my backups have started failing. Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive, but with recent studies showing that optical media doesn't last very long I don't want to come back in a few years and find that all my backups are useless. Not to mention that some of my DV files are larger than even a dual-layer DVD, and it would be near impossible to automate backup to DVD. How do other Slashdotters back up their important data? I'd appreciate distinction between methods for frequently accessed files and for infrequently accessed files. Any suggestions will be highly appreciated!"
Why not make two optical backups. Store at least one in a fireproof safe. For the massive files, you might have to invest in one or two hot swappable drives you can use as 'tapes', storing one in your safe. Mirroring might help.
File -> Print
i think you might want to take a look at tape drives
Would tape backup work for you? It's archival quality, but you get what you pay for...
And supplement that with LaCie external firewire drives.
If you don't know what AltaVista is (was), get off my lawn.
Well, it's expensive, but maybe you can find a deal on an LTO2 or SAIT tape drive on Ebay. These babies boast 200GB and 500GB of native storage respectively . The transfer rates are nothing to sneeze at either.
And as long as you store the tape properly, it should last a long time.
Have you considered compressing your video using one of the many codecs available? DivX is quite popular, and RMVB offers some of the best quality:size ratio I've seen. I understand how nice it is to be able to store raw mpeg for later use but is it really necessary for your purposes?
I typically take the edited footage and back it up to another miniDV tape from the computer (using my miniDV camcorder). I then lock the tape to prevent accidental erasure and store the tapes offsite. For photos, I'm taking my chances and burning them off to DVD. I also periodically make digital prints and send them to my parents and sister, who live in two separate locations. Worst comes to worst, at least they have a hard copy available should I lose the original digital version that I have on my computer.
The PC Weenies: 11 Years of Online Tech 'Too
(i have no experience in the matter) you should place all the un-changing files on a hard-drive which will then sit in a draw and is only plugged in when required. I have been lead to believe that this will reduce the likelyhood of harddrive failure to close to 0. Then you can setup a RAID type setup for you changing files.
Let's face it, one method won't fit all, so I hope your search proves fruitful. That said, here's what I do.
I have a 'cheap' system (sub 500) that acts as my data server. It houses 3 DVDrom drives, and a DVDRW drive, as well 1 200 GB drive. (the processor speed and ram really aren't too important, but for curiousity, it's an athlon 2000+ with 512 meg of ram). It runs gentoo, and I essentially pull the files to burn to DVD over the network weekly, and I keep the stuff I don't access alot on DVD, and the stuff I do access alot on HD -- but I primarily use the HD for holding images waiting to be burned.
War isn't about who's right. It's about who's left.
To a hard disk in a USB enclosure. Better yet, but more expensive, to a NAS box.
I concurr. For digital media, I would definitely do two of at least two different back up strategies.
First that comes to mind is Tape backup. They store huge about of data, and are very cheap these days, and have been proven to last for a while. Keep a good backup schedule, and keep one copy of the tapes offsite.
Secondly, I'd do optical. Optical's cheaper, but it's also not as long lasting, and takes longer to make the actual back up.
Thirdly, I'd do RAID. Mirror all the files onto a second set of hard drives. If you really want to get paranoid, mirror onto two sets of drives, and once a week swap out a copy of mirrored drives from a fireproof location.
If your data is truely irreplacable, then this is a good regiment. But it's also very expensive.. so you'll have to make up your mind.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
Not from TFA, from TF synopsis:
"Several people have suggested backing up vital unchanging files to DVD (video, images,) and continue backing up frequently accessed files to harddrive"
They've already considered hard drives. Since he's dismissed hard drives and seemingly all forms of optical media, the only thing that I can think of for this article getting posted is that the submitter *really* wants Slashdot to tell him that "Yes, it's ok to mortgage the house to buy that new Network Appliance SAN you've been drooling over."
"How do other Slashdotters back up their important data?"
I memorize it.
Vincent J. Murphy
Spandex Justice
First, there's the need to keep things around long-term. Second, there's the need to have things protected from disaster in the short term.
I once used an external firewire HD for backup, and was reminded of the importance of burning things as well when that HD went tango-uniform on me, destroying months of work.
I'd suggest looking into some sort of RAID - even just a simple mirror - for the short-term protection. That way you don't have quite as much a single point of failure that can wipe out your data, so you can do backups more because you need the space than because you need to sleep well at night.
As for the backups, optical discs are very convenient, but magnetic tape might have a longer lifetime depending on environmental conditions, and although I've seen CD-R comparisons, I've yet to see something similar for DVDs.
There are times where a high-capacity removable hard disk looks very attractive. Shades of the old Bernoulli's or whatever.
(This may not be first post, though there were none when I started. Maybe I'll have to settle for first useful post.)
Village idiot in some extremely smart villages.
Tell that to my bosses.
Except for one server which we don't even own, all of our servers (about 10) are RAID "backed up".
One of these days is gonna be really fun.
If you don't know what AltaVista is (was), get off my lawn.
One thing good about paper & film is they withstand decades of storage vs. years of normal magnetic storage. Photos and films from the late 1800's/early 1900's are still around whereas you're really gambling with current storage media.
Have you hugged your penguin today?
(other than the continual confusion of "backup" and "archive") is that the same people who talk about how unreliable CDR/DVDR discs are for longterm archival purposes seem to be the same ones who advocate buying a portable firewire drive for every project and putting it on a shelf until the client calls with changes.
Something about that seems horribly backward.
That said, Exabyte still rocks my socks
Raid is a hardware methodology to increase reliability of disk based storage systems. Backup is an archival strategy to recover data lost for many reasons including inadvertant deletion or modification. rm * or del *.* or delete from table or a fire at the site all will mean your raid system now has faithfully lost all your "backed up" data. Make copies to external media stored off-site and locally so that any catastrophe that occurs will not destroy all copies. Tape is still cheapest for archival.
When I was young, I had to rub sticks together to compute.
Rename all of the files so they have filenames like "Teen_Lesbian_fff_Hot!Hot!Hot!.avi". Now make them available through your favorite p2p service. Even better, prepend these files with short snippets of pr0n. You'll find that years later you can kick up just about any p2p client and you'll find your files are still available.
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
I'm amazed that someone has yet to come up with a combination of archival-grade photographic film or paper for storage and an optical 'reader' for truly long-term archiving...
Wouldn't it be ironic if paper backups were to become the way of the future.
A friend of mine recently started a small business to address exactly this need. His product is a Linux based RAID box that plugs in to a home network, and supplies reliable storage via samba.
http://www.permastor-us.com/
I use RAID to defend against hardware failures trashing my data, and I use logical volume management snapshots to protect against most user errors.
Neither is perfect. Some hardware failure modes could theoretically kill two or more of my four hard drives at once, which would destroy my data. Large power surges are the most likely danger, so I use a high-quality surge protector. I consider the remaining dangers unlikely enough to accept the risk.
Snapshots are also imperfect. When you create a Linux LVM snapshot volume, you have to specify how much storage is allocated to it. If changes on the source volume exceed that snapshot capacity, the snapshot stops storing the deltas and the snapshot becomes effectively useless. However, the most likely way that I might screw up and trash my data is by deleting large numbers of files. Since deleting files only updates the blocks that store the directory and inode data, not the contents of the files, a relatively small snapshot partition would hold the changes from deletion of all the files on the source. Now, if I were to accidentally run "shred" on bunches of files... I'd be screwed. I choose to accept that risk, too.
Although the RAID+LVM combo doesn't do quite as good a job as "real" backups, its failings are pretty minor, and unlikely, and it's advantage is huge: I don't have to think about it. I don't have to mess with lots of removable media and I don't have to remember to do backups.
The one thing I still worry about is some sort of catastrophe that destroys my whole system. Suppose my house burns down, for example. I'd lose it all. So I still need to find some way to get offsite copies of the most important stuff.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
"Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies." - Linus Torvalds
Ethics II Axiom 2. "Man thinks." B. Spinoza
Mea navis aericumbens anguillis abundat
The NIST report didn't say optical media were inheritantly unreliable at all; it said that there were big differences in media quality, and that storage conditions were important.
Personally I think hard drives are the pits for data reliability. The drives are good for MAYBE 3 years, subject to all sorts of electrical failures, and even if you have a RAID you still can lose the whole thing due to a {virus,controller,power supply,filesystem,usererror}.
I use redundant MAM-A gold stabilized CD-Rs for my data which were the most stable option in the NIST report. That works great for everything I have including digital photos.
DV might be a pain with CD-R so I would probably start with staggered redundant sliver DVD-Rs until I saw some more data on the lifetime of this media.
No way would I consider hard drives an acceptable archival solution.
This is the pot calling the kettle black, though. Is there a support group out there?
1. Human Stupidity, one mistaken format of the raid instead of that USB drive and poof.
2. Localized disasters, Flood, Lightning, Tornadoes, Blizzards, and Fire are all things that will can trash a raid.
3. Human malice, theft, vandalizm, hackers, viruses, worms and the like. Offline storage is less suceptable to these issues.
Storm
Your computer's own hard drives should keep only what you are actively working on. Get the rest of the stuff out of your way.
Buy GOOD DVDs ... burn all the files you are not actively working with to these - two separate DVDs for each archive, of two different brands. Check for file integrity, label them well and store them in a convenient, off-site location, cool and dark. Delete the originals from the working drive. Check the archive disks fairly often for degradation and re-burn as needed. They are no more labile than negatives and videotape.
For the large files, buy removable drive bays and holders, and copy them onto large hard drives. REMOVE the drives and store them with the DVDs.
On your working system, continue to back up the data for the active projects. Consider getting a RAID 5 system for data integrity, because if you back up data from one drive to another you risk overwriting a good copy with a bad copy.
I keep reading comments about how CDs/DVDs are unreliable. Here's a great trick i use to make sure my data is safe: i always include 50-100mb of parity files on each DVD. The disc would need to be REALLY messed up to be unrecoverable.
At work we have to archive Broadcast Quality TV Shows (Yes, I work in a TV station). These are 50 Mbit/s for the video plus at least another 4 Mbit/s for the audio. Needless to say this takes up a lot of space. For this, we use LTO-1 tapes that store 100GByte per tage uncompressed (compression gets us zilch with the video and audio). The tapes have error correction that we pay attention to. If there are getting to be too many errors we replace the tape and have the info copied to the new tape. Since we have so many shows, we are moving to LTO-3 tapes that store 400GB per tape. The LTO tapes are expensive. However, as long as you do not do constant reading from them and use them as a true archinve they should be fine. For massive redundancy, put the same files on two different tapes. Also, the reader/writer is a little expensive, but you only need one. Also, LOT-3 drives can read/write to LTO-1 tapes (only as 100GB, not 400GB). Write speed is pretty good to, being above 14Mbyte/s. Shelf life in a temperature/humidity controlled environment is pretty long. A bank vault should be pretty good as well.
Fly me to the moon Let me sing among those stars Let me see what spring is like On jupiter and mars
First that comes to mind is Tape backup.
So what you'd suggest is that he downloads the video from the MiniDV tape to the computer, then archives it onto backup tapes. Why not just keep the original MiniDV?
"I've got more toys than Teruhisa Kitahara."
While I love raid, RAID is not a backup - raid is about availability and consistency. So if you delete one item in a RAID it is SUPPOSED to be lost to the entire array.
/. readers, but it's two 1-line scripts and I've seen them on here before :)
/etc belongs in "current"
In everything I've read, the moral definitely seems to be harddrives, lots of harddrives, for price performance. I'm assuming you have a reasonable LAN or can set one up.
Here's the setup I haven't finished implementing yet: PLEASE give me any comments about it to help me improve my setup.
1. Setup a file server using at least one big, inexpensive disk. (This can also be a desktop as long as it can reasonably serve files.) This is your "USE" server.
2. Separate you files (on a per-directory basis) into categories based on how frequently they are changed. The important consideration is: 'If a file is changed/deleted from USE how long should I wait delete a file in the backup' Personally, I only need two categories. "current" = a month or so depending on disk space and "archive" = never (family pics, videos, etc.)
That means that if I delete something in my "current" tree _AND_ I don't notice for a month, my backups will delete it and it's gone forever.
3. Setup a 'backup server' using at least one inexpensive hard disk. Set your backup server to login to your USE server and sync your files.
It should be able to do both "full" (copy everything) and "incremental versioning" = "IV" (if something is changed, keep BOTH copies, marking them appropriately) backups. Neither of these kinds of backups should ever eliminate any information automatically - they should just add information.
4) For me, I'd run:
1) An IV backup of "archive" every night.
2) A full backup of "current" every week.
3) An IV backup of "current" every night.
4) A job that deleted the oldest backups of current every week.
Notice that I'm _never_ running a full backup of "archive" but I'm also _never_ deleting the backup.
Notes:
rsync or rsync over ssh is my preference for doing this kind of backup. It works very nicely, but I'm too tired to get it right just this minute so I'm leaving IV/full backup commands as exercises for other
cron is fine for setting it up automatically.
wget has similar functionality to rsync for a website and you don't need any privileges.
I think most of
Do make sure you log the output of your syncing software. Also make sure you monitor disk usage. If you want to be fancy, it could keep all of the full-backups of "current" until space is short (with a reasonable margin) and then always delete as many of the oldest ones as it needs to to make enough room. This means your number of snapshots will vary with disk space - some people think that's evil.
This system scales reasonably well - for more size add more harddrives per server and/or more servers. For redundancy add more backups per live copy. As long as you can keep it organized and your network handles it, there's also no reason a USE server can't be served by two backup servers or a backup server can't also serve several smaller workstations - or any combination thereof.
Do not add multiple harddrives to a backup server for redundancy. These servers are essentially free and you get much more redundancy (and some scalability) if you use two backup servers. With a setup like this, any server should only have one copy (excepting multiple versions of the same tree)
You could just do a full backup of current every night or whatever, and you could have many possibly more complicated "current" backup schemes. But for me the total size of "current" is massively smaller than "archive" so it's really not important. Remember, having more of these isn't more redundant - they're all on the same drive.
This backup server should generally run no services except possibly ssh and certainly shouldn'
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
Could be a number of reasons to do the seemingly illogical double-hop method.
First, the new data may have been processed (edits, color correction, etc).
Second, the backup media may be better rated for long term storage. I'm not familiar with MiniDV, the stuff I work with is all DLT and HCART2 under Veritas Netbackup, at 200GB raw/400GB compressed per tape.
Third, it may be helpful to have the indexing done for him by a good backup program.
However, as I say, I work with Netbackup. To say it's pricey is an understatement... but it's changed my views on what's a "workable" backup system to only liking enterprise grade stuff.
This might sound obvious.. I'm a photographer.. I have a 2 bay firewire drive set I use for "HD" backup of my photos and video. Its 700 gig. I also burn DVDs.
When I backup my stills onto dvd I use jpeg 2000, its lossy but really not that bad once the image is in a good state.. I did some tests in college on jpeg/jpeg2000 vs tiff (uncompressed) of the smae image to see how much is lost. Not a lot it turns out. I love uncompressed images, but the loss when storing as jpeg isn't so great to matter unless you do a lot more manipulation. I'm also still shooting film which can always be rescanned at a later date.
However, you shouldn't backup all the DV (raw video) you dump on the computer. The original tap e can act as the backup. its still on the tape even after you dump it into the computer. Label it and set the right protect notch. Voili, instant backup footage.
I'm assuming you edit this down and give the client a dvd/video. Just keep a DVD copy for yourself. Thats all they can really ask you for. If they come back at a later date, because the dvd is bad try yours. If that doesn't work you have to go back to the tape and redit and recharge.
Why not just keep the original MiniDV?
Probably because mini-DV holds about 13GB and an LTO has a capacity of 400GB. Get a 4 tape autochanger and you've got 1.2TB, or about 92 Mini-DV tapes.
According to the article itself, some CDs stand up well to prolonged high light exposure. The best is silver + gold coating with a phthalocyanine dye. According to http://www.silverace.com/dottyspotty/issue12.html, the only disc like this is Kodak's Ultima Silver+Gold CD-R. Other discs that do very well use the phthalocyanine dye. According to Roxio.com, the "phthalocyanine dye is pale green, appearing yellow-green on a gold-backed disc."
Also, keep your discs in the dark at controlled temp and humidity if possible.
I have been in the same position the Author discussed, and I have come to ONLY negative conclusions. In a few words, and I hate to say this, but buddy:
WE'RE FUCKED.
Digital is a loser's proposition. backing up to analogue or even digital data on analogic substrates (such as DV tape) fail. Simply nad purely.
The *only* thing that comes close is some kind of RAID, and those, even with the plummeting price of storage, are still too expensive given the needs.
Also, a RAID assumes a continuity of several things that are not likely to be continuous:
With Video:
Framerate, number of lines, colour depth, aspect ratio, file format, compression format, Operating system compatibility, etc etc etc. All of these things are variables.
With Audio:
sample rate, compression format, bit depth, file format, etc.
Basically all of it points to very bad places.
I am fairly well convinced that our age will simply disappear. They will find our garbage, the few books not pressed on acidic paper, our paintings (fat lot of good the abstract stuff will mean to them) and drawings, that's about it. the rest will just be shiny little bits of crap in the landfill.
Since we will have used up all the dense energy forms, they will be appalled at the energy requirements just to get the few remaining museum piece devices to work. Archiving the 21st century will be impossible. To the 25th century, the 21st century will be seen as a dark age - not only for the holocaust of the die caused by the failure of the petroleum based economy, but from the simple fact that very little of the information formats we are totally geared into will survive, including this note on /.
His problem of saving personal video is just the tip ofthe iceberg. His problem is the problem of our very civilisation, writ small.
That's why I am abandoning video, and going back to painting. In 500 years, my painting CAN survive. the video simply won't.
RS
Shoes for Industry. Shoes for the Dead.
For a decent RAID 5 system, always a good alternative if affordable and you require access to images all the time, why not check out the links from related /. stories such as the one below taken from a /. story earlier today?
http://groups.google.ca/group/alt.comp.hardware.pc -homebuilt/browse_frm/thread/63ac764decdbfef4
FYI: It describes building a (reletively) cheap('ish) RAID5 server with >2TB of sharable drivespace to play with.
Rsync ( http://rsync.samba.org/ is really great for backup of Unix-like systems. The ability to hardlink identical files allows me to store hundreds of daily full images of 100GB of sources to a single target 250GB hard disk. Rsync is very smart about moving only changed data over the network, resulting in speedups of 10x to 100x. This allows me to do full backup on my offsite colo without using a lot of bandwidth. Note that Rsync is great for Mac/Unix/Linux, but it does sometimes have problems with windoze clients. But then, so do I ...
Dirvish (originally written by jw schultz) is a Perl wrapper around Rsync. It facilitates the scheduling and management of Rsync based backups. We have a fairly active mailing list and contributions from around the world (open source is so cool!).
Backups should be safe against:
Backups should be automatic (or they will not get done) and cheap (hard disks are cheaper than tape, and much cheaper when you use hard linking). Rsync stores the data in a file system closely approximating the original, which facilitates restores.
If a cheap electrolytic filter capacitor dries out in your power supply, and the 5V output decides to start making a 15V squarewave instead, everything in your computer case will get fried. Including every one of the RAID disks. External USB enclosures (or airgaps!) protect against host and power supply failure.
If I was really paranoid about protecting my data, I would run a long ethernet cable to a nerdly neighbor a few houses away, and put a second dirvish server there. While I do rotate my drives into ziplok bags in a fire-resistant safe, the maximum credible accident (a furnace explosion) would tear open the firesafe. If I was paranoid and rich, I would use a high bandwidth VPN connection to a big disk in a colo machine in a different city.
The best backup is server-pull, frequent, automated backup onto multiple R/W media in multiple places, and frequent checking of that data. The closer you can approximate this, the more secure your data will be.
Keith
Keith Lofstrom server-sky.com
If he recorded on MiniDV, then GREAT, there's never a reason to tape over the master copy of anything!!
But most of the time, it's a digital camera, where it's flash ram it's recording to. For digital video, it's most likely in need of some editing which is the whole reason to bring it into the computer anyways, which is when you need to start the backup proceedure.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
If you want a higher level of home/small business office fireproofing, look at gun safes. They have to be more heat resistant to contain a cookoff.
Example - I don't work for Cabelas but I order alot of shit from them so thats where I went to off the bat.
"Stack-On Fire Resistant Personal Safe
Perfect for storing valuables, documents and more. Both of these fire resistant safes are ETL verified to manufacturer's fire protection specifications. Up to 1,700 F. for one hour with the interior temperature remaining below 350 F. Solid steel, pry resistant door with four-number combination and key lock provides greater security. Holes for mounting to the floor are pre-drilled. Fastening hardware is included."
http://www.championsafe.com/tech/fire.asp
Inbound pictures are stored on the work machine.
Every now and then these are moved to a RAID5 array.
Any totally critical stuff is also backed up onto DVDR.
Not optimum, but unless you want to spend a ton of money......
r.
While RAID was originally meant for data security, availability, and consistancy, it has a lot of other applications that weren't in it's original design.
First of all, disks are *so cheap* these days, hard drives are a more than acceptable backup medium. As disks tend to be identical in size and construction if you buy in batches, disk-to-disk backup is quite the good system, just as long as you don't always keep the disks in the same location (aka, not even on the same controller! *gasp*)
Secondly, you went into a lot of specifics that I didn't care to; a lot of backup systems are custom tailored to the situation.. so while this kind of system might work great for you, I doubt if it would work so well in this case, especially. Digital media tends to be very non-compressible, very volatile media. That being said, operations like MD5 are very crucial to insure the data from one location matches another, which means even more precautious MD5 storing measures. You're also dealing with larger files which means rdiff is almost entirely out of the question.. I could go on and on about different, application specific schemes, but I feel I did good enough with suggesting three different mediums and to have at least two copies of two of them, preferably in 4 different locations.
"Victory means exit strategy, and it's important for the President to explain to us what the exit strategy is." G.W.Bush
These are the only ones I can trust to be around in 100 years or more.
*All* digital images get written to CD-Rs are are stored in a commercial document-control facility. But the ones I really want to keep get written to film.
Best Buy can have you arrested
You people don't get it do you. RAID is not for archival.
It is to survive a hardware failure &/or increase speed. It
is not meant as a backup device to archive data. Accidentally
delete a file and its gone from your RAID. Accidentally
overwrite a file with same name, the original is toast. Lose
two drives in your RAID array, good chance your data is gone.
As others have said, optical may not be as reliable as once
thought, and is not practical for large files. While tape
suffers from drive obsolescence, the media aging rate is
fairly well known and less random than optical and can be
planned for.
If you (i.e., anyone) really needs to archive large ammounts of data, there is only one viable solution: tape.
The price per gigabyte of tape is much lower than hard disks. The large capacity drives are expensive, but in the long run, the higher media cost for hard drives will be even more expensive.
MM
By including this sig, the copyright holders of this work or collection unreservedly place it in the public domain.
. . .if you delete one item in a RAID it is SUPPOSED to be lost to the entire array.
Deletion, however, is not the same thing as corruption.
KFG
Actually, I was wrong.
I take back what I said before. I could have sworn it was the other way around.
Never mind.
For the past five or six years, I've been taking my data, applying steganography techniques to encrypt it into the background of porn images, and then distributing those images via usenet and a few porn sites I've whipped together (ok, ok, the bangbus videos.)
At any time when I need to recover the data, I just use google to find someone with a copy of my data, download, decrypt, and voila!
This is my cheapskate's Network Storage Device!
A "media" safe, say the Sentry 6720, with about 4x8x8 inches capacity, runs about $300. I recall the manual saying it was some kind of water buffer in a sort of gelatin like state, that slowly burns off, keeping film and CD's safe for the required time. It is noticably heavier than the much larger letter-size fire safe for paper.
Also, I wouldn't really worry much over fancy locks. I remember going into a safe shop where the owner pointed at a menacing 3 ft tall safe with the whole mechanism ripped apart. He said it was an easy 15 minute job with a crowbar. I hang the keys to my safes in plain sight. I'd rather a thief pop it open and decide it's not worth hauling 100 lbs for simple home videos or pre-school art projects.
My method:
1. Raid 1
2. Nightly auto rdiff-backup
3. Occasional manual rdiff-backup, elsewhere
4. DVD-R DL
It was called the IBM 1360 Photo-Digital Storage System [wikipedia.org].
Damn it I was getting ready to right a patent and cash in if anyone else came up with one....
rm -rf * shouldnt work unless your on root, and if you use root as your normal account, you are a friggen moron. /usr, that's irrelevant because all the hard-to-replace files are in the user's home directory. The fact that your system remains stable and functional is little consolation if you've lost all your files.
This common platitude doesn't hold water. While you can't delete
The rest of your post exactly the point I was making... nobody's arguing that RAID is useless... just that it isn't an effective backup mechanism, and you need to supplement it with something else.
I have half of your problem, namely the digital photo storage problem. My solution has been to copy the really important photos to negative film. It's usually expensive, but luckily a friend of mine has a digital film recorder, so I only have to pay for the film.
You can get a "decent" film recorder for 35 mm film for under $1000, but if you want better results you'll need to spend more...
Here are a few you can check out if you like:
http://www.ctcsouth.com/
There are companies that will transfer digital video to analogue film stock for you, but they are REALLY expensive. If the video is really important, you can check them out. Example: http://www.cinebyte.com/
sounds like you need BackupPC
Your hair look like poop, Bob! - Wanker.
Infomation now is much more perminant than it was in the past, and digital has improved this a great deal. The amount of information we generate these days is enormous, far more than ever before the digital age. Thus it's not supprising much of it gets destroyed. For that matter, most of it isn't worth saving anyhow.
Books are not such a perminant media as you might think. They wear out, and can be destoryed. A good example is the Mayan Codices. Records seem to indicate there were thousands, however Spanish priests burned them as "works of the devil" during the European conquest of the Americas. Today only 4 remain.
Digital data can be so perminant because it is so easily copied. Perminance of data does not come form trying to make a single, eternal copy, but from having many copies all over the world. Digital data can be copied for essentially zero cost very easily. Thus it's easy to give it a great deal of robustness. Also, as new formats come out, you simply copy and convert the data. I have data on my harddrive today that orignally existed on 5.25" floppy for the Apple II. It has simply been copied and converted a number of times.
Finally, it's not like book are going away. On the contrary we publish millions of works a year amounting to billions of books.
You seem to have a false sense of perminance, as though in the past things were archived forever. That's not the case, actually, most data was lost, that's one of teh reasons we have such an incomplete picutre of history. You don't even know all that was lost, because the record of it even existing, if there was one, is also lost. What has survived is by chance, or by effort, not because we had some wonderful archival system.
You don't have to have something on an immutable, indestructable medium for it to survive. The Nordic Legends weren't written down for centuries, yet today we still have them. They were passed down, as an oral traditon for generations. There was no perminance to them other than stories in people's minds, yet they've durvived thousands of years.
Firstly, please notice I'm an IBM employee working with IBM Tivoli Storage Manager (TSM) for eight years now.
;-)
Secondly I am a little disappointed in the answers you have gotten so far. Both in seriousness and in quality.
And as a remembrance to the first point, whenever you read TSM in my reply below, keep in mind there might be other software or solutions that could be of help...but I think myself TSM might be the best solution for you.
At first a small setting of the environment: you and your wife work with large files, and with a fair amount of total data. So in the end, you need high quality software and hardware that enables you to move all your data on all your computers offsite and/or to newer storage hardware. This hardware will be costly, but as you and your wife are professionals, you should take into account any extra costs for backing up and archiving as occupational costs.
If you do not have the amount of money needed...then you will always end up with solutions that are not answering all the needs you pointed out.
TSM will manage the data of multiple computers. These will need a network connection to the TSM server, which might be located on any one of your computers (it doesn't uses much CPU, but it will eat up 1 GB of memory though). So likely you could use any available old/spare computer you likely possess, as a DV professional I expect you to buy faster computers on a regular basis.
You likely want to keep track of your backups which is about storing different versions (that is, when editing a file you create a new version) of the same file over a certain period. A restore should likely be possible for say some version of some time ago. And sometimes you would like to be able to group some files together for backup and versioning, as the one without the other maybe pointless. And whenever a files becomes obsolete (say after ten versions or any old version after a defined period), it should be removed from your precious and expensive storage
You also definitely want to archive which is storing a partical version of a file for a certain period (maybe forever).
This is all possible with TSM and the right hardware. Have TSM perform backups regularly (scheduled daily or moreoften if you want) or manually started in between whenever you need one. You can chose what files to backup, and what not. You can also create archives with TSM, which will be stored for any period you define.
But then it becomes serious: what will you be using for hardware? With TSM you need a storage library which can store all versions in your backups and archives online, so you can access it without hassles. And likely you want to use removable media so TSM can make duplicates of your data which then may be brought offsite (any vault or shelf in another location).
I think you have two possibilities: either a tape library or a optical library. There is really not much against optical libraries if only that tapes like LTO3 may store 400 GB uncompressed (800 GB compressed), which exceeds any optical device by far. But do not compare filesize to device storage as TSM will split any large file over any amount of opticals or tapes if needed.
Lastly, you and your wife are a DV/photo professionals, not specialists into business continuity. So you likely want a one-stop solution that is easy to work with, uses TSM and a library, and is pre-setup. You then might take a look at www.storserver.com. They have solutions with disk-based storage only, or additional tpae libraries etc. etc.
There might be others I do not know of, please consider an IBM Business Partner for more help in this.
In the end: such storage is much more expensive compared to USB-HDs. But these won't last forever, do not keep track of different versions in your backup, manage your archives and it won't keep all your backup and archive versions online.
Still, you could use maybe a single USB disk for each of the seven days for backup, and use additional USB-HDs for archives. As soon as your daily data exceeds the USB-HD capacity, you come into trouble again.
The video files that your NLE uses are exact duplicates of the data your camera writes to DV tape. Take a hint from that and just save your DV tapes. All modern NLEs work with EDLs (edit decision lists), so save your session files, overlays, transition parameters, etc to a CDR and push the lock-tab on your master tapes. Keep your tapes labelled and organized so you wont have a problem finding them again. It's trivial to recreate your project at that point, and it thankfully isn't MPEG-compressed on a video DVD.
Alternately, all modern NLEs have 'export to tape' functions. Just record your final product back out to your DV deck or DV camera and make a master archive on tape.
No you're going on with the load, the load of shit that all digital data is important, that it all needs to survive. No, not so much. Most of what we generate will be lost, and that's just fine. Been that way for centuries. If it is important, precations can and will be taken to preserve it.
/. posts, as the amount of data increases, the amount that doesn't matter also increases. I post on /. for my own amusement but I do not delude myself in to taking history will have lost something of any importance should it all be deleted.
You like to talk about the bible but realise the immense effort that went in to each copy prior to the printing press. It was an amazing amount of effort to copy all of it and attempt to do it without error. These days, you spend 1/10th the effort and you'll have something far more perminant.
If I have DV that's important to me, $50 and a bit of time will allow me to archive it in several stable formats that I can then place in controlled settings.
Generally, however shit that's important isn't that high priority. I have "important" data, in that I don't want to lose it. However it's not important enough to make any serious backup attempt of. Having it on 2 seperate servers with redundant drives is good enough for me. The world won't stop turning if it's lost.
This concept that because there is data that has little perminance somehow manes all data, espically important data, has no perminance is fucking stupid. Ya, your home movie of your kid's 8th birthday party that no one cares about and that you'll never watch on DV tape will probably not survive. Important data, like weather information, will because it's copied thousands of times over to stable formats and stored.
You seem to forget that, much like
>> One thing good about paper & film is they withstand decades of storage vs. years of normal magnetic storage. Photos and films from the late 1800's/early 1900's are still around whereas you're really gambling with current storage media.
So, all you really need is a good laser printer and lots of paper. Oh, and some Chinese kids who can type 1's and 0's quickly should you lose something...
PocketGamer.org - For the gamer on the go!
A HDD will last about 10 years with constant use, but just sitting on the shelf in a dry enviorment it should last pretty much forever. Optical media on the otherhand slowly reacts with the air and light.
Anyway the big problem with optical is that you can only store 4.7 gigs on a DVDR, which is nothing to this guy. HDD's and Tape are the only possible solutions for this guys problem. I'd go with two HD's on firewire or USB2.0 and storing atleast one of them off site at the end of the day. Tape can be ok too but what is the seek time like on todays tech? If he is looking for one clip is he going to have to ff through the whole tape?
The answer to me seems like some form of software raid setup for write once only to external HD's.
God, root, what is the difference?
No.
They are a minimally acceptable backup media for short-term storage.
Consider the fact that with tapes, you really just have to worry about tape errors. If the tape drive fails, you can use another.
With hard drives, you have to worry not only about errors on the drive, but about hardware failures in the electronics as well.
In 10 years, that hard drive will probably be dead no matter what you do. But a properly stored tape backup would still be around.
> Up to 1,700 F. for one hour with the interior temperature
0 .shtml#SEC759-SUBSEC3
> remaining below 350 F.
Fine for paper (Fahrenheit 451).
Not so fine for magnetic tape (125 F) or CD/DVD (248 F) media, both would be damaged long before 350 F.
There are a number of data media rated fire resistant safes that will keep under 125 F for an hour for a 1800 F fire.
From
http://www.kodak.com/global/en/service/tib/tib430
The glass transition temperature for polycarbonate is approximately 140 degrees Celsius. If the temperature gets within 20 degrees Celsius of the glass transition temperature, there is a likelihood of significant disc deformation.
From
http://vsg.cape.com/~pbaum/magtape.htm
Other than a fire, the real danger of high temperatures (above 80 degrees Fahrenheit) is an increase in tape pack tightness caused by wound in debris, tape distortion caused by this pressure, and possible layer to layer adhesion. Print-through is increased by approximately 1.4 dB for every 10 degrees Celsius (18 degrees Fahrenheit).
-- Dan Jenkins, Rastech Inc.
That's more or less the setup I'm using too. I have the big router/home server with a shared 120GB harddisk, and a backup server controlled by a timer. At 2AM the timer turns on the computer and it makes it's backup using rdiff-backup. As it only copies differences in changed files, I don't really worry about the backup size, at the moment it's merely 1.6GB. After the backup, it generates a log in html and copies it back to the main server.
All this I wrote in bash in one afternoon.
"It's too bad that stupidity isn't painful." - Anton LaVey
> A HDD will last about 10 years with constant use, but just
> sitting on the shelf in a dry enviorment it should last
> pretty much forever.
There's considerable debate of the "last pretty much forever."
How long can bearings sit before freezing up?
Fluid bearings suffer from evaporation, for example.
Hard drive platters have a very thin lubricant layer, if I recollect. How long does that last before chemical degradation?
Devices cannot be presumed to "last forever" just because they are not being used. In fact, just sitting and not being used may actually shorten their life.
(While it is not a fair comparison, try starting a vehicle that's been in storage for a couple of years - seals dry out, mechanisms gum up, and so forth.)
So, the jury is out on how long a hard drive sitting on a shelf will last.
-- Dan Jenkins, Rastech Inc.
If the problem with optical backup is degredation due to air and light, then the solution seems easy enough. Put the discs in a vacuum sealed, acid-free plastic bag using one of these and store in the dark in a cool, dry location.
And burn a second set of DVD's for actual use, so you don't have to break the seal on the others.
Also, as history shows, storage media will continue to grow in size and decline in price. In five years, he will probably want to re-archive everything, anyway, to condense it down.
Lots of gmail accounts. Lots, lots of them.
Few people are professional photographers or videographers. This whole article is about an exception to your rule.
Under capitalism man exploits man. Under communism it's the other way around.
1. As I just mentioned in response to another post, I very much encourage your backups to actually be on another machine - if your server is own3d, or your OS/RAM/MB freaks out you have no idea what it'll do to your backup drive. If you're posting this on /., I figure you can get a machine out of the garbage and put this together... Again, it's my opinion that having more than one copy of the data per computer is a waste of HD. (other than for high availability, a la RAID)
2. I'm all about using your friend's internet connection to do this. Furthermore, in response to someone else - if you think your friend is spilling Cheerios on it.. a) get better friends and b) get MORE friends/backups. I'll take redundancy over perfection any day.
3. RAID is great in those situations where your intra-backup loss (ie, from a day) is very great.
I agree with you that a lot of people recommend RAID 10, but I think they are quite wrong OR they are using crap systems - my lengthy explanation follows.
Good RAID controllers use battery-backed write cache - that means they "accept" your write immediately and use a battery to actually put it on the harddrive LATER, even if the power goes out. This is a HUGE speed improvement for multiple small write situations, even with just ONE disk. I ignore this effect in the below discussion.
I'm going to assume a system where you have two similar drives on different buses on the same machine. I'm also going to assume that you're HD I/O bound (ie, the harddrive platters/heads are what's causing the slowdown, not your CPU) I'm also going to assume you do more reading than writing - at least more files if not more bits (which is pretty typical)
--- First, why RAID 0 is stupid (unless you're using very large files AND not using them at the same time) I'm going to compare RAID0 to just putting different stuff on different drives (for instance, OS/swap/apps on drive 0 and data on drive 1.) I'm calling this setup "noRAID"
RAID 0 is straight striping - it writes half of every file to each disk. This means that the _write_ time (time from the time it starts to the time it finished writing) is twice as fast, but the _seek_ time (time to get the head to the right place to write) is exactly the same as a single disk. For writing very large files this is almost twice as fast. For writing smaller files it is not faster at all because the seek time (time to find where to write it) totally overwhelms the time to actually do the writing. For reading the same thing is true. The "bulk" of reading a file is exactly twice as fast but the seek is not changed at all. So most of the time it really isn't faster except for really big files.
The short answer is that RAID0 is stupid because it has no benefits when seeking.
Compare this to just using 2 drives: if you try to read or write simultaneous small files that are on different disks, noRAID is absolutely _twice as fast_ If you try to write a single very large file RAID0 approaches being twice as fast as the write time becomes much larger than the seek time. Of course, the weak point in this argument is that sometimes you want two things on the same disk - then noRAID is only the same speed as RAID0 for small files. So noRAID doesn't average being actually twice as fast.
In addition, RAID0 is half as redundant because either disk failing destroys everything.
--- Second, why RAID1 is good.
RAID 1 is straight mirroring. On a modern RAID system (like Linux's SoftRAID) this gives you performance that - compared to a single disk - is exactly identical on write to a single disk (Assuming your CPU can always keep up) For multiple file reading, though, it peforms better than any other setup, even _better than noRAID_ because it only needs to read from 1 disk and it reads from whichever disk has a head in a convenient spot to do THAT read.
It doesn't have the disadvantage of noRAID, because it ALWAYS has a copy of the data it needs on the
Looking for freelance Actionscript (Flash/Flex) or ColdFusion work and/or freelance developers. Email me, put Slashdot
In considering whether you need a fireproof safe, and whether such a safe would be OK for media, it's helpful to go back to first principles:
If what you are trying to protect is truly a backup, and not an archive (that is, it is a spare copy, not the only copy), then it is not necessary to protect both the original and the backup from fire (or flood, etc.). It is necessary only to ensure that the hazard doesn't affect both at the same time.
If you want to store both in the same room, or even in the same house, then indeed some sort of fireproof safe would be needed. But it you can store one offsite, then no fireproofing is needed. There's defnitely no need for a safe-deposit box at a bank!
Once I simply rented an additional locker at my athletic club (in the hallway, where they were cheaper, not in the humid locker room). Another time I stored my home backups at the office, and vice versa.
At the offsite location, you may need theft protection. Hiding is the best way, but there are many safes that are very good at this, even if they're not fireproof.
Another reason for not using a safe-deposit box is that you want the storage location to be easily accessible so you will use it often. You don't want your backups sitting on the hall table for two weeks waiting for your next trip to the bank!
[Serious mode on] I once heard in a conference that the most reliable filesystem ever was the ed2k network. [Serious mode off]
So, here goes a suggestion:
1) Zip/Rar it with a password.
2) Divide it into downloadable chunks.
3) Rename it to "Lesbian chicks hot stuff!!!!", or similar, you get the idea...
Et voilà.
I'm a little late on this one, but I worked for a photography studio for a while and we were all digital. Our best way of backing up photos was to make use of external harddrives and keep a pretty basic filing system for customers pictures. It works very well for pictures, and it's not too expensive in the long run. Never worked in video. No idea about the needs there.