Slashdot Mirror
Best Way To Store Digital Video For 20 Years?
An anonymous reader writes "My kid is now 1 year old and I already have 100G of digital video (stored on DVDs, DVD quality) and photos. How should I store it so that it's still readable 10 to 20 years from now? Will DVDs stil be around, and readable, 10 years from now? Should I plan for technology changes every 5 to 10 years (DVD->Blue-ray->whatever)? Is optical storage better, or should I try to use hard drives (making technology changes automatic)? And, if the answer is optical, how do you store optical disks so that they last?"
CDs are still readable, after almost 20 years
store it in analog engravings in diamond
Different media, copied over to new media after a few years.
Pictures: Backed up to HDD, DVD and Flickr. For $24.95, it's cheap offline backup and the grandparents love it.
Movies: Taken on MiniDV, backed up to HDD.
The only worry I have is that the MiniDV's and HDD are in the same house although they are stored in separate locations. But every picture is backed up offsite.
Claim up to 300 years.
http://www.smarthouse.com.au/Home_Office/Storage/U9P4F7L2
2D bar code?
^(oo)^pig~
HDDs are so cheap. Buy an external one with like a terabyte of space. Fill it up, rinse, repeat.
Crackin' Wise - Blogging about whatever we want
Storing them on DVD->Bluray is probably the way to go. Keep them in a dark storage area away from heat and moisture. It will be neccessary to extract the data and keep up with technology as even optical storage has a shelf life. However 20 years from now I don't think the problem will be the disks themselves but finding a drive that can read them.
Currently, There is no better way than store a backup on DVD and store the main data on a raid-1 disk set. Move the raid disk set to new disks every few years.
All the other technologies are more expensive, and even possibly more dangerous (loss of data due incompatibilies or for any other reason).
If you strive to keep it all accessible all the time, you will move with format changes as they occur. US networks are not capable of HD video streaming, so I put OGG Theora in my video blogs with links to better quality for those who want it. Disk storage will improve in time to keep up with your vorracious demands. Raid would be good to have. Optical storage media that has to be loaded one disk at a time is a last ditch archive that you should keep in a seperate physical location, just in case.
Intellectual property was the desert property of the twenth century.
Depends on the manufacturer and dye formulation. Some have failed in as short of a time as eight months while others are good after nearly ten years. For very important stuff, it is far too risky to be relying on the manufacturer. It's probably safer to make it a habit of regularly make multiple backups your data.
ELOI, ELOI, LAMA SABACHTHANI!?
In the department I work for we typically use archival DVDs in a temperature and humidity controlled room (also used to store photos, slides, and vellum). For the really important ones I'll copy the disc onto a server in the same room as an ISO. Every month I mirror the data drive onto an offsite server in another building on campus. It's not fool-proof and it's pretty expensive but it has worked for about 8 years now.
"Never let your sense of morals prevent you from doing what is right" - Salvor Hardin
Well why not optical platters? We have 50 year retainment requirements for certain documents and were looking at Plasmon optical devices. They claim it will still be readable and are the only type of backup media that survived both 9/11 and Katrina. Although when I asked if it was the same cartridge that survived both, the vendor gave me a dirty look. I think though you would be fine with dvd-r and just make a new copy every 5 years.
Only wimps use optical media, _real_ men just upload their important stuff on ftp and let the rest of the world mirror it.
Store the media on whatever the current standard is. Think about it, what if you had a closet full of tape reals that had all of your old sweet groovy 60's music? What would you do with that now?
-FizzGiGG (Geek)
Build a simple storage array with RAID from a barbones PC, your favorite Linux distro, configured for fault-tolerant RAID. It doesn't have to be complicated, and it doesn't have to be powered on unless you're actually pushing data to it.
Every couple of years, you can add an extra couple of drives. With drive capacities increasing as fast as they are, cost shouldn't be a huge issue.
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.
From what I understand, tape is still one of the best archival storage types
Regardless of the methods you choose, I would highly recommend using at least two different media.
If these videos are important enough to be stored for 10 to 20 years, then they are important enough to be backed up - it is always difficult to foresee long term failures in any technology. If you read the article on tin whiskers they mentioned that some failures can not be tested using short time span methods.
a few years ago, this would have been exorbitantly expensive overkill, but this stuff keeps getting cheaper by the day
with raid5, your videos will last forever, as long as someone keeps replacing the dead drives
any other media format is physically static, which can degrade. raid5 ensures that the files live on after the physical components degrade, as long as new drives are continually added to the system
and when the technology becomes ancient and archaic, simply move the files over and upgrade (obviously to a new file format as well)
as long as some continually performs low level maintenance, your videos will last forever
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
My kid is now 1 year old and I already have 100G of digital video
/.
Riiiight, this is
So people: How can this guy be sure that in 10/20 years time his Bangbus collection will still be readable?
Use multiple different media, with redudancy.
Store it on HDDs. Mirrored RAID like RAID 1 or RAID 10 is preferred. but even RAID 5 buys you some extra integrity protection.
Then back it up. CDs. DVDs. BluRay. Tape. Whatever. Multiple times, multiple ways. Every few years do some copies onto new media.
Keep at least one copy off of your premises. A safe deposit box might be good.
My blog
Step 1: Review video footage.
Step 2: Carve memorable/important parts into stone.
Step 3: ??? (mummies?)
Step 4: Profit!
crazy dynamite monkey
If you can afford it, I'd recommend a utility computing platform, like Amazon S3 or whatever Google's offering in that space. Verify that they're built out for long-term, fault-tolerant storage (ie: replication + automated verification and repair.)
I wouldn't trust that 100%, though, so keep them locally as well.
My dad has some now that are over 50 yo. Thank you. "Hello, world."
Video I posted 20 years ago is still there....
Unix, an obscure operating system developed by bored researchers in an attempt to get a better game playing experience.
I watched a video of my parents wedding a few weeks ago. I was surprised that it still worked. It was on VHS and 20+ years old. (second marriage, I'm older)
I believe the general consensus is that no form of media will last much longer than 20 years. However, digital media does not experience generational loss (you can make perfect copies). Therefore you can just make a new copy every 10 years or so and it will last forever.
One of our competitors trademarked the term "hypothesis". From now on, we will call them "boneheaded ideas".
Instead of hiding behind the camera the whole time, actually interact and play with your kid. The videos and memories aren't as interesting as who the kid will become.
MPEG-2 will always be around. Like JPEG and MP3 decoders.
Amazon will be around FOREVER. That's what they tell me anyway.
Make 2 sets of copies and every 1 - 2 years copy it to the most appropriate format.
So first time round it'll be 20 DVD-r's, then in a few years or so 4 blu rays and so on. After 10 years it will probably just be 2 copies of some holomagical disk that will take about 20 seconds and thus be so easy as you'll forget to do it.
As a general rule, long term storage of pretty much anything means keeping it away from oxygen, water, and sunlight as much as possible. I have audio tapes from the 50s that still work fine because I stored them in a fairly air-tight Coleman ice chest. (an antique model - steel with a plastic liner, insulated with fiberglass)
Nothing worthwhile ever happens before noon
1. Rename to "xxx 18yr old bj strip"
2. Upload to P2P protocol of choice.
Let it proliferate around the internet and retrieve it when necessary.
As for storage, I would personally go through and put together maybe a movie and and picture viewer DVD for each year. And then have those professionally mastered onto pressed discs. Keep those in your fire-proof storage and use burned copies for everyday (I hope not) use and sending to relatives and what not.
Shift happens. Fire it up.
As the other guy mentioned, CDs are still readable, almost 20 years later. However, they didn't have a viable alternative until about 10 years ago. I think that you will easily be able to find a DVD drive for many years to come, at least the next 20. The problem becomes ensuring that the actual media doesn't get scratched. I wouldn't trust DVDs to last that long, even if you just leave them on a shelf, away from the sunlight. If I was really interested in saving the stuff, I would put it on hard disks with at least 1 redundant copy, if not 2, stored in different places, and transfer over every 3-4 years. Still, it's going to be a lot of data. Your kid is only 1, and you already have 100 GB of stuff. Just think about how much that will balloon to once the kid has an attention span of more than 43 seconds. The first hockey game, all the school plays, all the other junk you could record.
Personally, I just don't bother with recording much. My wife gets on my case for not taking a lot of pictures with the kids, but I'd rather be interacting and paying attention, rather than trying to ensure we have everything recorded. Sure sometimes like during school plays you can record and not miss anything, but a lot of times, I find when I'm trying to take videos, or photos, I end up missing out on the actual fun.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
isn't long term storage, though it may have it. The strength is that you can reproduce it with high fidelity to the original numberous times.
The best way to store digital vidio for 20 years is to make numerous copies of it. 10Gigs is about 3 DVD's at the lowest density. Add a dvd of checksum files (something like a PAR) and you should still be able to make five sets for under $20 if you are shopping around for DVD media.
Once a year or three, load up one of the sets and run it through the checksums. Correct any errors discovered via the checksums and copies from the other sets, and make another five sets.
Volia. Repeatable as long as there is any sort of cheap digital recording media that can easily fit your files out there.
The real question is how you do this when you have 1,000 Gig to backup.
You'll have 2 Tb? Or a lot more, if technology goes asymptotic?
I suggest you invest in an editor, and slim your storage down to what is reasonable given current technology - perhaps 5 CDs?
Then transfer to new technology as it appears, keeping only the amount that each new technology can reasonably handle
Alternatively, buy yourself a data centre..
Oh, there is one other way. Just send a few messages off to Iran asking about nuclear materials, and then send all your kids' photos over the net. The governments of the US, Europe and the Middle East will then keep all your data in a high security storage facility, free of charge...
I would Say CD because those have been around forever and I don't see them dieing out any time soon, look at tape. DVD's would save you the hassle of storage, even with blue-ray, because DVD has been an established standard for so long I don't see any means of reading it going away any time soon.
Quit being creepy by chronicling every time your kid goes number 2. He'll thank you, as will everyone else who knows you.
If you're not trying to go cheap, get a tape drive (DLT, LTO, or AIT, not the quarter-inch or DAT crap). If your time isn't worth that much, migrate from optical format to optical format every few years. Either way, keep your backups off-site.
Hard drives and just not suitable for (home) archiving - one robbery, fire, or natural disaster and everything's gone forever. If you add backup to those hard drives, then we're back to "what format?".
Socialism: a lie told by totalitarians and believed by fools.
These days we are offered the luxury of cheap storage in a variety of formats (Flash, Magnetic, Optical, Offsite/Online services).
The last thing I'd trust would be magnetic media, especially media with moving parts (portable USB hard drives being the worst offender).
Optical is nice enough, if you take care to store the media in a safe place, or offline easily. I'm not a huge fan of burning data to disk, as it's usually a task of finding a sharpie before just setting the thing somewhere and forget what's on it. (I've literally got a whole spindle of "unknown stuff" because I'm too lazy to commit to the task of doing it right).
Off-site/online backup is very convenient with software being able to automate it nearly seamlessly for you, assuming you have the bandwidth to get it up in the cloud and trust your storage providers. Assuming you do, at least you can rest somewhat assured that they have redundancy and backups which removes the worry/maintanance on your part. It's a nice "set it, and forget it!" mentality, and prices are reasonable so you don't have to worry about it.
I'm a victim of not being able to take my own advice however. I usually just fill up a 250GB harddrive, take it offline and let it hang out in a static bag in a cluttered desk drawer. I'm surprised that my QuickBasic files from 1992 migrated their way somehow from floppy to mass storage.
90% of the stuff I squirrel away is nostalgic crap however.
Store some on DVD, some on Blu-Ray, some on flash. Store multiple copies. When new formats come out, copy to the new formats. Archive source code for the decoders. Redundancy is the key.
Who is going to want to watch 100 gigabytes of your kid? At 5 GB / DVD, that's 20 movies = 40 hours. I don't care if my kid is the next Beethoven, I'm never going to watch 40 hours of diaper footage in my life.
I suggest keeping it on CDs in different places. Hopefully about 15 seconds of footage will survive.
Nobody is ever going to want to watch videos of your toddler 20 years from now, including yourself... that is, unless he, your wife, and extended family are all killed and you end up drinking yourself to death with old home videos playing while you plot your revenge. That is really the only scenario where anybody will ever end up watching that crap. Since this is obviously your first kid, it's understandable... you'll realize in a few years how little digital home video storage really matters when you have real family issues to worry about.
Keep backing up to whatever current ext storage devices exists. 1TB ext HDDs are available for $199 at newegg.
In 1.5 years it will be 2TB, and so on.
Just keep staying with current technology. The nice thing about digital is there is no loss when making multigenerational copies.
Video is now beginning to escape the restrictions of how the MP3 of the mid 90s was, when it would take hours to encode and consume an HDD pretty quickly. With CPUs and storage doubling every 18 months it won't matter much, it will be like your MP3 collection is now and how easy it is to move it from one storage medium to another.
With S3 you'd pay $15/mo (+bandwidth) to have it hosted online, instantly accessible. Will it still be around 20 years from now? One can't be certain, but if not, I'm sure you'll have enough warning to copy things off to another medium, and I'm sure there'll be similar services to take its place if need be.
Generally, optical disks offer the best digital storage option. Of those, the best long term storage are the ROMs. DVD-ROMs are better than DVD-RWs. CD-ROMS are better than CD-RWs. (for long term storage) This is because of the recording method used, and the materials in the disks. I saw a study once, but of course I don't have it at hand, that valued the lifespan of a ROM disk at nearly 2x that of its RW counterpart. It was serious research, and I wish I could find the link for you.
If you're anything like my parents, then you'll end up with a rather large collection of home videos and other related media by the time you want to look back on it. In my parents' case, they had to transfer hundreds of hours of video from beta-max to DVD, which was a real headache. DVD to a future digital technology may be somewhat quicker to transfer than making the analog-to-digital jump, but swapping out DVDs over and over for days still doesn't seem like a fun prospect. I'd recommend storing data on a convenient medium such as a hard-drive and using redundancy to make sure that your data keeps.
I was under the impression that unspun drives tend to deteriorate relatively quickly - the heads clashing with the platter or some such nonsense. Just spin them up once a month and you're fine, from what I've heard.
How are sites slashdotted when nobody reads TFAs?
The only way this could be true is if the data were rewritten.
Reading alone has no effect on the data.
Counting on a single storage device for 20 years isn't a smart move. Anyone here still able to read 5 1/4 inch floppies on their main computer? What about 3 1/2? Even those are limited to cheap external drives. Pick a storage solution for the short to medium term, and make plans to switch to the next generation when the current one reaches the end of its life. In other words, DVD now, Blu-Ray down the road, and any future successor later.
And if you want an extra layer of redundancy, buy a decent external hard drive, copy the videos and photos to it, and place it in a safety deposit box. Pick a standard that should be usable with the next generation of technology (USB 2.0 if USB 3 continues the trend of backwards compatibility). So if we all switch to using 3" super high density optical discs before you can transfer your files off of DVD, you'll still be covered.
My backup is another hard drive. It made sense the first time and the second and the third for things like the photo album. A 500 GB portable drive has enough space for everything I want to preserve. Rsync keeps it all fresh and up to date. Offsite backup is a good idea. I still make DVD backups but they are still a last ditch thing to use. The archive I share is the one I care about.
Intellectual property was the desert property of the twenth century.
Newegg has Quantum DLT SATA drives (160GB native capacity, 35 GB/h throughput) for about $700, so it won't break the bank to get proven multi-decade shelf-life media of reasonably size and speed for a 100GB dataset.
Every real OS has tape backup support (though you may have to hunt for drivers). If you're stuck with Windows, type ntbackup at the command line - it doesn't suck for home use.
Socialism: a lie told by totalitarians and believed by fools.
There are two concerns to deal with for long-term storage:
1) Physical harm to the media.
Most physical storage is susceptible to something be it moisture or grandchildren. Sharpie ink will eat through disk labels over a decade or so and that nifty external hard drive is going to be DoA when you try to plug it in. Unlabeled CD's and optical media should be good for 20 years, but be careful with storage and labeling.
2) Accessibility.
20 Years ago, I still had a nice collection of 5.25" floppies and though digital watches were a pretty neat idea. I'm sure I have ZIP and JAZZ drives around somewhere, but they're probably SCSI. What's to say that you will even have a PC 20 years from now let alone one that can read USB 2.0 or a DVD?
The best answer is to avoid the "store and forget" option. Keep a copy of your data on whatever media you currently use. Make regular backups and keep them offsite (safety deposit box). If you need to change media, you'll have a much easier method of converting at that time than 5 years further down the line.
Personally, I'm too lazy for this, so I'll go with:
The easiest answer is to use web-hosted data storage from a major company. Pay for it and you'll be pretty much assured of getting notification before it gets lost to the forces of capitalism. You can also be certain that data backups and storage procedures are handled with at least some professionalism.
chiseled into stone.
Seriously, I ask myself the same question occasionally.
Recently I had to clear the house of my deceased grandmother. None of my relatives was willing to do it, they wanted to put everything in an auction. I went through everything and kept things that I thought were sentimental or just plain shouldn't sit in a cardboard box at an auction. I'm glad I did. Aside from some real family treasures, I came across a footlocker in the basement containing about 20 pictures printed on tin of individuals and families. They had to be 80-100 years old. While I was looking through them I started wondering about the way I store pictures now. If someone found a cf card or cd or hard drive of mine 100 years from now would they be able to read it? Would it be readable even if they had the right hardware?
Honestly, you're probably not going to look at most of the pictures in 20 years anyways. Sure it's good to have a lot, but with 100 GB of pictures/videos, that's pushing it a lot; babies don't exactly do much. Go through the collection and cut out the bad/redundant pictures, print off the especially good ones, and put em on the wall: they'll get more use that way. But... with what you do save, try burning to CD/DVD/blu-ray, since they don't degrade too much over time, and if you have space/money, archive it to tape: it'll last the longest. HDD's won't last the 20 years, but if you want to, try a RAID, but it'll have to be recopied and replaced every so often. And... with whatever media you choose, keep it away from light/dust/too much humidity.
01110000 01010111 01101110 00110011 01100100
No one's brought it up yet, so I will... As the price/convenience/long term compatibility and viability of storage goes down and down, I wonder to what end we will end up keeping this stuff? How many hours of video that you're paying (in time, money, security against fire/damage/loss, etc) to keep up you're actually going to watch? Sure, it's nice to have every single event in your child's life on demand at the touch of a button/click of a mouse, but aren't just plain old memories ok? Does his entire life have to be recorded and watchable?
At some point, I came to the realization that I had downloaded over 6 solid months worth of music. This doesn't include CD's, LP's, or 7 inch records, of which I probably have 1000 total. If I were able to put all that music on a big loop, and not repeat anything, I'm thinking it would last over 12 months. Some of these I'll probably never listen to. I'm thinking the same is true for the submitter's videos.
My parents have a big box of photographs from their childhoods, as well as those of their parents. There are some great photos in that box, and I could and have spent hours going through them. Each time I do, I make a mental note that one day I'll scan them and make them digital. Then I realize that we only drag out that box once or twice a year, and never do anything with the photos anyway, and resign to scan them once it gets even cheaper.
Use animal skin. Draw your data on goat-skin. I had recently an example in my hands dated from 1502. It still looks very good.
100G of video in the first year? I guess you should archive it...he can show it to his therapist when he gets older. One way to solve the archiving problem is to do some editing (serious editing) so there's much less to store.
People who say HDD have their heads in the sand. 20 years. Think about that. 1988. SCSI-1 40 pins. Nearing the end of MFM/RLE. Parallel.
People who say CDs and DVDs again have their heads in the sand. That's the Floppy Era.
The best format IMHO is the "current" format. DVDs + HDDs along with a live copy on your computer. DVDs and HDDs should be at two of your friend's houses.
5-10 years later, once one of the formats is obsolete (EXT3 is now EXT8, DVDs are now expensive again in drug stores), it's time to copy these to the new "current" format, and repeat the process.
Without meaning any offence (and I honestly mean that), what makes you think your kid, or anybody else, is going to be interested in so much video from when they were that age?
I appreciate the desire to record the life of your pride and joy, but aside from the personal impulse you have that this is important, what is the point?
My mom probably has a few hundred individual photographs of me from when I was a child, and although I haven't ever looked at them, I'm sure a time will come when I will. Nostalgia is like that. Still, if I was born in 2007 and this was twenty-six years from now, there is no way on earth that I would be reviewing terabytes of video. I wouldn't have the time or interest.
I have sat down with relatives and watched their holidays videos and found it to be the most tedious experience of my life.
Photographs are great because they give you a glimpse at a moment in time, and the person (presumably, somebody you care about if you are looking at their photos) will tell you the associated story. Its interesting, its interpersonal and it is succinct. Videos are boring as hell because aside from what is on screen, there is no extra story told by your friend/loved one, or if there is, it is the same story you would get from a photograph, except you had to watch five minutes of a baby crawling in the kitchen instead of a snapshot of same.
I know you posted this to get advice on storage media, but for what its worth, here is some advice on a related issue. Stop recording so much video. Record a few, were the video enhances your story in a way a photo couldn't, but after that, take lots of snapshots and look forward to hours of story telling with your nearest and dearest.
But face it buddy. 20 years from now, you won't be looking at the video of your kid that you shot when he/she was 3 weeks old. I have about 60 hours of video of my daughter. It is all digitized and sitting in the harddisk attached to the lap top in the living room connected to the TV. All it takes is about six mousclicks to watch any video in the big 50 inch set. Very rarely we do. But occasionally when friends come over we watch very old birthday party clips. Most interest generating ones are the panning shots where their children are captured.
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
I would probably be damaged and retrospectively focused if I had a terabyte of video (of "me") growing along with me. Learning to crack kid? To erase Daddy's archives!
//de ~ 9cimi
When the fuck are you going to finish your movie? More time filming and less time posting stupid shit on slashdot. huh?
Real men backup their stuff to FTP and let the world mirror it
So in his first year of life, you've recorded around 34-35 hours of footage? (going on single layer dvd capacities and mpeg2) Ask yourself, when are you going to watch all of that?
I wouldn't store it in a digital format for long-term if given the choice. I'd rather (and do) have all my video on Mini-DV so that I can encode it to the digital format of the time. If you're set on digital copies then plan on building and maintaining some sort of system over the long haul. Whether you use higher grade long term DVD/CDs or hard drives you'll want to make a good database or paper (gasp!) file system so that you know where your kids first time saying "Daddy!" are in the stack you're going to develop (you should also make one of these if you go miniDV as well, I think I'm up to 25 hour long tapes in the first 2 years) because once you get a mountain load of whatever your stockpiling, finding it is the next important thing after confirming it will be there when you need it.
-EB
Do you ever walk alone like a drifter in the dark?
Collude with a friend to email back and forth encrypted copies of your photos. Arrange for them to be perpetually stored in transit on somebody-else's mail server awaiting delivery.
Better still - uucp them over some convoluted circular path back to yourself.
Or rig up an ultra-high-speed moonbounce communications system...
Just keep them all in motion and they won't get lost.
Nullius in verba
Optical disks? Hard Drives? Do both and let us know which one worked better, if at all, in 20 years!
Not to be mean, but seriously dude... In 20 years, your son isn't going to care to watch videos of him getting his nappy changed, or pushing out a turd. And if those are the moments you wish to relive, you have far more serious issues than contemplating media longevity.
Why worry about the physical format? You can pretty certainly still find a 3.5" floppy disc drive today, if you really need to. It'll be some hunting, but you'll manage.
But try getting hold of a copy of DOS 5.5 and Word 1.0 to actually read the files you saved there.
So will DVDs still be around? Maybe the discs will. But I doubt the codex will still be the same.
Assorted stuff I do sometimes: Lemuria.org
I used to fret, like you, on the best format for backing up video/pics. I do not want to lose all the precious kid memories.
External HDD - this is still onsite with your computer - not good
DVD - optical - usually people stick them right next to their computer - not good
With both formats you may look at data integrity issues and format issues in the future. I have gone to storing my data offsite onto web storage or data storage. You can usually find 300GB of storage space for between $5-10 per month. Let them worrry about backing it up and keeping format compatability. Again this is only a *backup* of my data, so if the remote company goes out of business, I still have the originals on my local HDD.
I have a variety of photos etc that I care about a great deal and want to make sure they stay around.
I keep them on a hard disk, because I've had too many failures with optical media. (Fortunately, I kept backups of the optical media, and was able to recover my files, but sometimes the backups had failures too and I was just lucky that I was able to retrieve some files from the original and others from the backup.)
I then have a backup of the hard disk, on a duplicate hard disk. This backup occurs every other day by default, and I make it happen immediately after I add anything important to the primary. The backup disk is sometimes left off when not actively backing up, to try to give the two drives differing lifespans. Ultimately, I intend to replace this arrangement with a RAID 5 array, but I haven't had both time and money to put together my desired server.
Once a year, or as often as I feel like, my father and I meet and exchange backups of our photos so there is offsite storage. This is the cheap method, but it doesn't account for photos which have occurred since our last exchange. We may change to an online storage solution, after I get time to investigate services and pricing. (With about 50 gigs of photos each and growing rapidly, not counting home videos, it could get expensive.)
Yeah, a different HDD stred offsite is great, assuming you re-sync them from time to time, and replace the drives on a reasonable schedule, and have at least a 3rd drive so you still have an offsite backup when you're syncing up ("oops, copied the wrong way" is more common than you'd think). I still do this for data that doesn't change often, but the convenience of tape is becoming quite attractive.
Socialism: a lie told by totalitarians and believed by fools.
I would go with optical + magnetic (redundancy) swapping the media every 5 years and keeping the previous ones. Having flash based storage would also help. Sure, you'll have craploads of media, but who cares? Those memories are probably much more valuable than the cost of the media itself and that of storing it.
Two different media... and I would go one step past that and suggest two different types of media.
If it's that important, multiple backup copies stored in different places is a good idea.
Also, in my opinion, one of those types of media should be DVD, but not DVD for data, DVD for your DVD player. This is nearly a trivial exercise with OS X, and I'm sure there's software for Linux and Windows that can do it just as easily.
Here is my reasoning:
Can you still buy a cassette player and even recorder in a store? Sure. Can you buy a cassette data drive for a computer? No. (If you can, I'd be very surprised.)
The Video DVD is not going away any time soon. You can still buy VHS players (good thing for me, because I still have many VHS tapes). I doubt that the DVD will go away before you have plenty of time to convert to the new format, whatever that may be. If you keep with the "video" DVD format, you will have something that is more likely to be supported in the future (IMHO) because of the large install base of **AA media out there. Have you noticed that just because Blu-Ray is out there and sort of the "de facto" standard for the new high-def format that good-old-fashioned DVDs have not vanished? New movies are still coming out on DVD.
Yes, video DVD is not as efficient as data DVD because you can't compress the data as much, but it will remain a viable format for a long time, and it has the benefit of being a format that a non-tech-saavy relative can use. I am not aware of too many people who can't manage to use a DVD player. I know plenty who can't manage even to play a video DVD on a computer.
Burn it to write-once PROM chips. That should last for quite a while.
Probably not, but it's really funny seeing how my parents (and grandparents) dealt with changing technology. We've moved to digital pictures & movies with substantial quality increases and supposed longer lifespans, and I look back to how my parents stored "precious" moments of our family in shoeboxes and bins stored in a humid attic.
Earlier this year my mother came across some 8mm reel to reel films from when I was 2 or 3 years old (30+ years old) and had a company convert them to DVDs. The movies were in storage for years before we discovered them, just laying in some cardboard box stored in an attic before being moved to a cellar for a few years. The biggest reason we knew they were still good was because my grandparents left the old reel to reel player in the box so we could watch the movies. The same goes for the boxes and boxes of slides my stepfather had, he also had a slide projector handy to view the images.
Sometimes keeping around older tech comes in handy. Make sure the videos are in an open format and maybe keep a backup copy. Best of luck! :)
I like big butts and I cannot lie.
Print the bits out on paper to be scanned later, as necessary. You should make several copies and store them in different locations incase of fire or water damage. To answer your next question: Land in Montana and the Yukon territory is cheap.
If anyone knows of a way to read my old CD-R's, even if it is [b]one[/b] time only, let me know.
It's pretty unlikely that you'll ever know how well its kept. I doubt anyone's ever going to watch over 100 GBs of digital video about you child. Sorry. What you have is no different than the video tapes that my generation's childhood is stored on. They continue to gather dust in the basement, decade after decade.
Unfortunately there isn't a guarantee on any technology. CD/DVDs were supposed to last 100 years until that pesky mold and poor quality make them unsuitable for long term storage. HD-DVD was promising until it lost the format wars. HDs reliability varies with manufacturer and model. My suggestion is to back it up every 5 - 10 years onto new media to keep ahead of the curve. It's more work but you'll make sure it gets saved.
8mm -> VHS -> DVD -> Bluray -> Profit!!Well, there's spam egg sausage and spam, that's not got much spam in it.
No offense, but unless there's been a breakthrough in archival methods of digital data, why do we have to re-visit this question every few months with only a minor change?
One time it's digital files, then it's digital music, now it's digital video. Unless you are willing to go analog* with your digital data, the answers are the same now as they were 2 months ago, 6 months ago, 1 year ago. Yes, for 2 years ago nobody might have suggested Blu-Ray or HD-DVD as they weren't particularly affordable/available. But given their medium (the same in principle as a CD-R or a DVD-R), I wouldn't expect them to get recommended no either (discussions about their life being too short, anecdotal evidence of CD-Rs with warez from 10 years ago still working, blablabla) and instead you'll see tape and/or harddisks (oh but that's too expensive (wtf?), and drives fail, too! oh noes!)
Seriously. Hit the previous stories on archiving digital data (doesn't matter much what the data is, although I *suppose* with audio and video you get some slight tolerance you do not get with, say, binaries.) Then come back. Bit too late to get this off the front page now, I suppose.
* spend the cash and get it transferred to film. Actual film; they tend to last pretty well when canned and stored appropriately - the hundreds of old-old-old movies having been released onto DVD should be testament to that; any deterioration tends to show up as color shifting/loss, noise and the occasional splotch; as opposed to an unreadable file / stuttering when playing back / block artifacts forming from fill-in. The former being, to me, less objectionable. Just keep in mind that it's now analog and any copy you make from it to another film is just going to degrade further and further.
First, just let me say are you absolutely sure you need 100GB of media of your kid after only 1 year? That's more than the average perv has in porn on his hard drive, sheesh! :)
Treat it like a server. You need at least one 1 TB hard drive for long term growth. Then either get a blu-ray DVD drive, or a tape backup system and make redundant backups. Keep one backup copy in at a reliable friend or relative's house, a safe deposit box, at work, or some other place away from your house, keep the other copy in your house. Anything smaller than Blu-ray or tape backup will simply be too inefficient to work with given so many files. The hard drive gives you portability for the future and serve as your main file locations, and really it's the best method to store the files since you have so many, and I anticipate you'll probably have a lot more in the future. The backups are just there in case your hard drive goes on the fritz.
"All great wisdom is contained in .signature files"
Use a VCR
And bets on wether goggle will be around in 20 years?
FGD 135
store it on paper; lasts for thousands of years.
it's a matter off density and good scanners :o)
Should've used Kodachrome. Hell, even Ektachrome will last over 80 years with proper storage. I'm sure your work is important to you. The question is, will it be important to your great-grandchildren?
Oh, wait. You've already answered that...
Might want to invest in a UPS too. RAID wont save you from sudden power outage related drive crashes.
Caesar si viveret, ad remum dareris.
If you worked for me, I would fire you. RAID? Seriously? Are you just looking up words on Wikipedia and putting them in your post?
RAID is NOT a backup method, in any way, shape or form. Corruption written to one disk is written to all disks. A failed controller can ruin an array, software or hardware RAID. It is not for backup, ever. It is for high availability and/or high performance storage NOT as a means to store things indefinitely. In fact, hard drives are never for backup, at all. They can fail on spin-up and you're done, dead drive.
If you ever ask someone about backups, and the word RAID comes out of their mouth, fire them if they work for you then slap them in the face.
Tape is the ONLY real backup solution you can get your hands on, period.
IMHO, only live hard drives is archival storage - and such data needs to be backed up somewhere offsite.
Tapes, CD-ROMs, DVD-ROMs, etc., are good for backups, but not archival purposes IMHO. Even if you have a plan to read every disc after 5 years and re-write it onto something new, you can expect a certain number of failures.
Of course, for truly long term storage, baked clay tablets and incised granite are probably your best bet.
Forget Wikipedia, ask the people who spend their lives trying to figure this out.
http://www.digitalpreservation.gov/you/digitalmemories.html
http://www.archives.gov/preservation/technical/guidelines.html
http://www.archives.gov/preservation/family-archives/digitizing-photos.html
http://www.archives.gov/preservation/storage/
its my conscience!
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
In general, the best format for video storage is sealed tape, but the problem is that you frequently can't play it back.
CDs generally have a 3-5 year life expectency, longer if sealed and kept in a constant cool temperature so as to reduce flaking and minimize exposure to solar radiation.
DVDs are pretty much the same.
The best recommendation, given the nature of the shift in recording formats and players, would be to use a Hard Disk, and move the files from disk to disk when you buy new hard disks or solid state recording medium.
The main problem will be the audio format. For that I'd recommend the most recent popular (note key word: POPULAR) video format, so that it can be imported into whatever video player software exists in the future.
I should point out that, even if the needles and players may be hard to get, that LPs are actually a very long storage format, and if protected with sleeves and plastic inserts can easily last 100 years. But this only works for music, for the most part, not video.
Avoid cassettes, the tape used for those is too fragile and tends to crease.
-- Tigger warning: This post may contain tiggers! --
one robbery, fire, or natural disaster and everything's gone forever.
Well, duh, you don't leave them at home!!! Leave it at your parents' house the next time you go for Sunday Dinner.
then we're back to "what format?
Something with very little loss of information. DV if possible, or high-bitrate MPEG2. After all, 750GB drives sell for US$100 at Newegg (or at least they did a few days ago).
"I don't know, therefore Aliens" Wafflebox1
OWC Mercury Elite-AL Pro(TM) Storage Solutions is good ext drive case with Quad eSATA FireWire 800 + USB 2.0 is likey to still have at least 1 port that is still on system that will be out in 5+ years.
http://eshop.macsales.com/shop/hard-drives/external/elite-al-pro-quad
I would
1.) Keep the original tapes. (Well assuming it's DV)
2.) Burn everything to high quality archival format DVDR discs.
3.) Buy 1 hard drive and rotate it offline backups using a USB/Firewire hard drive swap device. (Keep the hard drives and media in a safe cool and dry location. Only reconnect the hard drives to the computer to update the archives. Do not leave them powered on all the time.)
4.) Keep the "originals" live on a server with redundant RAID disks and keep a nightly backup on another hard drive that's always online.
When DV tapes are obsolete, get rid of them. When the DVDRs are obsolete burn new copies to BluRay. When hard drive interfaces change replace the backup drives and online drives with new while preserving the contents of each. Repeat this process as necessary to keep the images accessible.
That should do it.
Case in point the few old VHS tapes I have of my childhood have degraded so much that they are virtually unwatchable. If I had the ability, I should have converted them long ago to a new
format. Man has not yet made a storage medium that is good indefinitely. Perhaps, instead of working on 2TB hard drives, they can start looking at a storage media that lasts for 100+ years guaranteed because it's internally redundant.
There is a company called Plasmon that makes UDO (Ultra Density Optical) media with a guaranteed life of 50 years (but will probably last 100) and a capacity of 60GB. Plasmon are a pretty useless company, and are constantly in financial trouble, but the UDO media are very good. Konica Minolta manufacture the UDO drives. Plasmon puts a badge on.
I did some research on a similar situation a couple years ago. Optical media has a rather long lifespan (in upwards of 70+ year ratings on some of the high end dvds, if I recall correctly). The question to ask yourself is will there be hardware available to read the media in the future.
One of my first ideas was 'fine, I'll just put away an optical reader'. There are two issues with that conclusion.
1. Optical readers have moving parts. Can the readers handle this much shelf life without regular use?
2. The optical reader has to plug into a computer. Will computers have the appropriate connections in the future? I could put away an entire computer, but that seemed excessive for me.
In my opinion, pick something relatively dependable (and cheap) and run with it. When the price of the media and drive(r)s to read it start going up due to new technology or availability, then it's time to convert.
I use a WOM drive (Write Only Memory). It has infinite storage cacpacity and the format never changes.
Every time you call tech support, a little kitten dies.
Flash has a data-retention about 20 years, after that you're risking data loss.
A HDD service life is about 5 years, so if you plan to have the HDD running it wouldn't last 20 years.
Keep in mind that in 20 years storage technology won't be the same. Just keep it backed up on new hard drives, or storage every few years. Make sure you have redundant copies
1. Convert digital media to metal punch cards. Include schematics on decoding. ... (Wait 1000 years?)
2. Boost payload into solar orbit
3.
4. Profit!
I recently dug up and accessed an 8 year old system to harvest some old videos.
The videos were in tact and did not suffer bit rot.
I'd suggest a parity raid system, plugging it in and firing it up at least once a week.
the likelihood of more than one disk failing at once is very small, and if one fails the system will recover the data and repopulate the drive.
VLC FOR MAC IS DYING! IF YOU DEVELOP, PLEASE SAVE IT!!
Are you for real? 100GB for a 1 year old kid? Are you filming every time he/she poops?
:-)
I am not trolling here, just curious. I have 3 kids and we don't have that much video on all three of them combined.
Are you one of those parents that will have just ONE kid and think he/she is your snowflake?
God help us if you think that way!
Seriously? Delete most of the video. Your under-one kid didn't do that much that is video worthy in his/her first year.
So, the answer is simple. Delete all the crap video you have where your kid burps or farts for the first time.
Trust me, it is not exceptional or unique if you kid does something expected of a kid in his/her age range.
Now, if you kid(s) solve the "Goldbach Conjecture", then please, film them and submit their shiat.
Best,
Another parent that is not delusional about his/her kids being "snowflakes" and not delusional about filming every bodily function of my young children!
Chisel what you want to keep onto stone tablets, or use clay tablets then encase them in a clay envelope. Then bury them in a pit in your back yard.
Steve's Computer Service, Hobbs, NM
gmail accounts
If you can tolerate a little extra effort, why not go for the gold standard in longevity? Collect your favorite videos and print them out. Printed matter is incredibly stable, if you can keep the mold and silverfish at bay. I'm no collector, but I have a number of perfectly readable books that were printed prior to 1900.
Of course, you'll still have to find a player that can recognize the file format, and typing all that back in might be daunting, but your great-grandchildren will be able to watch, and wonder who the heck all those folks are.
I'm always amazed at how many of these questions go on the premise that you're going to create the data once and then ignore it in a dustbin for 20 years.
Auditing your storage solution, whatever it is, once a year is hands-down the best thing you can do to make sure your data's around when you really want it again.
As mentioned in previous posts, those TB lacie drives are so cheap nowadays. Grab a bunch, mirror half of 'em, done. I doubt USB will somehow be unheard of twenty years down the line. I also doubt you'll somehow not see this coming and buy converter plugs long before it's a problem. It's not like you're going to be in space for twenty years, powerless to make minor tweaks to your solution over time.
Be sure and fire up older drives once every now and again. Run some scan disc utilities. Make it your spring cleaning project, and a fun way of looking back on how things have come along so far.
That said, I'm curious how solid-state will come into play as the sizes continue going up and the price keeps coming down.
a) Check integrity
b) Make a copy for redundancy or because discs are failing
Now compare:
1. Inesrt DVD 1
2. Run parity checker / Copy DVD
3. Return DVD to storage
4-600. Repeat for DVDs 2-200
or
1. Plug in external 1TB HDD (for copy, plug in other too)
2. Run partity checker / copy-paste
3. Return HDD to storage
DVDs are quite frankly incredibly small these days. Maybe if you had some huge jukebox-system but then I guess HDDs are cheaper anyway. The only downside is that you're putting your eggs in MUCH fewer baskets, but you should think of fire and other nuke-all events anyway. I certainly got tired of keeping track of my DVD backups...
Live today, because you never know what tomorrow brings
Is it just me, or do you find that besides a few photos meant to invoke memories, its better to remember something than to record it? I find If I over-indulge in 'capturing the moment', all I seem to have left was the content and I forget what it was like to -be- there.
Bye!
Man, you got really upset over that RAID suggestion. With a name like 900ftJesus, I would have expected a more kinder, gentler approach.
RAID5 does not last forever. If one drive fails, the array survives, provided you replace the failed drive before another one crashes. It sounds unlikely, but it does happen - sometimes there's another failure before you can replace the faulty drive.
RAID6 is better in that it tolerates two simultaneous failures.
Don't know why you got modded a troll.. It makes perfect sense. The data is stored via a magnetic field - these fields eventually degrade. The smaller the field, the quicker it might degrade. (all in theory as I am not a physicist)
If I were designing a HD I would not be concerned with loss of data. It would be easy to occasionally refresh the data - just like dynamic memory. The firmware could monitor disk access and simply refresh those parts of the hard drive that are not used regularly. A refresh every 12 months would not be noticed by the user. So long as the user was using the hard drive the way it was designed to be used there would be no problems.
Willy
You damn AC people have to stop!!!
mac fanboiz this and that
pc lovers all over
Linux zealots! BLAH!
So tired of this crap it has to end
FLAMERS
TROLLS!!
go back home and quit this crap
But that was funny as shit!
I finally updated my sig, but now it's lame.
Hard drives and digital disks are way old-school. The way to store video, records or anything for more than 50 years is in an archivally stable holographic disk. Check out In-Phase Holographic Storage. Some film studios are using this to archive films. Each holographic disk can store at lease 360 different holographic images. The disks come in cassettes, and even can be submerged in water and still work fine.
Every 50 years you should transfer them to a new cassette.
I'm sure my family will be able to enjoy them for generations!
-- Many men would appreciate a woman's mind more if they could fondle it
truth is that 99.9% of all things done for posterity's sake never get examined again. trash it now, save yourself the effort.
1 year old with 100GB of DVD quality footage all ready? Do you really want to no the answer? No matter what format you've stored their life in, they'll not really want to ever look at it. It's one thing if they chose to record it, store it, and share it, but by the time they turn 10 years old they'll never really view any of the content of their life. When they move out at 20 years old, they'll feel odd not having so much active in your face recording of their life. (Though at that time passive recording of their life will be everywhere.)
The easy solution is buy one of those TB external HD and just keep buying what ever the current largest HD size is whenever you need more.
As a general rule, for long term storage any technology that relies on magnetic data storage has a much shorter lifespan than say optical. Over time the strip looses its magnetic charge, causing data loss and corruption. I would go for optical storage (DVDs a big yet cost effective). They dont have near the base degradation over time that Magnetics do. A few things to keep in mind for DVD backup: -Buy Good Quality. Cheap disks tend to flake off and loose the back coating. I dont know how many movies and anime Ive lost that way. -Store them somewhere cool and dry, much like any electronic. -Be careful of wallets. Ive had several wallets and binders that cause friction on the back surface, causing scratches and flaking of the foil coating. Id go with spools. -Dont mess with them. The more you touch them, take them out to look at, etc, the more chance of them getting damaged. You also run the risk of getting dirt in the storage (wallet or spool, etc) which will cut down their life again. The recommendations of multiple backups is also a good idea.
Common Sense isn't as Common as people think...
HTH.
And tape is an abysmal archival medium.
How long have you worked in the industry?
Deleted
How is continually adding drives to your system to replace dead ones any different than continually adding CD/DVDs to your system to replace dead ones?
Even if CDs, DVD or BD last 50 years where is the antique player to play them on ? You need to select the most cost effective option available at the time, probably that is mid size ( 250 - 750 gb ) hard disks at the moment. But that is not the medium that they will be on in twenty or thirty years.
The important thing is the process by which you keep them, not the medium they are on. Make two copies and locate them as far apart as you resonably can, home and work or a relatives house, and probably eventually the internet to guard against flood fire and theft.
Test that they are readable once a year, or more often if you are concerned and the most important part, copy them onto the the most cost effective media again after a few years, make sure the media you have them on is always current and in common use.
In twenty years the space they occupy on the media will seem ridiculously small compared to the relative large size they appear to be now.
A lesser problem is the encoding format ( mpeg 2). My guess is that there will always be software to read legacy formats but it is something else to keep in mind. Try to change formats as little as possible, jpeg and mpeg 2 are lossy formats and converting them to other lossy formats will eventually lead to lower quality images.
I write all my important files to an external 200 G Drive and then offsite it.
I mount an old drive in a USB enclosure
copy the files to it.
remove the drive and use the factory packaging.
and store it at someone else's house.
Offsite, cheap, fast.
Because I do every 6 months or so, usually with a drive that I'm replacing because its too small the cost is minimal. The "archival" copy is the one on my desktop, 20 years from now, that 200G of video will take up just a small fraction of the 400T drive that I use to run Microsoft's Ultimate Galaxy OS.
I don't worry about disk/filesystem formats because if my desktop is always the "master" copy, and if it fails, it could always read the last few copies I made.
burn your videos to DVD-Rs. Take at least two different brands with gold reflector and make two copies. Put DVD's in lint free paper envelopes. Put envelops in tight sealing case with sikagel to take up moisture. Put box into deep freeze. Wait 20 years. I bet Video DVD's will be still readable by the optical drives available then. Take vinyl records or Schellacks: you can still get players. Take the CD - you can still play old CDs. Same will happen to a mass media such as DVD!
Can you point us to one of your vlogs? What do you talk about and where do you publish them?
I think Winamp supports OGG so I should be able to see them.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
Get a Drobo and use it as a backup disk.
Even better: get two and use one as primary storage and another as backup.
you need a clean room and a couple of grand worth of a team of specialists time to forensically recover the data, and something might be eternally lost
raid5 just requires a drive swap
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Buck up for a video tape backup system if the data is really that important.
I was under the impression that unspun drives tend to deteriorate relatively quickly
Quickly? No.
- the heads clashing with the platter or some such nonsense.
Head crashes have been a non-issue since the late 1980s.
The real problem is the lubricant in the tiny motors. It can get gummy, and then the read arm won't move.
Just spin them up once a month and you're fine, from what I've heard.
I'd say bi-annually.
"I don't know, therefore Aliens" Wafflebox1
What about storing online, let them worry about sifting through upgrades...
http://www.webhostingpad.com ... unlimited bandwidth and unlimited disc space, although you have to pay the 2 year fee upfront... and people have access to your files through interwebs, but other than that there has to be a decent online solution?
Dapipminmonkey
... AFAIK. That would mean anything from solid-state hard drives to USB thumbdrives to micro-SD cards would be a good bet.
:)
However, I think they are also manufactured quite cheaply and more susceptible to sudden physical failure due to defects than are hard drives or tapes. This is significantly lessened if they're kept somewhere secure like a safe deposit box rather than somewhere dangerous like your pocket, but still a risk to keep in mind. So, be sure to make at least one redundant backup of each stick, and then keep the redundant backups in a separate secure location. I don't think the particular interface matters so much after that, since you'll probably want to upgrade it when that type begins to be phased out, regardless of what it is (USB, SD card, solid-state e-PATA, etc). Don't forget to store a reader/player with the sticks, if possible. A player that has a screen and/or the capability to output to plain RCA A/V would probably be even better, IMO.
-=[You cannot consistently judge this statement to be true.]=-
It's never going to be cheap to maintain digital data. Unless you can afford a raid storage array, I'd just make sure I had a couple of copies on HDD (one offsite, of course). I put one HDD (500GB now) in my safe deposit box, and swap it with the up to date drive every month. I keep one copy of my data on my desktop, and backup on another box that I use as a pure backup server (no NAS, you just have to backup the nas).
When something comes along that will replace DVD-R, use that. Is BluRay enough of a leap forward? I'm not sure, a 1TB writable disk that lasts 20 years sure would be nice.
In any case, I think your DVDs should be your last line of defence. I don't trust them to not rot.
Salut,
Jacques
Best thing to do is to put it on something general that doesn't expire quickly . For instance , a flash disk ( for 100 GB could be a problem ) . Technology doesn't disappear that quickly, just keep it in mind . I think you are save with changing media every 10 years . Just do regular checks
Slipping shoelaces ?
You archive your information in some open, standardised format to an immediately accessible system, like an array of disks. That is your archive, it can be easily and automatically moved to new systems as technology changes. Then you back it up to prevent short term loss of the archive in the case of robbery, fire etc. The backups are not meant to be the archive. The format of the backup can then be the backup system flavour of the month.
Backups are NOT archives and archives are NOT backups. The requirements are entirely different.
Deleted
Seriously, when they turn 21, is your kid going to sit through around 20 hours of unedited video of when they were 1 year old? (I'm assuming here you've archived DV video, if its already compressed to DVD format, or god forbid MP4, then it'll be even more). Plus the video you shoot over the next 20 years of their life? Edit it down to a disk or two of the highlights and worry about keeping that in whatever formats are still readable over his or her lifetime.
Dude. I'm pretty sure I speak for most of the human race when I say that we don't really want to be subjected to terabytes of video of your progeny. At 1 year old, the kid's done NOTHING much worth talking about. Let me know when he's potty-trained or something. Sheesh.
I've very recently had to consider this
question for a friend who wants to archive
the family genealogy research data, along
with photos and scans of newspapers, etc.
Their design goal was 100+ years and
maximum possible compatibility.
Use the best possible media. The problems
with optical disks are:
- dyes fade
- material of disk itself decays
- reflective layer oxidizes
- reflective layer too easily scratched
Check this media out:
http://www.mam-a.com/
Use the "Gold" version of anything they have.
Also, single layer DVD+R is the best format
for longevity at this time, except for CDs,
which are way too small for you. The website
goes into details about why. (Hurry up HD
formats!)
After getting the best possible media, it's
storage: dark, cool, dry place. Regular consumer
CDs/DVD+Rs might do pretty good stored in a
light tight box with some desiccant packets in a
cool place.
Finally - the obvious: don't scratch!
Build a simple storage array with RAID from a barbones PC, your favorite Linux distro, configured for fault-tolerant RAID.
If you're not a Linux person, that *IS* complicated by default.
Not that it would really be less complicated with Windows, but only a Linux person wouldn't recognize the inherent complication of RAID.
paintball
Seriously - the vhs home movies my dad made in the early 80's (back when this bleeding edge stuff) still works.
Or what about digital vhs?
Use online storage/backup such as Carbonite and let them worry about it! I think they charge about $50/year...it's well worth the cost for unlimited storage for all your data.
This is a fundamental issue in the long term preservation of digital multimedia materials in archives. I am speaking from the perspective of the field of archives and the approaches being developed for digital preservation in these contexts, but generally the goals are the same as any digital storage project:
1. Standard, non-proprietary formats:
For video a bit tougher than audio, where the Broadcast WAV File has become the preservation standard format for audio, and still images where TIF reigns--there really is not video equivalent. For native DV, you'd ideally want to store it uncompressed--which isn't going to happen for most of us at home. JPEG2000 holds some promise for lossless compression, but isn't widespread. The goal here is to compress as little as possible and store in a standard (ideally non-proprietary) file format. That stated I'm aware of people using various kinds of Quicktime files for this purpose, although it makes me shiver.
2. Redundancy:
More than one HDD in more than one place. RAID and LTO (or some other datatape) if possible. DVD-ROM, DVD-Video. In an archival preservation context this is achieved through geographically separated servers and datatape back up, LOCKSS, etc.--ideally that is. I still have a lot of HDDs with audio content sitting on shelves.
3. Migration:
Migrate from storage hardware/media before its reached the expected end of its physical lifespan, and before hardware support for access has vanished. Batch convert files to new standard formats before the old formats are not supported any longer.
If you know how to make them go, checksums are a good plan too.
andy
The best place to store your memories is in your head :)
That is, until dementia kicks in.
The best place to store all of your memories is in your head. until dementia kicks in.
the best way to store all of your memories is in your head, until that nasty thing called dementia kicks in.
Well , you could hire webspace to store the files. A good contract would ensure regular backups and redundancy . That way , you don't have to worry about it , but it's probably the most expensive option. And off course , in 20 years , that company might go bankrupt , leaving you with nothing
Slipping shoelaces ?
I don't imagine some your former "Cow Orker" could be trusted for any knowledge not directly related to the orking of cows, and any necessary knowledge for maintaining a cow orking facility.
At any rate, HDD's hold data magnetically. When a HDD powers down the heads park automatically and there is no way for the HDD to lose data unless you screw with the platters in some other way. Large magnets and powerful electromagnetic fields might do it. Typical HDD lifespans are from 600,000-1500000 hours of operation.
This sig isn't original enough, it's time to come up with something witty...
Has anyone mentioned Key drives or SSDs yet? Technically those items are limited by the amount of writes you can do to them, which is sometimes an obscene number. So far, the key drive I had for over 5 years is still working, even though its been run over by a car.
Thoughts?
A morning without coffee is like something without something else.
A few good pictures and a handful of short videos become "precious memories".
A slag heap of hundreds of hours of raw material become a burden that someone will eventually stop maintaining because it is such a chore.
Lots of pictures are less of a problem than video both because they are smaller, but because you can look at them faster to see if there are any worth copying, printing etc.
Send copies of your "best of" to friends and family so that you have off-site storage should your house burn down.
This issue is a bit more complicated than you think.
One thing I haven't seen mentioned here yet is to make sure that you have more than one copy and that you keep the copies separate. In other words, if your house burns down then it probably won't matter how the data is stored. You might want to put a backup on DVD's and put them in a safe deposit box or some other safe location.
with your 100G of pornographic digital video and pornographic photos?!?!
Yes you are in trouble because on a sysadmin list for the local universities, we get one cry for help every year. The template for that cry is:
Help!
We have lots of tape of type xyz, and our tape drive is broken. Do you guys have a tape drive that can read this.
Raid is good for keeping the data alive, it is a backup in the sense of avoiding failure of devices causing major data loss. Raid 10 or a Mirror are your best bets for redundancy. However, a RAID is not going to be a preventative measure against other forms of data corruption, virus, batch file run amuck, accidentally deleting a folder. So you need a separate storage medium, you can pick your favorite, everyone has ones that work well for them. There are several alternatives depending on the size of data you are backing up. For me, I use online storage through a respected vendor MOZY for my home use, which kicks the ass of any other back up medium I have found, including tape. Tape is a dying backup medium in my opinion. I think what most people will be using within the decade are personalized net storage solutions.
BTW if you worked for me you would be fired for being an ass and pretending to know what you are talking about.
CS: It is all sink or swim...oh and did I mention there are sharks in that water?
Here's the real solution:
Store it on your computer in RAID. When a disk in your RAID fails, upgrade it. When you get a new computer, copy the data over.
Since you'll be using these disks on a regular basis, you'll know as soon as one fails and can replace it.
paintball
People like to debate the 'best' format for long term storage. The answer, when talking about digital data, is copies, copies, copies. No media lasts forever. The advantage of digital media is that, as long as it's still readable, you can make bit-perfect copies.
Store the data on multiple media, and plan to make new copies every 5 or 10 years. Give some copies to relatives/friends to hold on to (offsite storage :) ).
It can be a little daunting to try to keep track of lots of different copies, and ensure that every file you want to keep is re-copied on schedule, no doubt, so good quality media is still important, but the best defense against digital data loss is having multiple copies, and it's better to have multiple copies on reasonably priced media, than a single very expensive, very-high quality media, because even the most expensive, high quality media still have a chance to fail, or to be damaged (scratches, abrasions, or cracks in optical media, exposure to strong electromagnetic fields for magnetic media, etc).
My wife and I have taken thousands of digital photos and videos since 2002, but the cold, hard truth is that the vast majority of them just aren't very good. I suspect the OP is in the same boat. The solution is to take the time to identify the best images and videos. This should result in a relatively compact archive only a few hundred digital photos and videos. This "best of" collection will be much easier to duplicate in different format and making physical prints on archival quality paper won't put a huge dent in your pocketbook. Remember that just because you can take 2000 photos of your spittle-covered toddler doesn't mean that you need to keep them all.
Chances are pretty good that your great grandchildren aren't going to give a damn about inheriting a massive archive of pictures and videos starring weirdly dressed dead relatives they've never met, and if it's physically large it stands a good chance of ending up in a garden shed or unprotected lunar storage pod. If you condense your family album into something more manageable, it will be more accessible and enjoyable for all - now and in the future.
... and email the files to yourself. We'll see what "unlimited" really means.
I kid, I kid. I'm guessing that would violate the terms, and even with a thick pipe at home, it would take a long time to upload that much data.
RAID and DVD are good suggestions as many have pointed out. Creating a second set of DVDs to store offsite is probably good policy too.
100 GB of DVD-quality video sounds like a lot more than it really is. DVD has a max bit-rate of 10.08 MBps. 100 GB = 800 Gb = 800000 Mb. 800000 Mb / 10.08 Mbps = 79365 s = 22 h, 2 m, 45 s. If we assume the bit-rate is half the max, that gives us just over 44 hours of video. The kid is 1 year old; 44 hours over 1 year is an average of about 7 minutes a day. That's about on par with recording a single hour every weekend, something a new father could probably hit easily. If we take it a step further and assume 1/4 max bit-rate, that's still only 88 hours or 14-15 minutes a day - under 2 hours a week; still seems plausible to me, though I can't imagine more than a handful of that is actually worth archiving.
Your experience is very interesting because mine is the opposite. I make (at least) quarterly backups of my data and have since mid 1993 (CDs since 98, floppy before that). This spring I got bitten by the curisoity bug and started going through all my old backups looking for forgotten and interesting things. Every CD older than two years had at least one unrecovereable read error. Every CD older than five years, except for one, was completely unreadable. Between two and five years the number of read errors grew with many files being lost and several CDs being unusable. The 3-1/2 floppies were all 100% readable.
In that time period I've been through probably a dozen CD burners, both expensive varieties and cheap ones, and I've used at least as many brands of media. All the CDs have been kept stored in dark, dry, clean places. I tried reading the "unreadable" CDs on multiple computers and met some limited success accessing additional data. I didn't try any recovery software.
Fortunately for me I kept most of these backups out of habit and I didn't really care about much of the older ones outside of curiosity.
I smurf everything and everything I smurf is perfect.
I noticed that the more "memories" (photos, video) you keep, the less valuable they become.
Its like in those movies, you know the poor kid has a grainy, beat-up picture of his mother that he didnt see for the last 10 years. The value of the picture is huge.
Now if the kid had a couple hard drives, I'm not sure I would see values.
Nowadays at wedding you might get 1000s of digital pictures, but what people really want is an album with maybe 20 or 40 of them.
Nothing more boring that people showing you their 500 photos of their vacations. Please select one or two dozens and do some editing work...
If you do happen to survive the nuclear apocalypse to come, your data will be erased by the vast EMP wave attacks that China will drop on our cities to mitigate any surviving technology.
And then the baby jebsus returns and we all go to the hebbens.
miniDV tape is still your most economical for retaining raw, unedited content. It may be more practical to archive the edited content on harddrives, but stick to the miniDV format if you wish to preserve the original quality.
As a digital medium, you can bounce it as many times as you want (including back to tape) to trade off stability and volume archival needs.
I'll admit, however, that I'm not up to speed on the latest HD formats, if that applies to your needs.
I have CDs from back in 1998, when I've coaxed my boss to buy a 4x TraxData burner.
They still mount better and copy and open easier than some printed ones I got with various magazines over the years.
Aaah... but back then - a writable CD was about 5$ apiece and they only did up to 4x as did the writers.
I've also had (and still have) a large number of silverbacks burned by various people over the years.
Some of them were unreadable or had problems mounting the moment I've put them in the drive.
Just because it says somewhere on the sticker that it is 52X compatible or capable - it does not mean that it is.
That rule works for both disks and drives.
Also, whether it is cutting costs in manufacturing, packaging (I've found fingerprints on some "fresh" disks), quality control, transport or just plain lying about the performance - the ultra-cheap ones are cheap for a reason.
Mit der Dummheit kämpfen Götter selbst vergebens
Really? I have a 75meg HDD from a 386 that I would love the data recovered from. It's been powered off for 15 years, so there is NO way that the data is gone! I'll send it your way! kthanksbi!!
Copyright 2010. All rights reserved. This comment may not be copied in any way including, but not limited to caching.
I've been keeping my data backed up to several different hard-drives using the free SyncToy software from MS. When technology marches on, I'll have an up-to-date copy of my data which can be transferred to from a single device to another medium in one go.
Mostly this works very well, but I've discovered that across tens of GB of data copy errors begin to creep in. It's only a bit here and there, but if you're unlucky it can make the file unreadable.
So, Slashdot: there's plenty of software out there for making backups, but how about software which checks them?
Besides, sometimes the controller fails and all the 5 drives get corrupted, and then (as in our exchange server in 2004) the dell service guy comes in and says he's seen this before.
Go at least raid 10 (5+5) and if you can, make offsite archive to data storage service (rsync or some such)
In any case, offsite from a company, and don't go cheap on the service or the service will go cheap on you.
Don't want to pay? You can't afford long-term reliability.
"Piter, too, is dead."
Migration, migration, BEES!
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
Memory. Human, neurological, living and breathing memory. Get out from behind the damned camera and play with your son, and talk about him with your family and your friends. Tell stories. Add the memory of him to the persistent memory that is community storytelling - in 15 years, somebody will remember something that you've forgotten and remind you, and you'll get to relive the whole thing in your imagination all over again, clear as life.
Cameras are overrated. Storytelling isn't.
Live today, because you never know what tomorrow brings
That's because you can't fucking spell it, it's called Taiko Yuden.
No offence, though, and have lots of fun with your correctly-spelled discs.
A horse can't be sick, you know, even if he wants to.
The best archive longevity can only be had from archive rated disks and the only disks rated for archive use (part of the disk spec) are DVD-RAM disks. Note that DVD-RAM disks are engineered with a minimum archive period of 20+ years while both CD/DVD do not even have a minimum archive period in the specification. Simply put, most if not all new burners can actually handle DVD-RAM disks, which although expensive are worth the money as they are also rewritable. Write Speed is not important (2.4x) as the usage patterns for DVD-RAM is archive instead of day-to-day read.
As part of an archive strategy, you will need to refresh the archive about every 5 years, which means checking the integrity of all disks in the archive on a regular basis. You will also need to construct the archive in such a manner to provide redundancy of files even if it's as simple as creating two copies of each archive.
Mod me up/Mod me down: I wont frown as I've no crown
Use cheap iSCSI units to build a virtual RAID-1 or RAID-5 spread across your house, garden box, garage.
If your house is not UTP cat. 6 wired, use power line network adapters. It's not a jumbo framed GbE wiring, but for a media library it should be enough.
Cheap iSCSI units are manufactured by Buffalo Technology:
http://www.buffalotech.com/products/network-storage/terastation/terastation-pro-ii-iscsi-storage-system/
Powerline network adapters from Devolo:
http://www.devolo.com/co_EN/index.html
I'm amazed that no one mentioned it. Just get 16gb usb flash disks.
It has theoretically unlimited life for archiving. The only time it deteriorate is when you continuously write/erase it.
Print out pictures and store. Your eyes are never obsoleted by upgrades in technologies. Dig out pictures of your great-grandparents for help in this ancient method of data storage.
print it on archival paper with 2D barcodes.
if you were to use 0.1mm size pixels then an 8x10" page can store:
80*(25.4**2)*100 = 5161280 bits = 5Mbits/page
or 2.5Gb/ 500page ream.
so you just need 400 reams per 100GBytes and that includes some room for error correcting codes.
Given it will last 20 years, the rather large volume may be worth it.
Some drink at the fountain of knowledge. Others just gargle.
Have you seen Beowulf? In 10-20 years you'll be able to reproduce your happy memories without the need for your living kin (or any storage media.) And you can make yourself better looking, too.
I got many cassette tapes and 5.25" floppy disks that are still doing great after 25 years. ...Though you would have to convert your pictures to Doodle or Koala Paint to properly store them and save space.
"Enjoy what you're doing! If it becomes drudgery, you're doing it wrong!" - Jim Butterfield
I have a fujitsu 3.2 GB HDD from 1997 and it still reads and writes alright, I suggest you invest in a server-grade SATA/SCSI hdd solution and you should be good for about 10 years, which by then the storage market will most likley be flooded with SSD's and who knows what else?
Because a genuine evil laugh beats a robotic voice any day.
A horse can't be sick, you know, even if he wants to.
Hell, if it's a portable hard drive, just leave it attached to a non-network connected POS machine with a USB port and use cron to ask for an md5 sum or something every few days.
I remember reading somewhere that in unspun drives, the head can actually bond to the surface of the platter, slightly. Hence, apparently why knocking the drive lightly can sometimes revive a dead looking drive. Although keeping it spinning regularly sounds like a better idea...
-- All your booze are belong to us.
WW900FTJD?
I have 60 GB of photos from when I first started taking digital photos back in 2001. I keep three sets on three hard drives: 1) In my computer (obviously) 2) One portable 120 GB I keep at home 3) One portable 320 GB I keep at work Once a month, I back up from my main computer to the two portables. As they fill up, I'll replace them. If one crashes out. I'll back up from the others. Simple, cheap, and effective.
I can attest to the simultaneous failure weakness of RAID 5. I've only dealt with two different arrays... but they've both suffered simultaneous failure of 2+ drives in the array. It sucks.
Luckily, the second time, I had /good/ backups to another server, and was able to recover. The first time, we had to use data recovery specialists.
Matthew Walker
http://www.tweeterdiet.com/ - My Diet Tracking Tool
actually I would say this is one of the cheapest solutions, if you check my other posts, I am a mozy fanboy. $50 a year for a total of $1000 for 20 years. Really not to shabby when you figure you get unlimited storage basically.
CS: It is all sink or swim...oh and did I mention there are sharks in that water?
fuLLproof way:
write down (or type really small) all data in binary form.
store in a vacuum container and in a dark cool spot.
-- Anecdote.
I had a newbie along with me; The kid had an MCSE and was also going for an MCSA.
Our company policy was to bring plenty of spares along with us; We had no appeasement engineers.
I replaced a drive in a hot-plug array that SMART had alerted was destablising; That part went fine.
A second one went amber while we were wrapping up, and the kid said "Lucky we brought spares! I'll get it!" *YANK*
I almost feel sorry for him, to this day.
Maybe you can find someone that will transfer it to film?
For my single-enclosure SAN storage unit, I have RAID 5 with two hot spares. Only because the EMC AX4-5 doesn't do RAID6.
Check out my sysadmin blog!
Who cares about what format you want to back up to, more important, what CODEC will still be around in 20 years to be able to play the footage you shoot today?
Already I see my high bitrate, frame accurate miniDV camera becoming obsolete in the consumer arena in favour for highly compressed, not frame accurate, hard drive based cameras using MPEG4 video, moving to (consumer) HiDef cameras and they also use high compression MPEG4 and an awful audio codec instead of PCM audio (like for DV). What are the odds of any of these formats being playable in 20 years time, or just transcodeable?
Take Nobody's Word For It.
For long term storage, rent a film recorder and write the data out to the original optical media. Assuming the media isn't destroyed, it will still be readable/recoverable 50-60 years from now. This isn't exactly cheap, but its probably the least work long term. All it needs it proper storage, no pesky format conversions every 5-10 years, and as long as the Mark I eyeball remains in use, the images will be recoverable.
You probably don't really care about uninterrupted access, so instead, have one copy on an internal disk and backup to a drive in external enclosure. This allows you extra options such as being able to store keep the backup in another location (desk at work?) and merely bring it home to update the backup once per week. In case of a fire, you might be able to grab the external enclosure and take it to safety.
The real "Libtards" are the Libertarians!
Exactly right. Optical media will be gone soon enough. Personally I copy from my dvd camcorder to hard disk using a program I wrote:
http://www.pixelbeat.org/programs/dvd-vr/
The fact that I had to reverse engineer the DVD-VR format emphasizes that the data format is at least as important for long term access as the media.
After extracting the video data from the DVD I then reencode and compress it further using the open dirac codec from the BBC which is specifically designed for this purpose.
AMPAS recently had a report called "The Digital Dilemma", which the NY Times wrote about:
If not operated occasionally, a hard drive will freeze up in as little as two years. Similarly, DVDs tend to degrade: according to the report, only half of a collection of disks can be expected to last for 15 years...
What are film archives doing? Where possible, studios are making long-lasting, non-fade B&W pan separation YCM polyester negative film backups, even when the film is mostly or totally "born digital". Then you put it under a mountain somewhere.
Government video archives worldwide are moving to LTO tape, typically using JPEG 2000 video encoding, with the recognition that every few years they will have to migrate their tapes up a generation of LTO. I suspect there may be a move from lossy JPEG 2000 to lossless JPEG 2000 and eventually uncompressed video as tape speeds and capacities ramp up.
RAID5 is beaten by:
A) Disaster: Fire. Flood. Earthquake. /" can happen to the best of us.
B) Carless mistake or Idiocy: "rm -r
Nothing but an offsite backup will do.
I'd burn a copy to good quality DVD's in an offsite location (an office drawer at work does the trick for me) as you accumulate data. Replace the media every 5 years, as new media come out.
On a Rainbow Versatile Disc!
Up to 450 GB on a simple disc of paper printed in colourful triangles, circles and squares! You can't beat that!
You just got troll'd!
:)
Really want to keep it?
Backup to an external drive. Like someone said not cutting edge.
Backup to DVD. Quality is up to you and repeat this one as often as possible.
Backup to a tape drive.
Then take 2 of the above off-site.
I use old DDS4 tape drives i got off ebay to match the unit in the office server. The 20G tape drives and a SCSI card are usually under $100. Windows backup is probably good enough for a simply backup. I store the office tapes at home in exchange for keeping my tapes at the office :)
Good idea to store a reader off-site with media for worst case scenarios.
Just watch for changes in O/S support. Like the old Travan tapes not longer (easily, i did make it work) supported under WinXP
although i usually use DVD-R for everyday use. Always remember to use DVD+R for archiving. +R format has some error correction built in that will allow you to read it even if some degradation has occurred. Its not a guaranty but its a extra level of protection.
Shut up, Twitter.
Remember, open source is free as in speech, not free as in bear.
... seriously, there are many sites you can use as permanent stores, such as GMAIL (Gmail drive - http://www.allscoop.com/tools/gmail/) and other tools, just look around!
Why not take advantage of it?
I found this informative:
http://adterrasperaspera.com/blog/2006/10/30/how-to-choose-cddvd-archival-media
about 20 miles & 4 hops apart.
3 TB on one at home, 2TB on the one at work
I use the extra on the one at home for my private video collection.
so raid5X2
they are kept in constant sync with vice versa pro
(server at both ends) over two comcast commercial connections that are as permanently connected over vpn as I can
they also have the infrant snapshot running
I also have a NICE workgroup scanner at work, and I scan all my crappage (taxes, bills, statements, bills, legal notices, liens, receipts, bills directly into pdf- and that gets mirrored as well.
I can use windows search to look inside the PDF's for anything I need- and my filing system can be a mess.
I atill open photo files randomly all the time knock wood- no issues yet.
once a year, I do back the PHOTOS, not video- up to a good quality dvdr
in case of a fire- the reccomendation to my wife is- grab the kids, get yourself, the turtle, and THAT BOX RIGHT THERE, out the door- and in that order
(I broke the clip off the ethernet cable on purpose)
every day http://en.wikipedia.org/wiki/Special:Random
1) Take still photographs on 35mm film (or medium format.)
2) Store the negatives in a fireproof box
I found a negative on a family farm that is approximately 80 years old. Nothing special was done to store it, and the prints look glourious.
I doubt you'll be able to say the same about your hard drive 80 years from now (although I have some files that are 20 years old.)
Skot Nelson music is my saviour / i was maimed by rock and roll
Hold onto the tapes. I'm assuming this is DV, yes? If DV, simply hold onto the tapes. They're less than a buck, hold 13G (1 hour of DV), and your video is already on them. You'll be able to read them for at least 10 years (they'll last for 20 but DVcams might not be around then).
Check out Drobo, too.. it automatically moves data around if a drive fails. When a red light comes on, just pop another drive in: http://www.drobo.com/
The absolute best thing you can do to preserve the important videos is to delete all the unimportant junk. I'm being absolutely dead serious. The vast majority of what you shoot is going to be of no interest to ANYONE in ten years, not you, not your parents, not your kid. And no one is going to want to sit through 180 hours of footage to find the good stuff. Believe me, I have parents with an attic stuffed full of "precious memories" in boxes. They can't find anything important and no one has the energy or desire to go through it, and when they die, all that stuff is going in the trash. There's no way my sister and I are going to sift through it all. The only stuff that's being saved are the 2 or 3 photo albums they put together.
By all means, keep on shooting as much as you want, and keep on sending DVDs to your parents every week, but at the end every year, you should go through the gigs of videos you've shot and put together 2 or 3 hours of the absolute best footage. Burn the best of onto DVDs and slip 'em in your safe deposit box, and send copies to your parents (they'll LOVE it). Replace the backup DVDs with fresh copies every few years. Keep the raw footage on RAID at home, maybe keep a HDD backup of "the best" at a friend's house. But don't try to save everything. I guarantee that no one will care about 99.999% of it.
We don't use the U word here.
Don't rely to any media standard to last more than a few years, especially DVD's: I have thrown away dozens of them, well stored (always covered, kept vertical, no sun, dry place, etc), then decided to stop using them.
Today in my opinion the only way to ensure one can keep a great amount of data for many years is by keeping them redundant (RAID, rsync backups, etc) then moving them to a newer media every few years, say 2-5. I have at least a half TB of data I keep this way.
Yeah, I fell for the misspellings of others there, and 'll have to suck it in..
Thanks, by the way.
A horse can't be sick, you know, even if he wants to.
Storage that can last 20 years is only a small part of the problem.
Think back, if you can, to what storage media and file types were being used 20 years ago. Now, how many computers today are capable of loading data that was recorded on a cassette tape machine? How about a 5.25" floppy disk? Are there programs available now that can read the data files from a Vic20, C64, TRS80?
All the suggestions I have seen here so far are just about keeping the data alive for 20 years. Compatibility is the main problem. You would not only have to keep re-storing the data in a compatible way, but also keep converting the data to the latest fad format before the previous format become unusable.
Multi Layer backup practiced here
:-) :-) :-)
1. Client PC contains working copy
2. Home server in the closet (Running RAID 1 & RAID 5 volumes) (~6 feet away
- pc data files are backed up to a working backup directory on this server. This is the only writable directory on the server.
- working backup is synced with the master backup every few hours after manually confirming the changes. The master backup is available as read only share.
3. Home server is backed up every few days to portable harddrive stored offsite (~6 miles away
4. Every 4-6 months all is burned to DVDs and/or tapes and stored abroad (~6000 miles away
Film 20+
Encoded and printed on acid free paper and India ink and with the ability to use a standard scanner or digital camera to convert it back. 100+
Baudot encoded paper tape just for nostalgia's sake.
Harddrives. 5+
VCR tape (lasts longer than CD) 5+ with degradation
CD 5-
DVD 5--
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty
this does not protect against theft/fire/flood so it is good to maybe get a third that you take to your office and bring home once a month and do rsync.
The other option is to let someone else worry about it and put your stuff online somewhere. A pro flickr account is $25 a year, and let's you put all your photos online and videos up to 90 sec.
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
When you think back to poets like Emily Dickenson, where most of her poems where discovered after her death, I would say the paper trail is the way to go. I love digital media, but it just does not equate. For instance if Emily had poems on a HDD would they have been found??? In the future, digital media may find a solution, but for now, nothing beats good old fashioned paper.
'sig' deleted due to the stupidity of it's 'nature'
I would build 100GB server using "windows xp pro" or "windows 2000 pro" and store all your photos there.
Then sign-up for mozy home $4.95 unlimited backup
http://www.mozy.com
It should take about half - year to catch up, but you'll be good after that.
This might seem expensive, but get yourself two of those USB/Firewire Western Digital 1TB hard drives and store the video on both of them, so you have a backup in case one dies. Each time the 1TB fills up, get two more drives at a similar price. They'll probably be a larger capacity by then. A drive like this goes for under two hundred bucks nowadays, but I think this is cheap given what you want to accomplish, and I'll explain:
A double layer DVD has a bit under 8.75443220139 gigs of usable storage, IIRC. And a 1 TB drive has 931.32257461548 actual gigs of storage, if a gig is 1024*1024*1024 bytes, not 1000*1000*1000 bytes like the hard drive manufacturers want you to believe. Meaning you can stick the equivalent of 106 double-layer DVDs on one of these 1TB drives. 106 DL DVDs will cost you somewhere in the neighborhood $120 to $135. So you'd spend $70 or so less if you bought 106 DVDs rather than one hard drive, and here I'm telling you to buy two hard drives and thereby spend $140 more than for 106 DVDs. Why?
First, the time it will take you to burn all this crap onto DVD. I think it takes a DVD burner 45 minutes to burn a DL DVD. I haven't done it in a while, so let's just say that's correct. Accordingly, it will take you nearly 80 hours to burn all these damn discs, and that doesn't include the time it takes to put in a disc, select the files, burn them, verify the burn, since you want to be sure, and this will double the burn time.
Second, many of the discs will fail to burn properly, so you'll have to redo them. I've experienced as high as 1 out of every 2 double layer DVDs failing to burn properly (this may be due to the discs or to the computer, or to the software, or a combination of any of them). This increases the time mentioned above and it also increases the price.
Third, since you NEED data redundancy, you need to burn two copies of each. Double the cost and time requirement.
Fourth, you don't want to split files across DVDs so you'll utilize less than all of the available storage capacity of each disc. Increase the number of discs you have to buy, the cost, and the time accordingly.
See, all these reasons show why it is cheaper to just get a bunch of external hard drives, and to duplicate everything. Even if some catastrophe happens to both drives, you can send it to those folks who recovered data from the Space Shuttle Columbia hard drive, which fell out of the sky along with all the poor souls on board. But that's not likely to happen.
McCain/Palin '08. Now THAT's hope and change!
Outsourcing is the key to your problem. Keep a copy in a couple of hard disks until your kid turns 5. Then show him/her how to use the computer and how to create backups. When the kid is 9, he/she will be such a nerd that it'll make us all proud of you both.
By that time he/she will create the backups with no problems at all. You'll just have to ask your kid for a copy in the portable cheap format of that time every 3-4 years.
Expensive but has saved my data from local failures a couple of times already.
WHY USE RAID??? That's just overkill. Keep two (maybe three better) backups and in a minimum of two locations. Use something like BackupPC (or rsync) to keep checking/refreshing them. If one backup dies, replace it.
The point of RAID is quick recovery and erad efficiency which you don't need with a personal backup.
O RLY?
Regards,
Werner Heisenberg
I don't think you would be interested in videos of my kids.
Intellectual property was the desert property of the twenth century.
Punch cards last forever! Just make sure to number them.
Only wimps use optical media, _real_ men just upload their important stuff on ftp and let the rest of the world mirror it.
Put it on a personal website and point archive.org at it.DVDs are nearly obsolete. I'm running MythTV, so pretty soon I would tend to view home videos in the form of video files anyway rather than physical discs (although I still burn DVDs, but only bother editing about one DVD per year anyway. We send them to family at Christmas.)
A couple months ago I got a terabyte drive, re-ripped all my digital 8mm tapes (30 or so) to files, and then gave the camera and tapes to my sister-in-law who had a baby (they need a camcorder more than we do at this stage). DV files take more space than MPEG but I didn't want to lose whatever additional quality the originals might have had. Even so, there is enough space left on the drive for the edited-for-DVD versions of all those home movies, various other media files, compressed tar backups of my other systems, etc. I then backed it up again onto a 750G external drive and store that one off-site. (And I would recommend to anyone to do the same... don't trust a single hard drive without other backups. At the very least have two systems and back them up mutually over the network.)
My next step: get an HD camera. Then I wonder if 1TB will still be enough. :-) But in a year or two I may just need an additional terabyte, that's all, and we'll see how much more the typical capacity goes up during that time.
What I'm waiting for is an SLR that can also shoot full HD video (1920x1080 or so). I figure that's a year or two out. 1280x720 video mode already exists on some digital cameras. Meanwhile I can shoot 640x480 on my existing point-and-shoot, which is nearly as good as the camcorder (if only it had better sound).
Ah, no I would not. I thought they would be about something else.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
I wonder also about the format. While i suppose it is now reasonable to expect standard formats like TIP or JPEG or BMP to survive and be readable in twn or more years, I would not be so sue about video formats.
I still have material from the early 90's on floppies (that I can't read because they are written using a different OS) - but even if I could read the disks, I couldn't access the content because it requires software that I cant access any more.
Finally, copying all this is easy - editing it and selecting the best is even more time consuming.
Stamp on you?
WW900FTJD?
Obviously he would use a tape.nice that you still have that stuff. seriously, this x.264 format was pretty cutting edge at the beginning of the millenium, but we'd have to find some _very_ old linux cds and install them on vintage hardware just to be able to play them! you reckon 's worth the effort ?
...but a safe bet would be DDS-3 or DLT tapes wrapped in anti-ESD bags and locked inside a fireproof safe. All this in a room deep within some mountain. This method will also be good for a hundred years, but may be considered overkill.
Above all make sure that the camera doesn't come between you and your kid ALL the time. It's more important to love him than to document him.
And btw remote online backups are the only thing that really makes sense.
Newegg has Quantum DLT SATA drives (160GB native capacity, 35 GB/h throughput) for about $700, so it won't break the bank to get proven multi-decade shelf-life media of reasonably size and speed for a 100GB dataset.
Personally I'd try to go with LTO if possible. It's got a larger portion of the pie, so find drives will probably be less of a problem in the future IMHO.Both DLT and LTO have WORM tape cartridges as well, so you can reduce the risk of accidental overwrites.
Also a safe/cabinet that's design to be fire-proof and hold data computer items would be useful. Note that a lot of safes say "fire-proof", but they're only designed to keep paper documents from burning. Computer items have different temperature requirements, so make sure it explicitly states it can be used for computer-related items.
Also, making sure it's some water-proof would be a good idea as well--fires are usually put out with water after all.
upload them onto some free webspace interspersed with some embarrassing photos/footage of yourself - they're sure to still be circulating in 20 years.
the lazarus corporation
Abuse RAID enthusiasts?
There's security in diversity, so as many mediums as you can would be ideal, but the Internet offers the single best solution IMO.
Your data will be stored off-site, backed up on fault tolerant computers designed to be the pillars of reliable accessibility. The main concern would be finding a company that is going to stick around, which I don't think is all *that* difficult. Google is a good example of something that will probably be around for awhile and offers a tremendous amount of storage space for a reasonable amount of money... arguably less than you might pay for a home-brew solution. And when Google is on its way out for whatever erason, move on to the next big thing in 5-10 years. You even have the added benefit of having access to your photos from anywhere.
The obvious draw back, of course, are download speeds (in the U.S., at least). With the increasing penetration of faster service though, this won't be such an issue in a few to several years.
And regardless of what solution you choose, I believe it is important to have a backup of your backup as well, on some other medium. Online + Optical media, probably represents the best bang for your buck and a pretty safe bet.
Fact: Everything I say is fiction.
Nothing special about "video". You can just ask how to keep digital data for 20 years.....
Just follow these simple rules and you will have a good change of keeping your data:
(1) The data needs to be kept on at least three different physical media
(2) The data needs to be kept in at least two different geographic locations.
So by the above at a minimum you'd make write off three copies to DVD and take one of those copies to some off-site location.
But re-read the rules. Notice it says "kept" not "stored away and forgoten" that mmeans you periodically read and re-write the media. You can't expect any physical device to last 20 years. that is why you make thee copies so when one fails (not if it fails) you still have a backup. You have to KEEP thise three copys alive.
Today hard drives make good backup devices. You can buy a 500GB drive for $80. Buy four of these and rotate them to an off site location. Just take one to work with you and bring one home with you every weeks or so and re-fresh the backup. This ensures that the backups are always readable and every four or five years you replace the hard drive with whatever technology is then available.
Today 500GB hard drives give the most storage per dolar. When the 500GB drives get old maybe the 2TB drive or some Flash RAM based device will be beter.
A secondary question is what video format is best. I say it's bet to keep whatever came off the camera un-touched plus keep your finished edited shows in whatever format you are currently using. Conversion ALWAYS degrades the quality, so convert if you want but keep the camera output too.
you're collecting 100 GIGS per YEAR?
When do you plan on WATCHING this stuff?
Odds are, by the time he's three, you'll be so sick of watching him grow up through a camera viewfinder you'll toss the camera into the back of the closet.
And if you ever have another kid, he'll grow up thinking he's adopted, because he can't find any photographic evidence of his childhood.
I speak from experience :)
It's supposed to be completely automatic, but actually you have to press this button.
Stability is meaningless if you can't read back the data 20 years later. I wouldn't bet on 12cm spinning optical discs of any kind being around in 20 years, nor would I bet on SDHC, SATA, firewire, USB, or even ethernet, though the last two seem most likely to me.
If we assume that storage density keeps increasing exponentially, and that you'll continue to have data to back up, it might actually make more sense to use something that's cheap and convenient now, that you can count on still being around in 5 years, and 5 years later you move it to the next big thing, along with all your new data. This way you'll never find yourself staring at a cable trying to remember what it plugged into 20 years ago, and wondering where you're going to find something that can read your media.
If I had to throw something in a time capsule, and I had less than $1000 to spend, I would use USB keys, because they're dirt cheap (you can afford more redundancy), they use an interface that's designed for backwards and forwards compatibility, and flash is pretty safe against spontaneous data degradation. If I had lots of money, I'd get two identical, mirrored NAS boxes, and plan on finding a transceiver to connect gigabit ethernet to whatever we're using in the future. Unlike system interconnects, such as USB, Firewire, SATA, etc., ethernet doesn't require either side of the connection to make any assumptions about the physical properties of the remote host, so bridging is much easier.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
Socialism: a lie told by totalitarians and believed by fools.
Here's what I do:
CDS/DVDs have checksums to catch read errors, but little to no redundancy. RAID has the redundancy, but relies on the HDDs to correctly remember the information. ZFS has both, so it's self healing. Plus, you can ask it to scrub itself on a regular basis to detect errors before they become uncorrectable (I do it weekly)
Offsite clone for obvious reasons (ask your in-laws).
When you upgrade, your drives, just replace the disks with larger models, and ZFS will automatically grow the filesystem to use the extra space. No muss, no fuss.
Good luck!
After having been programming and doing video for 30 years (www.videotechnology.com) I had recent first hand experience with recovering video, audio, images, software from old media.
It's the media formats, media failure and bit formats that are your 3 biggest enemy's.
With with USB and firewire Media format is a Non-issue since no reader will be required.
Maybe also get a Firewire drive also.
The Video Should be in MPEG2. and bit format will also be a non-issue.
Get several drives, make at least 3 or more copies on different brand drives because some will fail.
Wrap then in foil and air tight vacuum sealed bags.
Maybe also store in several location 100's of miles apart or on several different continents just to be safe.
Caves in dessert seem to work well, or any cool dry place, where there will not be too much thermal cycling.
"thermal cycling" this last one is important, it's caused by the change in temperature from day and night and summer and winter. You want to make sure the temperature isn't changing too much where ever the drives , or any media, are going to be stored.
Total cost around $500.
Looking back over 30 years,
how many of you can still read 9 track tape, QIC40, 8 Inch Floppy's, 5 1/4, how about 3 1/4 floppies even?
What about MFM, RLL, ESDI, SMD, or Old 40 PIN SCSI drives, or pre SCSI SASI Drives?
How about Jazz or Zip Drives?
Then there is the data formats?
I have had to do this for images from 30 years ago, it's painful.
Since back then there wasn't BMP or GIF even let alone JPG, it's all raw, or Run length encoded.
I did some video from 1991 that is in a format called CellB a type of VQ.
Even audio from back then is painful to play back these days.
I don't expect todays video and audio will be nearly as hard to play in 20 to 30 years, but still I'd really avoid anything non-standard like flash, ON2, WM9, DIVX or the like and go with
MPEG2. Even MPEG4 I am reluctant to recommend because of variation that can make playback problematic and in 20 years, it will be too late to go back and fix.
I am always doing that which I can not do, in order that I may learn how to do it. - Pablo Picasso
Redundant encoding into "junk" DNA. Multiple copies made when target species used for backup reproduces.
Also explains mysterious disappearances.
Someone is retrieving a "backup".
I agree, the moderation was unfair. It should have been Redundant, since you posted both comments.
Do you really think everyone on Slashdot is that dumb?
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
Save it for 5 years, 4 times in succession. Convert when necessary.
No Way Out. Nothing on this world lasts forever.
deleting the extra space after periods so i can stay relevant, yeah.
Chisel what you want to keep onto stone tablets, or use clay tablets then encase them in a clay envelope. Then bury them in a pit in your back yard.
Haven't you seen History of the World Part I? Even that's not reliable. Moses lost 33% of the Commandments by trusting chiseled stone.Kurt Vonnegut: "If you can do a half-assed job of anything, you're a one-eyed man in the kingdom of the blind."
I'm amazed that no one mentioned it. Just get 16gb usb flash disks. It has theoretically unlimited life for archiving. The only time it deteriorate is when you continuously write/erase it.
I'm amazed this got modded up. If you look at the data sheets of most flash parts, data integrity is typically rated at 10 years.Is that the serial number for a new western digital/fujitsu hybrid drive???
First: Do NOT store them as playable DVDs. If you transcode them at all (from whatever your camcorder records), put them in h.264, with a decent audio codec -- something like aac, ac3, ogg, anything but mp3.
That alone should drop you from 100 gigs to 20 gigs, without much loss in quality.
More to the point: The way I intend to deal with this is to build a CD-changing robot. You know, something like this -- the cheapest commercial versions are more than I'm willing to pay.
This, combined with scripts to periodically check discs, create parity, and replace/re-burn defective discs, should give me a fair amount of somewhat-reliable storage. Combined with a FUSE filesystem driver, it should also give me relatively easy, sort-of random access (very slow seek time). It would also be upgradeable to Blu-Ray, if that ever gets cheaper per gig than DVD5.
Not that I advocate this to anyone who doesn't have a few dozen (hundred?) hours to kill building a robot, but I think the same basic principles apply -- use parity, check periodically for bad media (and replace it immediately), and DVD5s are about the cheapest storage you can get today.
There is one other possibility worth considering: Just use Amazon S3. 100 gigs = $10 to upload and $10/mo to store.
Don't thank God, thank a doctor!
If you're going to use an optical disc format, use DVD+R, not DVD-R. DVD+R has far more robust error correction capabilities. Most DVD burning drives will support it, although you have to watch out for commercial DVD playback decks that won't read it. Do a test.
One simple rule for its versus it's
Simple, just rename the file "Allahu_Akbar_Terror_Camp_Video.mpg" and email to anyone, anywhere. Then in 30 years file a FOIA request with NSA to retrieve a copy.
I can't find the original tips'n'tricks doc NIST published for the best way to reliably store disc media, but here is another study on longevity. I'm sure the other doc can be found without too much trouble.
As others have noted, refresh media every 5 years, and use good quality media. You can be sure that the DVD format will be around in 5 years, but you may see some writing on the wall at that time that would cause you to change your media of choice.
Also, having a copy in two different formats helps a lot. Having a backup on HDD would make it much more likely to have at least one of the copies easy to use.
Overall, though, I suggest not abandoning the media for 20 years without thinking about it - that will just make the recovery job more difficult when you unearth the material.
Somehow I suspect Google, once they've got ahold of data, will NEVER lose it again. Pity those first year university students that will be faced with embarassing videos at their retirement parties.
Keep 'em dry and in a bug-free area, and they'll be readable a thousand years from now. Or, alternatively, paper tape.
Or microfiche/film of a Hex dump.
Poor means hoping the toothache goes away.
If you want to know, ask your bank what brand of media they use to archive their records.
I worked in Proof at a large bank and we used magnetic tape for initial backup and then converted to microfiche, up until around 98.
At that point we switched to CD's/DVD's once the burners were able to match the data write speed of the tapes and the cost became affordable, and still used microfiche for long-term storage. (of course we only needed to LOOK at the items we archived, so a non-digital medium was a good solution)
The cd's and dvd's we used were guaranteed for a 25 year lifespan if stored out of light and at the proper humidity/temperature, and in a non-reactive case.
The care you take in storage is usually more important than the quality of the medium, or to put it another way even a good medium won't last for diddly if you mishandle it. I don't mean that you can get the bargain-basement media deals and be ok, but make sure you know how to store the media before you spend a bunch of cash.
By the same token, if you are making a backup then DON'T USE IT UNLESS ABSOLUTELY NECESSARY. I know people who make a burn and call it a backup. If you handle it regularly it's not a backup it's a working copy.
You need to make sure your burner can create discs on every burn that can be read by different readers- it's not just enough to 'verify' the burn, you need to take it to a different drive and test it out until you can be sure your burner makes consistent burns.
just upload to archive.org. it's easy and FREE.
Why are you talking to yourself jbane/Odder?
DVD-RAM
If you have 100Gb now, edit it down to five minutes. Throw away everything else.
Next year, do the same thing. And every year after that.
By the time the kid is 20, you'll have almost a two hour movie. Then you'll say "my god, who do I want to punish by watching home movies for two hours?" But it's way better than ten tons of footage that you forever ignore because you can't bring yourself to select which of the 18 hours of footage of him/her stuffing food in their mouth is best.
I always thought the main advantage of raid was that it reduced the impact of a disk failure from some data loss (data collected/created since the last backup run) to zero data loss.
the trouble with raid is there are a lot of things it doesn't protect against so it must be regarded as a complement to backup not a replacement for it.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Your only worry is degradation.
Popular media will always be readable after a fashion. Even laserdisc can be played back as long as you don't mind getting the player off eBay. You can certainly get 8mm film read and converted to a more convenient format and DVD is a much more popular format than that was. It's also possible that future generations will remain backwards compatible. Just as DVD drives read CDs and Blu Ray and HD DVD drive read DVDs and CDs, it's quite likely that a next generation format will read blu ray, DVD and CD.
Even if it was the same person, they said different things so it would not be Redundant like your constant and annoying blither about how everyone is really Twitter. Will you give it a rest already?
Political torture and murder is not funny http://slashdot.org/comments.pl?sid=581079&cid=23757591
Keep a document of that standard.
Every time there is a major change, convert them with an automatic process.
Really though, how often is digital video format going to radically depart from what's out there?
Alternatively, keep the 'RAW' format on disk, and just convert from that.
The Kruger Dunning explains most post on
Three Color-separated 35mm negatives. If kept well, it will still be in good condition. And something will be able to digitize it.
Digital media won't be readable, for lack of a drive.
Where can you get a 9-track 200bpi NRZI drive today?
Grab a LTO drive off of eBay... tapes are not expensive (LTO2 tapes and drives are easy to come by as everyone upgrades to LTO3)... The media has a 30-year shelf life, which, I would imagine, can be extended with temperature/humidity control.
Stash it an encrypted archive with a boatload of porn and upload it to a few P2P networks and torrent sites. It should last at least a couple years. Throw in all types of stuff to keep people interested so they'll store it.
DLT is considered end of life tech in the tape world. For the VS-160, the transfer rate is very poor (~6MB/s) compared to even LTO Gen 1 drives (~14MB/sec).
SDLT drives are nearly as unreliable as their ancestors and extremely sensitive to heat (your case fan dies and the drive is unusable). And, again, slower than the competing LTO drives.
A used LTO2 drive may be the sweet spot (200GB native per tape and 28-30MB/s transfer rate). I prefer HP and IBM drives over Certance/Quantum drives and half-height drives run at half speed. Even with a SCSI card (U160 or U320) you may be able to get set up for less than the $700 mentioned above.
Or you could always splurge for LTO3 (400GB native) or LTO4 (800GB native). I'm not listing the compressed stats (usually double) because video (usually already compressed) doesn't compress easily.
Mediocrity knows nothing higher than itself; but talent instantly recognizes genius. -- Sir Arthur Conan Doyle
What video do you not have from twenty years ago, 1988, but which you really, really wish you still had?
I can only think of half a dozen items, and I actually still have some of them; although I'd need a VCR and TV to watch those exact copies again. (And the will power to dig them out of whoever's basement I abandoned them in). --Then I discovered that somebody was in the same boat but who cared more than me and so did the work to digitize and upload their VHS copies to the internet as torrents. (Thank-you!)
But basically, I did absolutely nothing, and I still have access to everything I would have wanted to keep.
Although, I admit, I don't shoot my own stuff; most of the things I might be interested in seeing are commercial product, which means somebody else is worrying about maintaining its life expectancy. Really, unless you have very specialized needs, are you really going to care about watching ancient SouthPark episodes when you're twenty years older than you are now? --I've lost enormous amounts of data over the years, and frankly, I consider it very healthy to let stuff go. How relevant are old "Cosby Show" and "Family Ties" episodes today? --Stuff which if you really want to see again, can be found with relatively little effort.
I suppose the whole YouTube phenomenon offers a different dimension; there's lots of video news and evidence which is available only on-line, and such footage might well be useful for historical reasons later on.
Still. . , the question is almost certainly entirely academic; if you think you're going to be watching movies in twenty years time, you're far more optimistic than I am about the continued viability of the human race.
-FL
The real "Libtards" are the Libertarians!
You're never going to watch that stuff, and your kid will never want to watch it.
Look how rarely you flip through photos, and that's a shared communal experience since you talk. Video is boring. You sit quietly and watch for hours on end. Screw that.
Have you ever wanted to watch any home movies your parents made? Of course not!
For my long term archives I'm currently using LTO3 Tapes (last year the drive was 1750$ from Dell, with a 5 year warrantee) and 5 packs of tapes are about 150$ on eBay, they hold 400/800gb (I get about 430gb a tape due to already compressed content)
WORM (Write Once Read Many) tapes bump up the life expectancy as well, I keep one local copy in the gun safe, and one in a bank safe deposit box, the tapes are backwards compatible for at least one generation of tape (LTO3 will be readable in a LTO4 drive, etc)
I cycle the tapes out once every 6 months, and make sure I can restore the data of the 'Bank' copy, and the local copy.
All these backups were on DLT1 tapes until last year, so as time progresses you can simply restore the tapes, and upgrade to the new formats (and backup software for that matter)
Don't bother. I have videotapes of my kids on VHS and Digital-8. I have never gone back to watch any of them.
Self awareness - try it!
RAID5 does not last forever. If one drive fails, the array survives, provided you replace the failed drive before another one crashes. It sounds unlikely, but it does happen - sometimes there's another failure before you can replace the faulty drive.
another issue with single redundancy raid is that sometimes a rebuild is not possible because while all but one of the drives are alive some of the drives that are alive have unreadable sectors.
raid and especailly single parity raid is not a substitute for backup (full backups also have the usefull side effect of testing all the sectors backed up for readability and making the drive remap any that are going weak)
unfortunately from what I can gather raid6 is a far more complex algorithm than raid5 meaning it is only seen on high end controllers.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
Isn't it funny how every account listed in that journal ends up posting in the same threads? It doesn't take a physics degree to figure out that ibane, Odder and deadzero are the same person. You just legitimize all that when you insist on replying to yourself.
WW900FTJD?
He would use a tape duh.Bits rot; Paper goes Poof.
Etched titanium long preserves.
But only fresh copies approximate permanence
That depends. Building a small computer to act as a NAS (and run a projector, and a few other things) with a 200GiB raid-1 recently cost my friend about %500, with monitor. Without the video card and monitor it would be even cheaper, or the same price with more storage/raid-5. And that's for a whole new computer. NAS w/raid on site, off-site something else. Either a network backup service, or DVDs in a safety deposit box, upgrading to the latest format every few years.
Not a sentence!
I should also note the possibility of using FreeBSD/OpenSolaris and ZFS for this, since the copy-on-write nature of ZFS+RaidZ means you aren't screwed if you accidentally delete/overwrite a file, and snapshots are easy. Also, like linux, it's easily administrable over SSH, so you don't need a monitor/video card, just a network connection.
Not a sentence!
Time proven long term storage:
Clay Tablets.
Simply chisel your 1s and 0s onto sun-baked clay tablets. They will be readable 1000s of years from now...
900ftJesus Saves.
To paraphrase a data adage, "If your god doesn't exist as three people, it doesn't exist at all."
US networks are not capable of HD video streaming, so I put OGG Theora in my video blogs with links to better quality for those who want it.
I watch HD streaming on Vimeo all the time. I live in Phoenix. H.264/flash is the way to go.My IT teacher warned me about storing data on DVDs written with an ordinary DVD writer...apparently the data is still readable after quite some time, but the quality diminishes significantly. I myself have experienced this with normal CDRs. I backed up my music on to CDs and after a year of leaving them on the shelf, there was a lot of noise in the background. I'd say optical devices aren't the best way to go. Apparently the good ol' magnetic tapes are quite secure...if you don't happen to be living in a giant magnet oO"
Neither would I, but I just wanted to comment that it's highly unlikely OGG Thedora will be around in any substantial sense in 20 years. Yes it's 'open' and all, but its not widely used and will very likely be replaced at some point by some other OSS flavor of the week.
Business. Numbers. Money. People. Computer World.
But I sort of hope you all die in a horrible car crash very soon.
Because honestly, who needs to post this sort of shit to slashdot.
I'm old, and you're ruining it for me.
Car crash / fire. Soon please.
Yes, this is what you'd moderate as a Troll. Not because I am trying to Troll, but because I really am THAT bitter.
Thanks,
Grumpy Guy.
Has anyone thought about SDHC cards? Yes they're expensive but does anyone know how long they're supposed to last? At least one HD camera records to them exclusively. Frankly, all the suggestions on here are too cumbersome. I don't want the pain of a RAID1 array and swapping out all the hard drives every few years. Yes this is slashdot but your average joe could never figure it out. Nor do I want to make millions of DVD copies. We need simpler suggestions.
Does God treat us as servants or friends? Check my homepage.
The best way to preserve your movies for a great length of time is to encode the images in a flip book made of leather. BE CERTAIN THE LEATHER IS FULLY PRESERVED! Make sure the flip books have cover sheets (2 or 3 each front and back). Then wrap in leather and toss in a peat bog. They'll be there when you want them again in a thousand years or so. If you're not sufficiently near a peat bog, super dry desert sand (Egypt, etc.) will work.
Another style of this is to encode the images onto metal sheets, then coat them with something non-organic and non-corroding (what to use??) {actually, aluminum could work since the oxidized layer preserves the rest of the metal... just be sure the encoding is robust in case you have to sand off the oxidation before use}, but still see-through. A sheet metal would be best as then you could use a machine to flip the pages for you, giving you the 30FPS needed for a good viewing experience.
I hope this comment is well received... I could have moderated instead!
Persecutors will be violated!
Consider having them professionally printed to photo paper and store them in a box in the closet (or a safe deposit box - usually reasonably temperature/humidity controlled). Don't laugh. I recently was looking through a huge box of old family photos and negatives. Some of them date back to the early 1900's and many still look as if they were printed yesterday. Most of the negatives are in excellent condition and I've had a few of them reprinted as 8x10 prints. Granted, most of them are black and white, but most of the oldest color prints and negs still look darned good. Color slides I took in the early sixties still look great. There's a lot to be said for pictures and documents printed on good quality paper and film. There are books/documents/scrolls over 2000 years old still in existence.
Printing video to film is a really good idea if one is interested in longevity, actually. Film studios do it.
Are you adequate?
About a year later I tried installing from the same CD and the install failed on multiple attempts. I grabbed and tried the second CD with the same result. Inspection of the data side of the CDs revealed discoloured areas bounded by discreet wavy lines, something like the bronzing described above.
My personal hunch is that this is some form of planned obsolescence, like the movies you rent that supposedly degrade after a day or two--I've yet to have a factory-pressed audio CD or fully-licensed OS install CD degrade in quality, although I have some from 10+ years ago. Meanwhile, any audio CDR I have has shown audible signs of degradation usually within a year or two of recording.
db
I am literally 3000 tokens away from the chaotic crossbow --Stephen
Yeah, I know this isn't the most serious thread around, but actually, serious answers to this question are interesting. So here goes: the very best archival medium for photos and videos is probably slide film. One of the big advantages of this particular analog medium is that it's easy to "read" the images even with your own eyes.
For all we know, somebody 80 years from now who stumbles upon one of our hard disks may have a hard time figuring out how to read the data and how to display it.
Analog has other advantages, even if the medium is not as self-evidently pictorial as slide film: analog can degrade more gracefully. A few bad bits in a critical section of some digital data can make it impossible to interpret the whole file. In an analog medium, that is less likely.
Are you adequate?
The permanence of digital media isn't an any one format, but the fact that it can be copied without generation loss as often as you like. So all you need to do is make copies to a new format periodically and keep those in a safe place. The DVDs are fine for now, copy them over to Blu-Ray or something like it. A hard disk would work too. Just every few years or so get the old copy, copy it to new media, and put them back in storage.
I have papers I wrote in highschool still on my computer. This isn't because I stored them on some non volatile media, it is because they get copied every time with my system. I get new hardware, I copy the old data on to it. The original disks are long gone, but the data remains.
Likewise, if the actual format for the data starts to get old, you can convert it. Suppose 10 years down the road DVD players are getting hard to find. Ok, no problem, just reencode the video in to a new format. I've done the same with the documents I mentioned. They were originally written in a version of Works. They are now in Word 2003 format, and can be updated again, when I feel like it.
You just have to get away from the idea of something that lasts forever and you never touch. That isn't why digital is resiliant. Digital is resiliant because it's cheap and easy to copy.
This is correct information. Gold cables really help in two ways. First of all, when transferring bits, gold follows the golden rule, reducing the number of evil bits that go through, making it less likely that you will wind up with malware in your backups. Second, because of the higher conductivity of gold, you aren't just transferring 1s and 0s anymore; here at my freelance gig we're seeing many 2s, 3s, and even the occasional 4s being carried through the cable. You'll find that your resulting backups are even more accurate than the originals.
100G of DVD video is 20 hours. If you keep up at this rate (although you won't -- trust me) you'll have 400 hours (2TB) in 20 years. Good luck watching all that... Still, probably the best solution is to keep buying newer HD's for a while. In 20 years something else is likely to come along. In the meantime, you might also want to think about encoding in something else (DivX?) if the amount of storage is a problem.
External array isn't necessary, but it is how I do it.
Don't use Hardware RAID cards - use OSS Software RAID so you can retain the RAID algorithm regardless of other upgrades. I was burned when Promise changed their cards and the old software was incompatible with the new card/software. My data on the drives was worthless after a controller failure.
There is something to be said for KISS too. Keep It Simple Stupid.
Okay, but he's talking about archiving 100GB, which would definitely put him in the MozyPro category.
One, it's $50/mo for 100GB, not per year. And also, assuming he can actually get 1.54MB/sec upstream, it will take over 18 hours to upload. If he's getting closer to 150K/sec, it's more like 180 hours (7.5 days).
When you're talking about accounting data, inventory, policy manuals and other small footprint important stuff, then Mozy is a good idea. Until they get hacked.
Mediocrity knows nothing higher than itself; but talent instantly recognizes genius. -- Sir Arthur Conan Doyle
1 Tera-byte = 1,000,000,000,000 bytes (sorry guys, not going digital here)
The prefix "tera" originates from the Greek word teras meaning "monster".
Equivalent to:
416,666,666 pages of text (30 lines double spaced by 80 characters)
714,285 floppy disks (1.4M)
1,428 CD's (700M)
223 DVD's (DVD+R 4.7G)
----------------------
Also: The paper perspective
833,333 reams of paper (500 sheets of 8.5 x 11 paper)
148,041,666 cubic inchs of paper (8.5 x 11 x 0.0038)
85,672 cubic feet of paper
131,944 linear feet (8.5" x 11") stacked this high
16,493 stacks of paper floor to ceiling (8' high)
a wall stacked 11" deep, 8 feet high, and 11,682 feet long
a foor to ceiling (8') stack whose base would be 103' x 103'
Just for the fun of it :-)
You forgot the most important thing in building a fault-tolerant RAID:
Do NOT use identical disks. Same reason why someone else suggested HDD + DVD: 2 faulty drives out of a dozen is better than all 12.
If you read the post it says that he *already* has more than 100G of data... having hundreds and hundreds of CDs is not a good option... our friendly poster needs an *easy* and *reliable* method of storing data.
Two things to think about:
RAID 5 suffers catastrophic failure when two drives fail within a few hours of each other. The stress of rebuilding the replaced drive from the working ones makes this more likely to happen than some storage vendors want to admit. RAID 6 should do better because three failures in a short time are a lot less likely.
As far as I know, hardware RAID is risky if a RAID controller fails because one brand of RAID controller often can't read an array created on another brand. Software RAID on a Free operating system is more portable.
Really, there's no single way to ensure your data. Redundancy is key. For a simple application, I'd back up to at least one external hard disk (NAS works great, too, and some use RAID) and then backup to an online backup service. There's a lot of backup software out there that'll backup to your external drive. Those fail, so it maybe a good idea to get more than one. The online backup service will ensure your data is safe if something happens to the device in your home (fire, theft, the general unthinkable).
From my experience, frequently used recordable CD/DVDs last about a year until deterioration takes its toll on them. I read somewhere that they'll last about 5 years on the shelf. Albeit this is a cheaper route, you get what you pay for.
Chewbacon
The Bible is like Wikipedia: written by a bunch of people and verifiable by questionable sources.
Raid kind of is a good idea, if you bump up the parity blocks. But you're right that it's better to store files unencrypted and uncompressed on regular media with extra parity bits, rather than doing it at the level of the block device.
Frankly, though, the state of ECC in Linux is rather appalling. Par2 is interesting, but the inability to traverse directory trees makes things complicated. DVDisaster looks interesting as well, but it's kind of wonky.
And neither of them is particularly well known (so I'm probably missing something), so there's no guarantee that they'll even be useful in the future (which is why it's definitely a good idea to store those files plain-text.)
Scientific data for instance is often stored in ascii text files, rather than encoding the numbers directly into the bits. This ensures that in the event of damage, at least part of the data will be recoverable by someone.
Really, that sort of idea: putting extra parity blocks on the disk with your data, and filling the empty space with parity blocks is pretty fundamental. It's surprising it's not a part of K3B or dvdrtools.
Can you be Even More Awesome?!
If you save your files on cds or dvds, you could include par files on each disk that would allow you to reconstruct damaged sections. Works great on Usenet. I use Quickpar to recover important *cough-porn-cough* data and it's almost magical to see it rebuild bad or missing file sections. Most of the binary files on Usenet, of course, use rar lossless compression and that might be a convenient way put the backup data into par-able pieces.
Real men chisel the bits into a clay tablet then carefully bury it in a tomb with constant temperature and humidity.
Comment removed based on user account deletion
Convert to binary, print on acid proof paper, lock away in blast shelter.
When you want them again, just grab the several thousand pages you have locked away, scan them, OCR the scan, and you're good to go.
It's the only way to be sure.
How's tape better than Hard disks? In both cases you have to send the media to a couple of off site storage facilities, but with hard disks, you don't have to worry as much about the media being readable with different equipment.
And I thought that the cost per gb of the drives is lower than the cost of tapes and related.
Honestly:
External HDDs are absolute win for storing huge amounts of data, since I formed the habit I've been far less strapped for what to do with all my p- err, photography?
~I'm abusing USB2.0 atm, even hooked up a permanent external drive on my main box, never looked back.
If you're going to store optical media in a fire-proof safe, be aware that a normal fire-proof safe isn't sufficient for optical media. Normal safes are designed to keep the contents under 350ÂF, the temperature where paper scorches. That's too high for optical media.
Here's an LTO2 drive for $800, slightly more capacity for a little more money, of course you'll need a $50 U160 card as well. Tapes are about $10 less so depending on how much and how often you backup it can end up saving you money.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Half the point of memories is being able to improve them and embellish them - video ruins that.
There's an obsession with recording events. Too often fun times are intruded on by "smile for the camera" or worse still "tell the video".
Engineering is the art of compromise.
You should cull some of it.
You might try this site. They may be able to answer your question.
http://www.supermediastore.com/taiyo-yuden-dvd-plus-r-media.html?WT.mc_id=cjtext
Hey, I'm twitter!
How much are you willing to spend? That seems like a lot of hours of video for a one year old. I mean we are talking 50 hrs at DVD quality and a couple hundred hours in a reasonably compressed format. 4 hours a week of kid video is a lot. By the time he is 22 and graduating college you would have 4400 hours of video. You would get lost in all of that and never be able to find anything meaningful.
So here is my suggestion:
1) Start making it into something usable now by doing some editing.
2) Upload your edited material to Amazon S3, it should take up a fraction of the space of your raw footage and cost you maybe $1 to $2 dollars a month per year of video (put all of your photos on S3 as well.
3) Get something like a Netgear ReadyNAS and keep your raw footage on that (or build something) You should be able to come up with a RAID 5 solution with a terrabyte or so for anywhere from $300 to $1200. I don't normally advocate Linux but here is an example where a micro ITX box and 3 decent sized SATA drives would be perfect. There are a couple of good "appliance" type Linux RAID distros.
http://www.norsam.com/
Disclaimer: No, I do not work for them, nor do I own any stock in the company.
Probably out of your price range, but it would certainly outlast anything mentioned so far.
If it weren't so sad it would be funny.. that I can still make prints with negatives my grandparents took at the turn of the century..that would be 1900 for the youngsters. Even home movies my parents took are still playable. Granted, home movie quality was never that great to start with back then but it has held up pretty well (ignoring that noisy projector). And yet stuff I took with a digital camera 8 years ago is in danger of either media failure or format failure. And does anyone feel comfortable with just one copy? The shoebox with all the family negatives seems way more reliable.
No, you're not, I'm Twitter!
You two need to quit impersonating me. I'm Twitter!
Will the real twitter please stand up?
I read most of the answers but could not get a real argument against the online storage. There are enough companies offering a very good price/GB ratio. Then they have to take care of the backups, which my web space company claims to do several times/day. And I get 1/2 TB for less than $8/month. Considering the fact that I don't care for anything than the upload, it is a deal. Long upload time? So?
xoda.org
For media that will last that long either some nice silver DVD's stored perfectly or if you want it really locked in, Magneto Optical discs are super. They are usually protected in a casing, and data can't decay as easily as the disc needs to be heated before it can be changed. Either that or just keep the data on a disc array and keep that array alive and with current generation hardware as time progresses :)
MooCow
I have not seen this mentioned here yet, but the fact is that hard disk drives are not a viable long-term storage medium either! The magnetic domains that define the 1s and 0s on a disk drive platter are subject to degradation via ambient thermal and magnetic effects, and quantum effects as well. Over long periods of time, the overall effect is to "average" or "smooth" out the 1s and 0s.
It is not a large effect, but it does not need to be at the data density of modern drives.
If one wants to store data on a hard disk drive for many years, in order to preserve the data it should be copied and re-written to the disk periodically. I am not sure of the optimum period, but for lack of hard data I would refresh mine at least every 5 years.
The upside of this is that it takes a relatively short time to re-write a hard drive. Re-burning CDs or DVDs would take a lot longer, and probably be necessary more often than 5 year intervals.
I would like to point out, too, that for long-term archives, if one wanted to use burnable CDs or DVDs, of course they are "light-sensitive"! They are temperature sensitive too. So for maximum lifespan (this is not a joke), they should be stored in a cool dark place. Just like beer.
How about we broach the subject of video formats? The dv codec is *NOT* optimized for space-efficiency or quality, but speed -- so a low-powered cpu can provide speedy compression. If you re-encode the video with one of various other formats, it seems you can get smaller file sizes at the same quality level as the original dv codec.
I know recompressing works very well for the jpeg's that my Canon camera takes, if I load them into photoshop and save the photos again using Photoshop's JPEG format. I've found saveing my 6 Megapixel photos at High Quality (9/12) is great, the pictures are almost identical when you zoom in to compare at the pixel level, and the resaved file is 1/2 to 1/4 the size of the original.
Can someone recommend what to do about recompressing video with a similar space-efficient codec? I actually haven't figured this out. Do you re-encode as MPEG4? H264? MPEG2 for superior backwards compatibility? Is leaving it as the dv codec really the best solution?
How should I store it so that it's still readable 10 to 20 years from now? Hardware and software will change in 20 years. Do you still use 5-1/4" floppy disks, and does your computer have a drive to read them? 20 years makes a huge difference! Ten years ago, I would have suggested quality photo film and prints--but that has lost market share, and once-popular photo products have been discontinued. If you want "permanent", buy granite and good chisels. Otherwise, resign yourself to migrating to new formats every couple of years.
If you want something to last indefinitely then you have to store it in multiple locations. You will need to store it on a hard drive that is either in a RAID configuration or is backed up on another drive. You will also want to back it up somewhere online (http://www.mozy.com is a good one). Over the course of 20+ years it is completely possible that a fire or earthquake could wipe out your home so you need to have everything backed up on a remote site. You could also back things up on a family or friend's computer if you trust them (or you encrypt the data). In general, you want the media backed up on several different physical media and in multiple locations. Since it is so cheap to store data on multiple mediums in multiple locations, there really isn't any reason not to.
As time goes on and new formats come out and old ones die, you will have to convert those files from one format to another. If these files are in some sort of proprietary format (not recommended), then also backup the installer to the application that is needed to read them. However, a DVD iso image should stick around for a while and it's not like the format will just disappear overnight. But if another format becomes "THE STANDARD" then I'm sure there will be tools for converting DVDs to that format.
"Oh dear, she's stuck in an infinite loop and he's an idiot" -Prof. Farnsworth (Futurama)
Rubbish, the heads are parked when the drive isn't spun up and they NEVER touch the surface of the disk (ok, very rarely, but it's a failure mode you notice immediately because the screeching of a head hitting the platter at 7,200RPM (let alone 15K) is 10 million times worse than nails on a chalkboard). The actual problem you are describing in caused by one of three things, the lubrication in the drive going, a groove being worn into the bearings (for pre-liquid bearing drives), or the head failing to leave the parked position. All three can often be fixed by employing the snap trick, as the drive spins up rapidly rotate it about 90-180 degrees along it's axis of rotation.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
I don't think you would be interested in videos of my kids.
I'm a paedophile, you insensitive clod!Ever seen the youtube video with ZFS striped across the USB thumb-drives? You can implement RAID5 (raidz) and/or RAID6 (raidz2) and lose up to two of the thumb drives PER vdev. Get 8 GB usb thumb drives and get lots of them. Create groups of 3 or slightly more thumbdrives per vdev and raidz2 away. You can safely lose 2 thumb drives and still re-import the entire zpool by plugging the USB drives anywhere. BTW, you'll need this: http://chris.pirillo.com/2007/12/31/13-port-usb-hub/
if you were me, you'd think the same way
Except in order to get smaller data domains the magnetic coercivity has gone up with each generation of drive so that today you are LESS likely to have a bit randomly flip then you were in the old days. Todays media have a coercivity of around 2500 Oe, or able to withstand about 10,000 times the magnetic field density of the earth at the surface (average). Not to mention that modern drives use very advanced ECC algorithms to account for the occasional bitflip.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Why do people think these disks are for archival storage, they are really just good for moving data around without a network. If you want to archive things you copy them 3 or more times then periodically compare the copies before copying them again into a new location. i.e. keep the data in multiple streams that are kept in motion so that issues around physical media become irrelevant. e.g. get 3 1TB drives and copy the data on to all three, then when those drives are almost full and 2TB are as cheap as the 1TB drives were, you then compare and copy the data onto the new drives. The ideal method is to have this managed by the operating system and a network file system, where you just keep adding more storage to the data cloud and the system allows the old hardware to be removed as it fails with no impact on the data. To be totally sure you need to also spread the data over multiple locations to account for potential large scale disasters. All a bit much for you? Just get a 1TB external drive that is mirroring your data (it has two disks inside), then start saving for the 2TB one to move across to in the future. You need to lease longevity, you can't buy it, which is that the DVD disk delusion is all about. It is an ongoing cost to maintain your data.
I like the idea of uploading the stuff to a service like Amazon S3. They're much less likely to lose all your data and will constantly be mirroring it and swapping out old drives. Upload the files and let them worry about making sure it stays intact.
...double-...
...sided.
Totally.
"I don't care about the Constitution!" --Bill O'Reilly, November 17, 2009
This is what I've been doing for years and it has proven robust and manageable. Someone ought to productize this sort of solution to sell to non-technical people... :-)
1. Obtain two geographically separated sites with basic broadband, e.g. your house and some friendly relative or friend's house
2. Put together two cheap headless PCs with basic all-integrated motherboards and at least 4 internal SATA ports and drive bays. Total cost fo r the (diskless) system should be easily less than $250 US. (Use an existing monitor and keyboard for initial setup.) Ideally, add a basic home/small-offic UPS that can be monitored by the Linux host for clean emergency shutdowns.
3. Choose your disk sizes for best value for RAID 5 with about 250% more space than your expected archive size. I use RAID 5 with three disks and have the option of rearranging disks later for expansion. 1 TB via 3 500 GB drives would cost less than $250 per host.
4. Install Linux and use software RAID 5 so the drives can be moved to new machines etc if there are failures.
5. Create a regular filesystem and a "backups" filesystem as separate RAID 5 arrays on the same machine. (Each disk has several partitions, one being a part of each RAID 5 volume.) Run nightly incremental-generational backups between local volumes via cron jobs. E.g. rsync with --link-dest mode can build link trees of shared data and only use significant space when new unique files are added.
6. Mirror the regular filesystem contents between the two machines once via high-speed LAN. Both machines continue to make local backups of their own regular filesystem.
7. Move the slave mirror machine to the secondary site. Leave both machines running 24x7.
8. Run regular mirroring from the master to the slave machine via cron, e.g. use an appropriate rsync+ssh job. Use dynamic DNS to establish the connections easily.
9. Enable SMART monitoring on both machines and actually check them regularly.
This solution protects against localized disasters that destroy an entire host. It also protects against simple disk failures, with less downtime or recovery effort than a total machine loss. The backup system protects against user error or corruption of the regular filesystem by keeping generational copies of older data.
The use of rsync makes it viable to sync frequently over broadband connections. Whether you can afford nightly or only weekly depends on your broadband speeds and rate of change of data.
These servers cannot be left alone for decades, but eventually need to have the data read/write scrubbed due to the limits of magnetic storage.
But, in a practical environment with new data being added, you will probably find that it is time to replace failed disks or increase storage space long before this issue arises.
If you strategically replace drives with equal size or double size drives, depending on cost and projected storage needs, you can eventually migrate to larger filesystems once you upgrade enough disks. For example, on a shoestring budget you can start with 3 x 500 GB disks; replace failures with 1 TB disks; eventually grow your arrays when you have either 3 x 1 TB disks or 2 x 1 TB + 2 x 500 GB installed. You will have a situation where your existing 1 TB usable space is only using the first half of each 1 TB drive (and one of the two 500 GB drives in the mixed size case).
Because of the dual-volume approach for regular and backup filesystems, you can initialize the new space on the second half of each 1 TB drive, copy all data into it, then reformat the old space when converting from a 1 TB to a 2 TB usable space. Also, because of the multiple volumes, you can us ea mixture of 500 GB volumes from "half" a 1 TB drive and a complete 500 GB disk.
Once you understand the reality of this situation, the juvenile desperation really shines through.
Would you really consider it a great thing if, somewhere, there were 100gigs of video of you as a baby/child? I shudder enough knowing that my mother has a video of my 10th birthday. Put another way, isn't this principally the same as the British government attaching a camera to the pole in front of your house?
Nobody thinks this stuff is a good idea except the parents. Watch the videos all you want until your child is old enough to understand that it's him/her in the videos. At that point, delete the videos and make no attempt to otherwise preserve them.
It's what a good, responsible human being would do.
Whatever - the solution is still spinning up the drives every so often. So long as my data is still available, I really don't care why the voodoo works.
How are sites slashdotted when nobody reads TFAs?
-- I have a private email server in my basement.
Post them to USENET, they'll live forever...
I have found that my older DVDs (~3 years) have already begun to fail. These were stored in jewel cases away from the light at a normal temperature and not used more than once or twice. I would say 1 in 20 has failed and 1 in 8 has some corruption. As for my much older CDs; they seem to have a slightly lower failure rate but not by much. Thus personally I would write off DVDs for long term data storage.
I keep my vids on their original MiniDVs assuming that the data will degrade a bit but mostly survive the decades.
Amazon S3, accessed through Jungledisk. Don't bother to encrypt the files, its just another fail point. If they are that valuable to you, paying 15 cents per gigabyte per month may be comparable in cost to the cost of HDDs or DVD-Rs you'd need to buy to preserve them reliably.
Amazon's system is vast, cheap and reliable. Its what you need. I use Jungle Disk as an S3 client because it allows auto backup, resumable upload and is crossplatform
I have been a user for about 10 years. This ends Feb 2014. The site's been ruined. I'm off. Dice, FU
As long as Moore's law is in place, there _never_ will be a "proven" long time storage technology which can handle today's data amounts. If you want a proven method for 300+ years, you restrict yourself to 17th century tech.
I've spent a lot of time looking into this since I'm an amateur photographer and have quite a large number of photographs that I want to archive. Based on what I've read, a lot of pros are following a three tier storage system:
1. Active RAID array
2. Backup RAID array (copied every night for example)
3. Offsite hard drives rotated every so often and updated with the latest data (one at home, one in safe deposit box)
Personally, since I'm going kind of low budget, I'm currently using one drive plus DVD backups with the eventual goal of picking up external hard drives to supplement the DVD backups.
A certain amount of that is hype. (Testing techniques are known to not fully simulate all elements of the aging process -- vibration, light exposure, thermal variation, incidental scratching, torsion, come to mind pretty quickly.)
The greatest factor in aging of CDs and DVDs, IIRC, is oxidation. Poorly constructed CDs tended to let air and humidity in. But well constructed CDs tend not to have oxidation problems, at least not within about half of their rated life. (Think glues and paints.)
Theoretically, if these are well constructed, they should be at least as durable as the best we can expect from conventional CDs and DVDs. And, at the prices shown, it would be a reasonable approach to being sure your stuff is readable as long as you can find hardware/software that can handle the formats.
But if they are not well constructed, the other factors will come into play.
I'd like to see some independent testing, including tensile stress, vibration, incidental scratching, etc. More information on the construction would also help. Testing them myself might be fun and instructive, if I could break the time for it.
Computer memory is just fancy paper, CPUs just fancy pens with fancy erasers; the 'net is just a fancy backyard fence.
Since when did possessing 100G of goat porn become something to brag about?
if you were to ask the media/industry of today, your :)
answer would be 'just buy it again'
I'd say you've got to be looking at the clouds. bank on Google being around, bank on the archiving working, and bank on the having format conversion tools available, and upload all your stuff. Don't bother writing out to a hard format if you don't want to copy-up every few years. Of course if you're filthy rich you could film out 3-colour ycms to black and white silver neg. The film industry does that for a reason, because 70 year old cans still look good.
On film.
Great minds think alike; fools seldom differ.
i think analog is still the best way.
how about printing all the pictures back to 35mm negatives. negatives have been able to withstand much longer times even with dirt, molds, humidity, warm temp, etc.
for videos, well convert them to 35mm movies? print to film? expensive but i think will last for generations.
Live your life each day as if it was your last.
Ok, you guys forced me to make my first post. All of you that are saying "put down the camera" are completely missing this very important point. Long term digital media storage is a big problem, regardless of the source. I heard a guy from the National Archives discuss this last year, and he was saying there is no real good solution right now. I think he said they had about a 10% loss over 10 years from CDs. And they know how to store them properly (temp, humidity...). His only real advice for long term storage was to put them on HDDs, mirrored, and transfer them periodically to new drives. And then hope something better comes along.
Upload the stuff to YouTube. They have a vested interest in keeping their archive in viewable format.
Digital files don't have a good storage medium yet. Any storage technology (and that includes the storage vendors' stuff) designed to hold digital files is subject to bit rot, cd rot, or equivalent. This isn't an immutable property of digital files, but I'm beginning to think it's an immutable property of the commercial environment for digital files.
The key to storing digital files is that you can make a perfect copy as many times as you like. So to store a digital file, copy it often and to multiple destinations
Analogue storage mediums do last, and don't need to be copied, but they also can't be copied without losing quality.
So, your answer is:
- If you want to store one copy and forget about it, transfer the footage to analogue medium and shove it in a safe.
- If you want to keep it digital, then be prepared to copy it regularly (and often) and maintain multiple storage locations.
Business/App ideas are like arseholes: everyone's got one, they're mostly shit, but very rarely they contain a diamond
Tape has guarantee (real guarantee) over 50 years and it will be there for decades thanks to the organisations using it like Banks, Govt., Military
http://www.answers.com/tape+drive?cat=technology
Everything is open, documented and it is designed for reliability.
Put the videos in their native format (Dv etc.), put it into a bank safe or a safe.
Use Tape. It'll last for a while, and there'll always be some kind of reader around. The data density is also unbeatable. I have an old tape drive with cassettes that allows me to store 8 GB per tape, and I got it in 1994. I've seen newer ones that store 80 GB easily.
I think you may be referring to an old Mozy plan as well. Mozy Unlimited for home has no restrictions on space, and if you are ultra paranoid you can use your own keys. Also the initial backup is what takes the most time, after that it is essentially just differentials with a 30 day window. You can also have DVD sets created for extra redundancy and have them sent to your house or an offsite location. My initial backup took a little while but since then the diffs have been so small there hasn't been a problem.
I used to have the same opinions of netbackups as you mentioned, back when the Z drive was popular. But they have come a LONG way to meeting my paranoia needs ;) Check out https://mozy.com/support I think you will be impressed.
CS: It is all sink or swim...oh and did I mention there are sharks in that water?
Anything short of a RAID 5 file server is inadequate, IMO.
Carve the binaries into stone.
The restrictions imposed by the GPL are similar. You are free to do almost anything you want with the code as long as you don't restrict the freedom of others.
We don't see the world as it is, we see it as we are.
-- Anais Nin
DLT4 had mechanical reliability problems (Quantum had a monopoly on drives in that range at the time, and didn't give a shit), but Quantum had competition from LTO during the SDLT drives dev cycle and they're OK. Sure, you can get LTO2 (200GB/14MBs) instead of SDLT (160GB/10MBs) but would you pay an extra $500 for that difference for 100GB of personal files?
LTO3/4 is $2500/$3500. At that price you could just make 50 copies on 50 hard drives and mail one to each state.
Socialism: a lie told by totalitarians and believed by fools.
Nice price! Certianly worth it if you already have a SCSI card handy, heck probably if you don't.
Socialism: a lie told by totalitarians and believed by fools.
I see someone found an LTO2 for just $100 more in this thread - worth it for that price difference.
Socialism: a lie told by totalitarians and believed by fools.
Um. No. SDLT still sucks. We test them all day. And we hateses them. They are cranky, quirky, extremely heat sensitive and slow. Yes, they are better than DLT, but LTO still kicks their ass six ways from Sunday.
LTO Gen 1 xfer 14MB/sec
LTO Gen 2 xfer 28-33MB/sec - yes, we actually fail/repair LTO2's for less than 27MB/s xfer rate.
LTO Gen 3 xfer 68-75MB/sec
LTO Gen 4 xfer 107-113MB/sec
SDLT320's xfer 10-12MB/sec
SDLT600's xfer 22-24MB/sec - and will overheat 30sec from power on without active cooling!
DLT/S-4's xfer dunno.
You can find used LTO2's on ebay between $500 and $600, if you dig deep enough. Retail is for consumers.
Mediocrity knows nothing higher than itself; but talent instantly recognizes genius. -- Sir Arthur Conan Doyle
I worked at a company that stored about 1500 recorded DVDs and faced a similar conundrum. We found the DVD's data unreadable / unreliable in as little as 2 years. Generally speaking we used the cheapest DVDs we could find when we placed orders, but I still would not trust even expensive "high quality" burned DVDs with memories you cherish and don't want to lose.
I don't want to redefine your requirements, so let me state what I think they are before I give you my suggestions. (Especially since everyone here seems to just wants to tell you what you should be doing instead of how to do what you want to do.)
1. Capacity: currently 100GB / year
DVD quality is pretty poor for video, especially if you shoot it yourself. Does your video camera record to DVDs? If you have a mini-DV camera that is much higher quality (~25Mbit/s with frame accurate recording vs DVD's 3-5Mbit/s) and will last a lot longer.
I would assume that the amount of data you generate each year will increase with new technology, which will be directly off set by new storage technology. So your 100GB/year number will grow approximately with technology so that your new storage costs remain similar each year.
2. Storage period: 10-20 years
3. Time commitment.
Many of the suggestions here require constant maintenance. Generating 100GB per year is equal to a small content company with professional and paid IT expertise. Many of the already listed suggestions require you to maintain a RAID array or to check your media every few years for errors. Are you comfortable doing that? I know my parents don't have to check in on the pictures of my childhood every few years and do error correction. Are you OK knowing that if you messed up or skipped a maintenance period your memories could be lost or significantly damaged? In the long run, you probably want something that does not require active maintenance by you.
4. Cost.
Initial setup costs for the original read/write drive. Media costs, taking into account media lifespan and replacement/upgrade costs. Redundancy costs.
5. Redundancy.
Do you live in a Flood Plain? Do you have fire sprinklers? Are you OK with losing these digital memories to a catastrophic event? If not your solution will need to include geographic redundancy.
6. Privacy
Do you care about these memories being leaked? What about being subpoenaed? There are a bunch of "what if" scenarios I could come up with and I don't want to expand this already long post too much. 3rd parties = less privacy/more chance for abuse; storing the data digitally makes is possible and easier to access to be used against you somehow in the future.
My conclusions:
Personally, I've been dealing with this problem for a while and my research hasn't actually revealed a optimal solution. I don't have any children (yet) but I have generated a considerable amount of digital memories that I would like to secure long term. Your options as near as I can tell...
CD/DVD: Cheap initial costs, cheap media costs, questionable lifespan, high likelihood that if you can still access the media you will be able to transfer it to whatever new medium comes out every decade or so. Keep copies in two locations for geographic redundancy.
Hard Drives: RAID 1, 5, 10 (maybe 50 if you can afford it) are your most obvious and secure options. Find and follow standard industry practices, keeping a geographically separated identical copy somewhere far away.
Tape: High initial cost (several $1000's for the drive). Moderate Media Costs (tape cost varies with capacity). My research indicates longest shelf life with low maintenance for removable media. Still need geographically separated copies. Due to the long term nature of the medium, hopefully less problems with technology obsolesce. Still will probably struggle finding a replacement drive with the right connectors in 20 years.
Paper/Stone/Metal:
High initial cost. Large volume = large physical storage cost. Long life span. Probably larg
He's right though. Don't be mad at 900ftJesus just because he is expressing his love for saving other people's data. Granted we are talking about some guys home computer setup and not an enterprise level solution. Tape isn't a good choice for home users simply because of the cost. External hard drives and Flash media are the way to go in this guys case.
Mod me down with all of your hatred, and your journey towards the dark side will be complete!
I think that this question you can ask to Hollywood. I'm complety sure that this question has been made before by this people.
See the Vista Failure Log, 24% decline in revenue and the latest from Steve Ballmer.
Political torture and murder is not funny http://slashdot.org/comments.pl?sid=581079&cid=23757591
If you are willing to cut down your material, this might provide a solution. Perhaps the biggest problem is that these files are really "files forever". Even though you control their access, it makes me nervous to think they will always be "out there". Currently charging $2.50/gig or .01/4Mb.
link: http://wiki.dreamhost.com/Files_Forever#Files_Forever
I haven't studied this in depth, but you might find something of interest at the British Library.
http://www.bl.uk/aboutus/stratpolprog/ccare/introduction/digital/index.html#preservationchallenges
(Doubtless there are similar programmes in the USA, but I'm sure they're nowhere near as good.)
68 year old paper tapes for collosus are still readable and were used to test the replica built at Bletchly Park.
Alternatively 19C music box discs still work even now they use a time tested medium and can store a massive 100 Kbits each on a 1 foot disc.
Tape? Really? Did you actually say that with a straight face?? Try asking somebody in the audio industry about tape media that is over 25 years old. Especially if the environment is not PERFECT in every way.
And, over a certain size - RAID is a necessity. Of course you would use HotSpares... and cycle drives often. Your complete lack of knowledge shows when you describe corruption. You need to go lookup RAID and how the technology works. If you are trying to say that RAID will preserve corrupted data that is copied to it in the corrupted format -- well yeah -- I expect media to store what I give it. Otherwise you are spouting uninformed FUD.
Not trolling or flaming here - but I am truly astounded at your naivete....
I bet you have not read this story on more and more common robberies....
You really are so far gone that you don't see the problem with what you're saying here.
Web2.0: I love when people Flickr my cuil and digg my boingboing until my google is reddit and I start to yahoo
"My data is very important, so I'm going to buy a tape drive and not do this on the cheap. Let's head over to the flea market and see what we can find for half price!"
Yeah, that makes sense. :)
Socialism: a lie told by totalitarians and believed by fools.
That's the rule I always use.
Archival may mean lossless file format or a media that preserves high fidelity and is not prone to damage. That means whatever is available at the time. I save all my CDs (as good as they can be) in FLAC in several locations. Then I use that FLAC resulting media file to make smaller files.
But the basic rule, for me at least, is archival first then a lossy format. And always keep the archival format safe and make backups.
Be ready to expect the source archival files or media to be damaged or obsoleted. This is why you make back ups and switch to non-obsolete open media when the times demand it.
And verify data. Nothing is more heart breaking than losing original archives when moving them to new media results in file corruption because of faulty hardware.
I leave this idea open because I don't know what the latest archival format or media is ultimately going to be. All I know is FLAC (or other lossless codec) and hard drives for now, but even they're subject to wear, faults, damage, obsoleting, etc. But I'm using audio formats as a basis for explanation here. For video maybe that 4K format (DVD is lossy, you do realize)? And maybe that'll be fine for the next 20 years? Who knows.
The key is to be on top of what's available at the time as much as you can afford it and if you're willing to follow through.
I would take a used LTO drive from a good company over a new, factory-sealed SDLT that someone gave me for free.
Mediocrity knows nothing higher than itself; but talent instantly recognizes genius. -- Sir Arthur Conan Doyle
Man, you really have it in for SDLT. I tested them for years and never had problems beyond the stupid DLT4 drives and their dropped leaders. Just how many used LTO drives are you selling right now on EBay anyhow?
Socialism: a lie told by totalitarians and believed by fools.
What I essentially do is keep all of my photos and movies in Perforce, and then use Perforce on all of my computers to sync the data. It comes in handy when moving between computers, because all of my data shows up automatically. In addition, on my desktop, (which acts like a server,) I have an automated nightly script that copies my Perforce depot to another drive. Unlike you; I only generate about a gig or two of data per year, which is very reasonable.
When I do a project that generates a few gigs of data, I burn it to 2-3 DVDs. If one (or all) get corrupted, I'll still be able to get the data back by combining the readable files. It will be very easy for you to get into such a habit; when burning a DVD, just burn 1-2 extras and stick them in a temperature/humidity stable closet.
For my music collection, I have an automated nightly script that copies it to a second drive, and I manually copy it to my work computer. The collection is 80 gigs; and it's survived many hard drive crashes. I also keep backups of purchased music on burnt DVD or pressed CDs.
Another thing to consider is purchasing a 100+ gig solid state drive. It'll be pricey; but it's unlikely to loose your data.
No, I will not work for your startup
If you have high-speed internet, use a web backup service like Mozy. It very reasonably priced and backs up an unlimited amount of data for 1 computer.
Every few years copy your pictures & video to a new hard drive as you upgrade your hardware. The web backup can serve as a safety net if something goes wrong.
Remember that kids are like kittens, they get less cute with time, and their activities don't change so much because they are decreasing their learning percent day-to-day, until at age 15 they know everything. You will shoot fewer pictures and videos each year. Just remember to have someone else take some pictures, when you are dead and gone, and have preserved these memories, your kid will want to see you, not the other way round.
While the storage medium will probably change over the years, hopefully the data format won't have to change often, I would expect jpeg to endure, but you might want a 2nd copy of videos saved in frame by frame stills plus wav audio.
Mc900ftjesus sounds like the guy who lost his job maintaining tapes because the company switched to RAID.
First, I think it is important to take a look at possible expected data storage needs. At 100GB in one year, it would not be unreasonable for it to at least double every year due to increasing resolutions, vacations, digital copies of art work, etc. At 2**(number of years) * 100GB, I estimate about 50 TB in 10 years. (Ya, I know that sounds like a lot, but 10 years ago no-one thought they would ever need more than 2 GB of hard drive space). With that said, it is not practical to create a single storage system with today's tools that will last 10 years. At best you can make a practical system last 5 to 6 years with these kinds of storage demands.
Mc900ftjesus did bring up several short comings but all can be overcome or also exist when written to tape. The argument corruption written to one disk is written to all disk is the same with tape. For the failed controller argument, replacement controllers can be purchased if using hardware RAID, but tapes can go bad, as can the tape drive. If concerned about future availability of drives or controllers, purchase two extra and a few extra hard drives. When using Raid 1,5,6 or 1+0 with a hardware controller, there exists built in fault tolerance that allows for one, two, or more hard drives (depending on the raid configuration), controllers, motherboards, and even operating systems to go bad if you have the knowledge to deal with these challenges.
I would point out that RAID should not be the only backup system though. As with any good disaster recovery plan, additional offsite storage in case of fire, robbery or any other kind of disaster that could destroy the whole machine.
I would also say that tape has many problems in itself, such as the inability to verify or retrieve data easily. Imagine going through a 3 TB of 400GB tape to find that image you accidentally just deleted from your main computer that you know was in the 'Grandma and Grandpa X-mas 2008' folder, but you don't know the exact file name . Also, for large amounts of data, RAID is likely to be much cheaper with even the most redundant forms of RAID. If you want to test this cost theory and want to include replacement parts for the RAID system (extra controllers and drives), please do the same with the tape backup system. I would also point out that tape does have a shelf life, even in the best environments, and should not be the sole backup.
In my personal opinion, tape is only for last resort disaster recovery, but still plays an important role. I hope mc900ftjesus does not fire me for this.
Even if the physical media are still readable, don't forget to think about storing and updating your video in file formats which will still be readable by future apps. I have text documents written in MS Works and the like on a Mac Classic in the early 90's which still exist, having been copied from drive to drive over the years, but are barely if at all readable.
Also, the BBC Domesday project is an interesting example of what can happen if you store something away in a proprietary format for too long.