Slashdot Mirror
Long-Term Personal Data Storage?
BeanBagKing writes "Yesterday I set out in search of a way to store my documents, videos, and pictures for a long time without worrying about them. This is stuff that I may not care about for years, I don't care where it is, or if it's immediately available, so long as when I do decide to get it, it's there. What did I come up with? Nothing. Hard Drives can fail or degrade. CD's and DVD's I've read have the same problem over long periods of time. I'd rather not pay yearly rent on a server or backup/storage solution. I could start my own server, but that goes back to the issue of hard drives failing, not to mention cost. Tape backups aren't common for personal backups, making far-future retrieval possibly difficult, not to mention the low storage capacity of tape drives. I've thought about buying a bunch of 4GB thumb drives; I've had some of those for years and even sent a few through washers and driers and had the data survive. Do you have any suggestions? My requirements are simple: It must be stable, lasting for decades if possible, and must be as inexpensive as possible. I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me."
Hard drives, while they may fail, are still probably your best chance. Using RAID-1 or -5, you can keep the drives running (possibly intermittently) and can avoid failure. With the rate of hard drive growth, you can just replace them with bigger drives when the time comes you need more space. It isn't exactly the same as throwing them in a cold room and forgetting them, but it isn't too expensive either.
Waffles rock.
Parchment.
We don't have enough history on this tech to know what, if anything, will "last for decades". Possibly "paper" and "microfiche" might fit in that list, but those aren't the sort of things you're talking about. Best option I can think of right now would be to get a couple 500gig drives, put everything on both, and then put them in different areas. In 3-5 years, back them up to something newer, and repeat that every 3-5 years. Maybe in those intervening years, we'll have more data and newer tech that's demonstrably suited for what your needs are.
creation science book
Obviously this question hasn't been answered for the general public because this is like the 4th year in a row that this question has been asked on Slashdot.
Long term:
Use quality DVDs. Redo the backup on a schedule such that everything is re-backed up every three years or so. Every month, say, you make one DVD. Keep the backups in a climate controlled, dark, secure place, such as a safe deposit box at the bank.
Short term:
Back up everything you want to save to an external hard drive weekly. Every three months swap it with a drive kept in the safe deposit box.
Daily:
If you have a Mac, use Time Machine. If Linux, some sort of cron job running a Python script that copies /home to an external hard drive. If Windows, I dunno.
Best Slashdot Co
How many times has this question been asked on Slashdot? I swear, it shows up on the front page at least three times a year.
As for the question itself, the answer is pretty simple, but unhelpful. Basically what it comes down to is that there is no safe place for your data. You're asking for the best type of basket to put all your eggs in. If you look at it that way, the solution is easier to arrive at. Your choices are A) spare no expense and build/buy the world's strongest basket and pray no flaw arises, or B) start copying your eggs around to all sorts of cheap baskets and continuously add more baskets in the expectation that the oldest baskets are going to fail.
Copy all your stuff to all your computers. Burn to DVD and/or CD ROM. Buy SD cards and USB flash drives. High capacity storage devices are so cheap now that you can keep all your valuable pictures of your vacation to Cleveland quite safe by constant duplication. That's the value of digital. Copies are perfect. Make lots.
If a job's not worth doing, it's not worth doing right.
Here's the thing. Flash drives will *probably* last long enough. I wouldn't at all be surprised if they were still readable in 20 or 30 years. But a) what's the odds of your current WinUx or iHoloTablet having a usb connector in 30 years? and MUCH more importantly, what's the odds of having anything capable of reading those historic Word 2007, Acrobat 5, or any other type of file format in 20 years? Yes, there are some folks technical enough that they can still read and readily interact with Geoworks, Wordstar, Xywrite, etc. stored on 8" floppy disks. But if you ain't one of them, and I'm happy to admit I ain't, the fact that the flash drive is physically capable of being read in 30 years simply won't matter. That's why I crack up reading various vendors' claims of CDs, DVDs, BDs etc. lasting 50 or 100 or more years. The disks will be readable but you'll have no mechanical or logical way to read what's on them.
Remember paper? That actually roots after a period of time.
That's why paper is specifically prohibited on my network.
It's impossible for guarantee 100% storage integrity, just like it's impossible to guarantee 100% uptime. What you want to ask is what risk of data loss you are willing to take.
This page compares some of the options in terms of Mean Time To Data Loss (MTTDL). For the amount of space you're looking at (~500gb), a three-way mirror is probably sufficient to last for your lifetime.
But there's always the risk of fat-fingering "rm -rf" or having the building catch fire, so maybe you want to have two synchronized sets of mirrors, stored in different physical locations. Only you can decide if that's too paranoid for you (or not paranoid enough).
But seriously, I've had the same (but growing) data set in my /home for over 15 years, and going. I find the easiest way is to just keep it on my drive, and have a few frequently updated copies on external media (optical or solid or dirve) and to keep it on another PC too, disk space is so goddamn cheap. I also have a large music collection, and instead of wasting time backing it up onto optical media, I just keep it on both my notebook and PC, its unlikely both will fail at the same time, and incase of a robbery, I can also archive it at work.
Don't expect any form of media to last forever, it's multiple, frequently updated copies that will ensure your data lasts forever.
Also, if you have friends and family you can trust, make a copy for them to keep for you, off-site backup is also important.
Obviously this all depends on how important and/or private the data is.
my 0.2
As the tag implies, Gmail is your friend. 7 gigs per account, searchable, accessible from any connected computer, free, and if in the future, google starts to decline, you can transfer to their replacement.
There is absolutely nothing that you can put away for decades and expect to be useful. Your requirements are not simple - they'll actually very, very hard to meet, even if you want to throw a lot of money at the problem.
You don't know that a jpeg, for example, will be readable in 30 years. The format may be so deprecated that there might not even be a viewer available. Like my old Microsoft Works 4.0 documents - although I have the data, I have nothing that can read them unless I want to spin up an old Windows image, assuming that I can generate a virtualized environment that can support an old Windows (Windows XP probably won't even boot on any PC being produced 30 years from now). And some of that data is only a few years old, not decades old.
You should store not only the data, but also the applications that created the data. And the computer you need to run those applications. And backups of those. And then every few years, pull it all back and validate it and update as required.
You may have only 500GB now, but 10 years from now that will be 5TB. And then you need a way to actually be able to find something you added to your "archive".
I deal with this at work regularly. An archive is not a backup that you keep for a long time. It's much, much more than that. Once you start thinking about all of the issues that come up, you'll see that the media is the least of your problems.
Its not really that cheap, and not that simple to use for personal backups. Unless you are willing to write your own backup scripts, its going to be a headache.
Querying S3 for a list of stored files is *very* slow, and you only get 1k results per query. This means you have to index what files you put in S3 in a local db. This allows you to ask the db what files are there (and how to grab them).
If you only have a few files you can use the S3 browser extension for Firefox (or one of a many file system mounting, ftp simulating, etc tools). Just keep in mind the 1k file limit per query and box things in folders of no more than 1k items. Otherwise you will have a very slow browsing experience.
I have around 120 GB of family photos and purchased mp3s that I would like to store. To store 120 GB at .15 per gigabyte/month for 1 year would cost me: $216 (at $18 a month).
We use it where I work, with great success, but it would be much to much work for me for a personal backup system.
Considering the cost, I would go with a consumer targeted app (there are LOTS of them). A number of them charge a flat flee for "unlimited" storage. Beware of how you interface them. Some support windows only.
I believe this to be a serious problem with no good solution currently. That's the truth. You'll get lots of dismissive posts saying it's no big deal, but it is.
Forget media integrity. The problem is technology drift. Everyone thinks "ubiquitous" (as in every computer has a USB port) is the same as "eternal," and it isn't. Twenty years from now, your USB thumb drives and CD-R's may have their data physically intact, but only museums will have equipment that can read them.
It is a fantasy to suppose that you can successfully perform Sisyphus-like task of systematically recopying your data to new media and formats. The proof of this is the innumerable stories of big, well-funded organizations that have neglected to do this. If the NASAs of the world keep finding reels of tape with important data on it that can't be read due to technology skew, what makes you think that you can do much better?
(What makes me bitter is failure of vendors to give adequate warning when software updates remove the capabilities of reading file formats that were formerly supported. I once verified that my new Mac could read my old MFS diskettes, and did not notice when a software update to the OS removed that capability. Microsoft was less than forthcoming when they removed the built-in ability of Excel to read Multiplan files).
"How to Do Nothing," kids activities, back in print!
You are the analogy of an investor who wants a high-yield, low-risk, completely liquid instrument. The term is TANSTAAFL.
I maintain two (yes, two) USB external drives. Every couple of years, I migrate to a larger, or otherwise better medium. I use an incremental backup system (for me, cpio) that ends up keeping too much stuff, but at least I have the stuff I want if I need to get to it.
In a decade - in my case, four decades - one can accumulate a remarkable amount of crap, along with things one truly wants to save. I have a total of about 90 gig of actual data, plus a far larger amount of music and video, which I consider more or less disposable. It is not difficult, nor expensive, to purchase another external drive and copy the data. My oldest backup is on IBM 2314 disk pack, but the data still held on that disk is also present on my current backup, a WD 160G in a USB-1 enclosure. Sometime next year, I'll go to a 500 G drive in a USB-2 enclosure.
An important consideration is to periodically check to see that the data ostensibly held on a drive (or CD, or DVD) is actually readable. DVD/RW in particular has a tendency to get flakey over long periods of time, expecially if stored under adverse conditions (jammed in back of desk drawer, under sixteen pair of scissors, stapler, a box of pop-tarts, and four old coffee cups. I always keep my last few generations of backups, and if I find an unreadable datum, I make an effort to recover it from the previous backup.
While it may be stating the obvious, it's a Bad Idea (TM) to wait to back up data until you have a problem. I back up all of my data every week or two, and critical data, daily, without fail. Critical data is cached as a three-generation dataset (IBMese).
Good luck. There are no real solutions, just ways to cope.
Don't take life too seriously; it isn't permanent.
That is an interesting suggestion, but 500GB would cost $900/year (plus transfer costs) which I don't call "dirt cheap". As far as being there forever... who knows if there will be an Amazon in 10 years? Amazon might be more stable than most hosting options, but forever is a long time.
1) Encrypt your data
2) Tack it onto the end of some anime or porno mpg
3) upload to kazaa/gnutella/whatever
It'll still be circulating the net long after our grandchildren are dead.
Very durable. Write speed is a bit slow though...
If you buy quality DVDs and take good care of them they will quite probably last for decades, perhaps half a century. They are expected to degrade over many years but some of the CDs written back when CDs were first invented are still readable today so nobody really knows how long they might last. There is a similar problem for HDDs, while in constant use MTBFs are well established, but for a HDD that is written to and then left unpowered for many years, well again nobody really knows because we haven't observed it yet. I'd say go for both, obviously HDDs have the massive advantage that you can plug one 500GB and write all your data to it all in one go. To store that much on DVDs will take you days or weeks to write to each disc one or even two at a time. I know, I have over 1.5TB of data backed up on DVDs which number over 500 already.
right up until you have an enviroment diasaster.
(enviroment can be as small as a tiny fire in the power supply of that PC)
Theft of the PC? are you covered?
FFS it is IN THE SAME CASE!
every day http://en.wikipedia.org/wiki/Special:Random
Having a tape around for 20 years doesn't do you any good. 20 years ago, I was writing to 1600 and 6250bpi tapes. Today, my data center doesn't even have a drive that can physically read them.
Today's tape technology is no different. 3 years ago was writing to SDLT tapes. By next year, I won't even have an SDLT drive in my data center, having migrated everything over to LTO.
Yeah, I have round tapes in my offsite storage. I have 4mm and DAT tapes out there. We're just wasting money storing the media, since we have nothing that can read them.
If I could read the old media and extract a really old database, would today's database app be able to read it? Probably not. And could I install that app on today's OS? Probably not. And could I install the OS from many years ago on today's hardware? Probably not. Could I compile source from 20 years ago with today's compilers? In many cases, actually I can't. And if it really did all magically get compiled, is anybody around that can still knows how to run the app?
Don't forget that 20 years ago, many systems didn't have TCP/IP installed. In 1988, mine didn't - it was a combination of RS232-attached terminals and XNS-attached graphics workstations. Drive sizes were 80-160MB. A couple of MB of memory was a lot.
For those of you not still in school, ask around and see how many folks in your IT department can name the server that held your financial data 10 years ago.
By the way, I *have* had an SD card fail. It was in my digital camera the whole time, worked fine for a couple years, then quit. The camera itself showed no sign of damage, so I don't think it was abused. It was a Kingston, too, which I consider reputable.
Depends on your definition of 'personal' files. Video can take up a lot of space.
... honestly, Slashdot - and others - have covered this time and time again. Nothing has changed. There still isn't a cheap digital storage medium that we know for sure -will- retain your data -and- be readable (in terms of media -and- the hardware to read that media) down to the very last bit for your grandchildren.
IF and when there's a breakthrough, I'm sure Slashdot is one of the first places you'll hear about it.. but it won't be in an answer to an Ask Slashdot.
http://it.slashdot.org/article.pl?sid=07/04/21/1257249 - Digital Media Archiving Challenges Hollywood .. and so forth and so on.
http://hardware.slashdot.org/article.pl?sid=06/11/20/2036247 - Archiving Digital Data an Unsolved Problem
http://hardware.slashdot.org/hardware/06/12/11/1714232.shtml - How To Choose Archival CD/DVD Media
http://hardware.slashdot.org/article.pl?sid=05/06/26/218250&from=rss - Archiving Digital History at the NARA
http://tech.slashdot.org/article.pl?sid=07/03/31/2141204 - How To Properly Archive Data On Disc Media
Yes, I realize that you stated "I'm not looking to start my own national archive; I have less than 500GBs and only save things important to me". However, it doesn't really matter whether you're archiving hollywood movies, NASA records or just your own random crap. If it is important to you - important enough that you want it to be "lasting for decades if possible" - then your concerns are the same as NASA's... and they're struggling with the exact same question.
The 'best' answer so far is one you will find in each and every single discussion on this - including this thread, so I'll just point you there:
http://ask.slashdot.org/comments.pl?sid=1061489&cid=26102825
You mentioned 'cheap', as otherwise all the answers saying "duuuuude, ditch the digital - go analog!" might have some validity.. take a wild guess as to what it would cost to have thousands of photos transferred to negatives/prints, or video transferred to tape/film, etc. Plus you mentioned documents.. some of those may not transfer to e.g. paper (easily) at all depending on the 'documents' in question; e.g. CAD files.
Jungledisk takes care of all the tedious backup stuff for you, and it is only a one time charge for the app.
But you're right, S3 isn't cheap. To store 500 GB of data would be about $75 a month, plus the $50 to put it on the server in the first place.
[From Slashwayback]: Dear Keypunch, I have data I want to keep for decades. Should I invest in a good card reader, or should I transfer my data to these far more efficient but newfangled "floppy disks"?
It's pretty ridiculous to expect one storage format to be viable for 'decades'. Not because it goes bad (even though it probably does), but because you're not likely to be able to maintain the necessary equipment for that long. If you find a storage solution, you need a retrieval solution to equal it. What equipment will you be able to find decades on that can access your storage, even if it stays good? You have no idea.
I've been maintaining a collection of Apple IIs and recopying the programs and data regularly (mostly through full HD backup, reformat with error block deletion, reformatting and replacing) to keep it readable. I have machines and data between 20 and 30 years old. I recognized long ago this had become a hobby in its own right, as most of what I had hasn't been of interest to me for many years. The little bits that have been useful have been transferred to newer machines and formats several times. That's decreased as more and more of it can be found easily on the web (previously FTP/gopher/etc.).
Get used to transferring your data to new formats as they come into widespread use, and recopying as necessary to keep them readable. Or else:
[From Slashwayforward] Dear Galactic EM Field Computing, I just found about 20 pounds of aluminized plastic disks that used to have data on them, but I can't read them to tell if I still want it. Is there any museum that might want these? Or are there still any operating plastic recycling centers that might give me a few bucks for them?
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
Actually Flash/memory drives are sensitive to radiation. Long term storage without regularly accessing the drive can lead to situations where blocks go bad beyond the ECC/CRC capabilities of the drive to fix. If you intend to store valuable data on memory devices for the long term you should (a) use multiple redundant drives (b) use a file-system with block-level ECC/CRC error correction and redundancy (like ZFS) (c) write each block to the device twice in different location (i.e. an mirror on the drive).
The future of Flash memory is such that unless they extend the ECC/CRC capabilities of the controller, the susceptibility of these devices for radiation will increase when the cells get smaller.
In case anybody doubts the impact of radiation on electronic devices, here is an interesting experiment you can do: take your digital camera, put the lens cap on and do timed exposure with increasing exposure times (1,2,4,8, ... seconds). Then analyse these pictures for bad-pixels, or better, subtract the pictures from each other. The random bits scattered around on these frames are impacts of cosmic rays. Now apply the same principle on memory devices with much longer exposure times...
To cut my somewhat rambling post short: use memory devices as long term storage? No. Not without thought about the required data reliability.
I think the issue is that people are thinking about this incorrectly. You don't really want to 'archive' this data -- keep it with you! Keep it with all of the data that you are using day to day and back it up and move it along with that.
My home workstation still has files from 15 years ago on it. I've replaced the computer many times, had a few hard drives fail, etc. but I've always restored both current and 'archive' data from backups and kept going.
So start building VM's of operating systems and software that are in use. Archive those. Far from perfect or complete, but it should narrow the scope of the problem a little bit.
As far as personal stuff, I think the best solution is to have 2 or more live copies of all important data and just migrate them to whatever makes the most sense at a given point of time, and then also have backups of stuff. That doesn't work with the question, but there isn't really a cheap answer to the question at this point.
Nerd rage is the funniest rage.
I have gigs of photos (wedding, long lost family picnics, etc) and music that I can't bare to lose.
Find someone who doesn't want to lose anything either and setup rsync over ssh. Synchronize often, rsync is very friendly to bandwidth.
The OP specified "decades if possible." So we have two problems here. One is that 30 years at $75/mo comes out to $27,000, which is a little pricey. For that amount of money he could probably hire someone to come to his house once a year, verify the readability of his media for him, and transfer them to new media as the old ones become obsolete. $1,000 for a few hours' work? I'd take it.
The other problem is that the probability that Amazon S3 will exist in 30 years is very low. This is basically the problem with any possible answer to his question. There isn't any computer-related service or equipment that you can be sure will still be there in 30 years. A more realistic goal would be to do it in 10-year steps; if that's all he wants, then the shoebox full of flash drives should work fine, and then 10 years from now he can transfer those data to something else.
I initially misread that as 500 Mb, which is about the amount of critical data I have that needs backing up. 500 gigabytes is kind of a crazy amount of data. One way to get that much might be that he has a gigantic collection of mp3s, or possibly a moderately huge collection of music in a less lossy format. But then that's not critical personal data, it's just a music collection. And the chances are that as the decades go by, he'll realize that the music he thought was so important and wonderful in 2008 no longer seems so important to him. I know plenty of people who still have their Kool and the Gang LP's from the 1970's, but it's not like they're willing to spend a thousand dollars a year to obsessively maintain them.
Find free books.
At my work (18,000 desktop Nat'l research lab), we have a special group to keep old hardware around, just so they can go and pull data from old archives.
I drank what? -- Socrates
You can definitely do that today with DOS and Windows 3.1 on todays hardware. If we move away from x86 architecture, then everything is out the window, but for now, yes, we can install OS from many years ago on todays hardware.
Tried that lately? Not sure about DOS and Win3.1, but certainly Win95 had a boatload of timing issues on newer hardware.
If your data is not in your possession, how do you know others won't see it or edit it without your permission? For archiving purposes, the best technology is magneto-optical. Despite the fact that makers have been exiting the market due to competition from faster and larger-capacity technologies, MO remains the champ for data storage duration. Remember, it is partly based on a Natural phenomenon that lets geophysicists detect which way the Earth's magnetic field was oriented, hundreds of millions of years ago --data retention just doesn't get much more long-term than that. MO disks are removable from the drive, and every modern drive can read any older same-size disk (they come in the standard 3.5" and 5.25" sizes, but have quite a range of capacities), so if the drive fails, just make sure you have spares. Perhaps, sooner rather than later, the manufacturers will realize that archival storage is a niche market that will demand that they stay in the MO business.
There really is a simple way around this - and it is what I've done - I've got data 25 years old and it's still relatively easily manipulated with a little work. I've found floppy disks are relatively resilient, and old hard drives seem to keep their data for a long time. I've got a computer, display, keyboard, and associated peripherals stored for every generation of data that I kept:
1.I have a Commodore 64 with floppy drive and cassette drive stored in a box with the floppy disks and cassettes from that generation (late 70s/early 80s).
2.I have an IBM PC/XT with keyboard, a 5 1/4" floppy, 3 1/2" floppy, internal 20MB hard drive, and CGA monitor stored in a box with a load of 5 1/4" floppies filled with data from that generation (Mid 80s).
3.I have an IBM RS/6000 with display, keyboard, and mouse and internal 500MB hard drive loaded with all my docs and projects from that generation (early 90s).
4.I have a Pentium 2/300 PC * 15" monitor with windows 98, CD R/W drive, 3 1/2" floppy drive, and USB ports - and a crapload of CD's and 3 1/2" floppies full of stuff from that generation (Mid/late 90s).
When the current generation looks like it's going to be moving on, I'll put away a Core 2 Duo system with 1 TB of hard drive full of stuff with the different OS's I used loaded on it with boot manager (Ubuntu, XP, FreeBSD), a crapload of USB keys full of documents, along with burned DVDs etc. That'll take care of the "'00" generation.
The answer lies in not only archiving your data "of the generation" but the essential equipment needed to access it. I may have a heck of a time moving data off of my Commodore 64 - but I can at least see it and access it - I believe I stored a modem with it - so at worse I could set up a terminal server that it could dial into and dump data to. All the other systems I'm pretty sure I could recover stuff from - even if the PC/XT does have an MFM hard drive, etc.
I have data 18+ years old . You're approach is admirable, but why not just move your data forward with technology?
When floppies started dieing, hard drives got large enough so I moved all data off the floppies to hard drives and optical media. When word processing software I used started dieing, I moved all my documents or obtained converters to MS Word format. Also, I don't archive music and movies. I do archive pictures etc.
I believe the best approach is just keep your data moving forward & current and not in some archaic format. This means I have 3 redundant copies of all my data on hard drives using a current OS.
Granted I don't archive "silly things" like music which I can re-create; but rather just personal data (i.e. personal documents, pictures, personal videos) so the total quantity of data after 18+ years is only about 13gb.
I think your post is very insightful, and I have an additional problem to throw into the mix: sorting through all the crap you've archived, even assuming you can read it all.
I don't know about you, but I've run lots of different backups on lots of different systems, and one of the problems that always comes up is just finding the revision of the file you want. People say, "I want the copy before I made this revision-- I think I did that about a month ago." Check the backups and there are no revisions from a month ago, but there are 20 from the month before. Next thing you know you're checking 20 copies by hand, and none of them are what you're looking for-- and that's even when your backup/archive system is working.
So when devising any kind of archive, I think it's at least worth considering, "How am I going to find what I'm looking for in 20 years?" Imagine yourself in 20 years, and you have every piece of data you've ever generated stored on some kind of media that holds hundreds of terabytes of data. You want to find some spreadsheet you made today (20 years ago). Maybe you don't remember exactly when you made the document-- you think about 15 years ago, but it's actually 20. You can't really remember what the filename was. You can't remember if you made it in Excel or OpenOffice, so you're not even sure what filetype you're looking for. What's going to be your method for finding that file?
I'm not suggesting it's an insoluble problem. It might be that it's not even a problem in 20 years because indexing/searching has become so good that your AI will be able to sort through terabytes in a couple seconds and make some good guesses about what you're looking for, but do you really want to rely on that happening?
For 500 gigs.
A couple of hard disks, stored in different locations (cities, not drawers), that you update+check once a month or a quarter.
Burn DVDS of the really important stuff (pictures, documents) around once a month, and mail them to your parents/family.
What NOT to do:
- RAID is NOT a backup solution, it is a high-availability solution. Of all the problems bakcups need to adress (theft, destruction, viruses...), it solves very few.
- don't keep your backups online and/or in the same spot: viruses, power surges, fires, theft... will destroy them
- don't have only ONE backup: Murphy's law, if your live data disapears, the backup will turn bad also
- don't forget to check that your backups are still good
- don't delude yourself into thinking that any physical media in use today will still be easily readable more than 5 years from now. (except for the consumer type media: CDs, DVDs)
The Cloud - because you don't care if your apps and data are up in the air.
Sure, you can easily shoot hundreds of pictures a night. But, FFS, sort those suckers! Maybe five of those are good shots, and if you're lucky one of them is a great shot.
Just toss the rest! Really! Nobody cares for the reams of out-of-focus or incorrect compositions.
Keep the great shots (one in a hundred, if you're a good photographer) and delete te rest.
What the OP needs to understand is that someone has to do periodic maintenance over those decades. Either he does it himself or hires someone (either directly or as subscription to a service). All archiving requires monitoring of the storage media and preventative transfer to newer media before the information is lost. Given that media is not perfectly predictable, you need redundancy and frequent monitoring too, so you can discover a data loss and restore redundancy before all media sets are lost.
Using flash/solid state drives is probably going to improve the shelf-life of a storage copy. But you still have to monitor for loss. It doesn't reduce the chances of theft, fire, flood or simple misplacement.
I do it myself, with cheap Linux software RAID boxes in separate locations, piggybacking on existing Internet services to do rsync mirroring between the sites. The machines are kept alive, running disk scans. The rsync mirror process is also periodically accessing the files at application level. I may add some cron job to do checksum verification some day.
The maintenance cost is the power supply, internet service, eventual hard disk replacements (for failure or size increase) and my time. It is the cheapest solution for me that has the level of reliability I can understand and control myself. It is affordable because I have a friendly site letting me colo my remote box for free/good will. I don't pay their internet or electricity bill. I am geeky enough to consider the several hours per year of effort to be part of my computer hobby.
My solution has evolved to this point and run stable for the past 5 years or so. Prior to that I did similar concepts but with less reliable equipment for each mirror, e.g. depending on fileservers at school and work that I could piggyback on with a few tar files, and just using single-disk machines locally. I've propagated my important date in "online" form for about 15 years now. Anything from before that was floppy disks and is lost to me, but also doesn't matter because we're getting back to teenage years by then.
Yeah - sure - until Amazon goes out of business or gets bought and then the new owner dumps the service and you're S.O.L.
RS
Shoes for Industry. Shoes for the Dead.
This actually is a good idea. If the porn files were maybe one half sex imagery and one half encrypted private data, and there was no easy way to separate the two halves, then people would download, store, and upload the files in order to view the porn. Anyone who had data in the private section of the file could download it from various P2P sites.
The cost of filming and creating the porn file would be covered by the people who would be using the file for long-term distributed storage. Say a 1.5Gigabyte file that was an hour of MP4 video entertainment and 500 megabytes of distributed storage. The fees received by the producers for the storage would pay for the video production costs. Since porn is cheap to produce, this may solve the problem of piracy and secure storage at the same time.
> I would figure that it's video... specifically, probably family home movies.
Digital video has opened a HUGE new can of worms. We have problems even today viewing video created just yesterday (especially over the Web) because of all the myriad codec standards.
Imagine what it'll be like in 20 years-- anything other than NTSC, PAL, or SECAM will be effectively extinct.
"Slow down, Cowboy! It has been 3 years, 7 months and 26 days since you last successfully posted a comment."
If NAND flash SSD lifetimes are determined by write frequency, then wouldn't this be fantastic for archival storage? Just write the data once, then read it as many times as you like.
Which, of course, refers to the process of getting the data (in binary form) tattooed on your body by tattoo artist Tegan Stadnyck of Seabrook, New Hampshire.
Of course Amazon will always be there, just like Ford and GM....
[ ]Half Empty [ ]Half Full [x]Twice as big as it needs to be
Ok, dudes, if he's got radiation problems, he's got bigger problems than data storage.
Main difference between the BSD license and the GPL license: one is from California and the other is from Massachusetts
I have trouble with this in my mind, since so much of my work is devoted to making sure that information always exists and is accessible all the time. I look at these personal solutions for backup, and I'm so used to evaluating enterprise-type products that I scoff.
I guess that I don't know anymore what is appropriate for 'home users' when they say they want to keep data long-term. The submitter stated that tape drives were inadequate when that's still the most reliable method that enterprises use.
Sure, there are VTLs, but to not keep your data offsite as well would be counterproductive. I guess I just don't know what the submitter wants.
Check out my sysadmin blog!
A lot of people shoot home movies and then become obsessed with preserving the footage. That adds up fast.
It seems that deciding what isn't important is a hard part of backing up.
After a loved one dies, even the lowest quality outtake with a thumb covering half the shot can be a priceless memory.
There really isn't a good solution right now.
After thinking about it a while I realized that:
1) Most of the stuff on my computers could be replaced.
2) The one thing I would really, really hate to loose are family photo's.
3) Hard drives WILL fail sooner or later.
4) Tapes are reliable for a while but even in a climate controlled vault I have had tapes at work end up bad after a couple of years. (not to mention the pain in the neck it is to find a working legacy tape drive after 10 years)
5) DVD's will probably have the same issues.
My solution for now is redundancy.
Digital photos get offloaded to my Linux pc. I use a program called Digicam. .5T USB external disk drives with the "one button backup" program that is set to run nightly.
I have a bash script that syncs the new photos to a Windows share on my wifes pc.
My wife has one of those
When I have a couple of new directories of photos I run another script that compresses the whole directory and splits the output into a bunch of 45 megabyte rar archives.
I then upload them to Microsoft's free "Skydrive". Microsoft just upped their free disk storage to 25GB.
I also have some documents saved on the free AOL Xdrive.
I figure in a couple of years there will be a better long term storage option. It will probably be something like a solid state drive that lasts for two hundred years. At that point I will save everything to that and store it in my safety deposit box at my bank.
Every wrong attempt discarded is a step forward - T. Edison