Slashdot Mirror
Storing Data For the Next 1,000 Years
An anonymous reader writes "This may be an interesting take on creating long-term storage technologies. A team of researchers at UCSC claims to have come up with a power-efficient, scalable way to reliably store data for a theoretical 1,400 years with regular hard drives. TG Daily has an article describing this technology and it sounds intriguing as it uses self-contained but networked storage units. It looks like a complicated solution, but the approach is manageable and may be an effective solution to preserve your data for decades and possibly centuries." Nice to see research on this using the kinds of real-world figures for disk lifetimes that recent studies have been turning up.
Part of the solution to very long-term storage, of course, has to involve a method to read the data you've archived.
:)
I tend to think systems such as the one described in the article aren't good long-term solutions. If their math works on the failure rates, that's fantastic- but just try to hook up a 2028 computer to one of these things to pull the data off.*
(Ever tried to get data off an obsolete tape backup?)
I think the most reliable archival system is going to be an active one, where data is saved on modern storage hardware and always copied to more modern tech as it arrives.
The other side of this is, for anything more advanced than text-- given that you can get at the data, what do you open it with? File types die over time and it's basically impossible to find programs to open certain files nowadays, much less such programs that will run on a modern OS. I think the answer to this has to be virtualization. Store the data *and* programs that can open the filetypes you need opened inside a portable virtual machine (e.g., a Windows vmware image). Over time, you may have to layer virtual machines inside virtual machines as OSes grow obsolete. But that's okay- virtualization is only going to become more elegant, and the end result is that you'd have your data in its original environment, completely accessible by native programs.
*Some elements of this problem could be solved by having backup servers use wireless and filesharing protocols that might stand the test of time- e.g., 802.11n and SAMBA. No need to just pick one 'most likely to be future-proof' combination, either: run bluetooth and serial access, webdav and a http fileserver, etc. Still, *not* storing data on modern hardware is always going to be a risky kludge.
There's probably room for a lucrative business based around this-- figuring out the most elegant way to archive and retain meaningful access to data under various computing/disaster scenarios. Hey, I do consulting.
Cave paintings work too.
No, not punch cards... but close!
Stone and chisel. That's the way to store data for 1,000 years. The reason why I say this is simple. The more "religious" the world's populations become, the closer to the dark ages we become. (The reverse is true as well as history illustrates.) I expect there will be a second "dark ages" at which point all other technologies will simply not be available.
From TFA:
Santa Cruz (CA) - Have you ever thought how vulnerable your data may be through the simple fact that you may be storing your entire digital life on a single hard drive? On single drive can hold tens of thousands of pictures, thousands of music files, videos, letters and countless other documents. One malfunctioning drive can wipe out your virtual life in a blink of an eye. A scary thought. On a greater scale, at least portions of the digital information describing our generation may be put at risk by current storage technologies. There are only a few decades of life in tape and disk storage these days, but a team of researchers claims to have come up with a power-efficient, scalable way to reliably store data with regular hard drives for an estimated (theoretical) 1400 years.
My "digital life"? Scary to lose it? Man.. these people never heard of backups, or having a real life, eh? Jeez, I can store my whole "digital life" on a 1 gig USB key, with room to spare.
I've lost my backups more times than I can count, my computers are toys, mostly for communication and play. Amazing how many people put their whole LIVES on a hard disk. Remarkable actually. What would I lose? About a dozen passwords and I'd need to reinstall and re-customize my system... OH WAIT... I backed up the important scripts and source code to a DVD.. TWO in fact. Bummer, guess I don't have to cry endless tears over the loss of my "digital life".
" What luck for rulers that men do not think" - Adolf Hitler
http://hardware.slashdot.org/article.pl?sid=07/02/21/004233
Almost as good as clay tablets or pyramids but easier to manage and with a higher data density. Cool!
tm
Support TBI Research: http://www.raisinhope.org
Since there will be many holes shot into this theory, let me be one of the first to fire a shot. Electricity (as we know it) may not be around then. I am not predicting the dark ages, but who's to say that far in advance there is still a live socket.
Any storage device that relies on outside power cannot be guaranteed for 100 years, let alone 1400. I would have more faith in a stone tablet.
This is a fine example of "academic" research dollars at work.
Flexible bare-metal recovery for Linux/UNIX
This is SO scary! I had just been looking at Wikipedia looking up some obscure phenomenon, and went over to Slashdot. While the page is loading my thoughts drift and I think how important isn't Wikipedia, for now and the future. Someone should print it out and... What? A Slashdot article claims that someone will print the German edition. I manage to collect my thoughts and login, and, notice THIS article... I'm drifting in a black void by now... We wikipedians have come to bring you back home... Sorry for some digressions. Still, the digital age must be preserved. How? Continuously updating and using data! No more diskstorage of WordPerfecrt 1.17 files for ppc Mac 2.23 in a damp basement. Use archive proof paper? 10,000,000,000 trees a months? Oh, what isn't the price for being future proof?
Did anyone else notice that the lead researcher's name is Mark Storer? How perfect is that?
How long will this array take to fill up the first time around? A 10 PB storage system could be built for about $4700 with an annual operational cost (power for running and cooling the system) of about $50. Unless 10 PB (petabytes) means something other than what I think (10,000 terabytes), where did they get the $4700 number?
I even read their definition of static cost (You have to go up a few paragraphs) and I still don't know.
[Fuck Beta]
o0t!
One thing remains constant in thousands of years of recovered cave paintings, manuscripts, papyrus drawings, and more. And that constant... is pornography. It lasts, it's popular, and it's always in demand.
Clearly, the answer for long term data storage is to use steganographic techniques to encode your data into various types of creative skinpics. Pick famous folks, pretty folks, strange fetishes... the whole gamut. Pick things that people will keep. A hundred years later, all someone needs is the key phrases to search for.
"We need that Higgs Boson experiment data from 2012, how will we get it? The infocalypse has destroyed all of our cataloged data!"
"No problem, my great grandfather left a note in his journal telling his descendants to search for 'Britney spears enema' and use 'wet riffs' to decode the LHC data in whatever we use for files."
"President Spears? That's crazy!"
Voila!
"This hard drive will self-destruct in 1,400 years."
Any long term data I keep gets moved to new mediums as they become available. There is no single medium that will last for the times described. The good news is that digital data has a very low corruption rate and a copy can be reverified for a guarantied duplicate every time its needed. I've moved from floppy drives, 44MB WORM, to ZIP, to CD, to DVD and am now using a 12 drive 1TB RAID-5 with AIT backups.
"Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
Wouldn't it be a lot easier to simply keep the archive on a live system, and rotate it to new media from time to time as the old media dies and new storage systems become available? After all, if no one is looking after this system, what's to keep it from being forgotten in the basement of a long-abandoned building?
In addition to taking advantage of the falling cost of storage for a fixed-size data set -- making future replacement media purchases much cheaper than redundant media purchases today -- you also have the opportunity to re-process the data into new formats, so that you'll still be able to read it when you want it.
Wow. It's a really big RAID hooked up to a flash drive filled with metadata and deferred writes. Good job guys, you just invented a hybrid hard drive. Oh, wait...
Of course, this has a pretty cool real-world application in that hybridizing storage systems, that is adding a flash drive to defer disk writes and store metadata and such, appears to reduce MTTFs in any real-world archive, be it a 3-disk RAID for a home office or an incremental backup of the entire internet.
I make websites and stuff. Buy one.
They completely ignored the fact that the chips and memory managing the system will likely have some degree of failure in the 1400 years the data will survive on their media architecture.
Look, I am into genealogy quite a bit and see this as a tremendous problem.
The only thing approaching a viable solution is the Rosetta Disk ( http://www.rosettaproject.org/ ) using etched nickel media (rock) in a human readable format, which you could theoretically create a binary cipher for a global archival format.
But, that would take a lot of foresight, which unfortunately us people don't have (yet).
However, seeing that as completely inaffordable for us mere mortals, that leaves me with PAPER, yes, paper, as the only trustworthy medium-term solution.
I do hope everyone here realizes that if we had some sort of cosmic EMP-like event traversing the globe, we'd lose 99% of data and be plunged into the dark ages, right? We couldn't even re-create all of the machines that surround us since virtually all designs are kept digitally now. Factories would just shut down and never be able to be brought back up and every history of our existence would be forgotten in a few generations.
Our civilization is sitting on a house of cards.
I find it subtly ironic that the last two links in summary of the article about data loss are broken.
-Arthur
Cave ne ante ullas catapultas ambules
The more "religious" the world's populations become, the closer to the dark ages we become. (The reverse is true as well as history illustrates.)
/. is simply common fare and is an easy way to boost karma, but seriously, what? Where is this link between religion - one would assume all religion, as the OP discuss the population of the entire world - and this surge to the dark ages?
I realize that taking swipes at religion at
From the demographic viewpoint, a simple look at the high rate of belief in deity/practice of religion and the United States - the world economic leader, and still, in spite of some losses in this area, the center of innovation in all (well, at least most) things technological - would seem to indicate that the causal link between a belief in religion and a return to the "dark ages" is tenuous at best. For fun, compare the rate of technological advance in the U.S. with that of the devoutly non-religious Soviet Russia or Communist China throughout the cold war.
Then, one could look at individuals - Mendel, Newton, a wide assortment of Muslim mathematicians and astronomers, etc. Even a look at more mundane topics, such as engineers and inventors shows a broad array of other religious folks as well. As a Mormon, the first two that come to mind are Browning, a perhaps unrivaled genius to this day in the design of firearms, and Farnsworth - largely responsible for the electronic television.
Now, I'll be the first to concede the point that several religious groups have shown less technological advance over time, Wahabi Muslims in particular come to mind, but so do numerous others. Some groups have eschewed technology altogether, such as the Amish, but these are exceptional cases. But to argue that the act of being religious at all is somehow tied to a magical turn to the dark ages is absurd, and to argue that a lack of religion has always led to some drive away from the dark ages is no better.
Laser engraving, seriously. There's some project out there....
ah yes, here, that seeks to preserve all the languages of the world by laser-engraving them onto stainless steel plates. They've changed things up a bit, but the basic idea is the same: put it somewhere it won't get lost or corrupted, and if it's important, people will figure it out later. If it's not important, then it doesn't matter.
Very few things in the world are really worth keeping for even a lifetime. If your grandkids inherit all of your stuff, what will they save and keep, and what will they throw away? If you know what they will throw away, why not save them the trouble and toss it yourself?
We've gotten ourselves into this mindset where making backups of every piece of data you've ever owned ought to be saved, for no other reason than because it's easy and cheap. I think everyone should have a periodic storage meltdown to force them to reconsider what it is they really need to have.
Given the media, specifications and some time and money, a trio of engineering, electronics and CS students will make a machine that will read any old tape, punchcard, early HDD, etc. A CD is laughably simple technology, an engineer 100 years from now will build a player (in a way that may not look anything like our current players) in no time at all.
Today's technology is even more well documented and certainly not beyond the capabilities of future generations to make readers for.
If you find an old tape and want to do it in an afternoon, you are out of luck. If you are an historian that really, really wants to get to the data, it is not all that hard.
Let's look at some numbers
A 10 PB storage system could be built for about $4700 with an annual operational cost (power for running and cooling the system) of about $50.
Ya, 10 PETA BYTES for 4700 bucks? I don't think so. And an annual operating cost of 50 bucks, that includes power and cooling? Again, no. Now, let's focus on the administrative overhead of replacing disks and failed system. The larger the setup, the more administrative work there would be.
The rest of the idea has merit, but it almost seems to be that they are trying to compare apples to oranges with their comparison to tape. Tape's appeal is that it is long term storage that requires little maintenance. The same can't be said for this.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
I hate to be a pessimist, what with me being a believer in the power of technology and all, but seriously: we have made no credible progress in technologies needed to push space exploration, minus a few notable exceptions. If we do not spread our eggs to many baskets, so to speak, and instead simply squat on Earth for the next 1,000+ years, it is highly probable (based on my own assumptions and observations only), given human nature, that we will wipe ourselves out or return to some kind of post-futurist stone age.
So this technology has one of two possible final uses: humans attempting to re-learn what was lost, or the equivalent of hieroglyphics for alien archaeologists investigating 'that funny little race of air-breathing bipeds on the third planet of the Sol system'.
On a more relevant note, I have trouble understanding how, even if we do reverse course and realize a future where humans have populated the cosmos, this technology will be useful in even a hundred years, let alone 1000. With the rate of technological gains in certain areas, this is almost doomed to be obsolete before it is ever used.
The only way to store data for more then 1,000 years is to inscrive it in on virtually indestructible materials, such as granite or basalt. These will last for thousands of years. The rune stones in Scandinavia were made in 800-1300 AD and they are still more or less unchanged. They also have pictures. The stelea (stone slabs) in Egypt are some 4,000 years BC and still intact. What we need is a laser printer that writes on stone surfaces...
Beauty is in the beholder of the eye.
The idea is that you'd use unused capacity on existing machines. Their cost estimates are just for the additional equipment - it doesn't include the cost of the drives, since you'll have to buy them anyway...
There are two sure-fire proven techniques for storing data long term - using a reliable non-volatile storage medium (engraving in a non-oxygen reactive metal will do nicely) and making many redundant copies of them.
Electronic storage is by its very nature unreliable -- electromagnetic properties (like charge accumulation, ferromagnetic hysteresis, etc) are inherently volatile.
And even if you manage to solve the problem of transporting your data into the future, you're still faced with the problem of making sense of it. Electronic formats change (just ask the guy out in California who makes a *FORTUNE* charging law people to retrieve files from obsolete formats and/or media). In the physical realm, this is true as well - languages change and become very difficult to read. (If you don't believe me, try reading Beowulf in its original old-English form, circa 700 AD).
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
Seriously. Assuming that scanning methods continue apace with other technology, wouldn't a simple hardcopy in easily scanable format be the best longterm storage. Maybe we should be looking at more durable forms of paper with smaller print, more fire-resistant and technology that will scan this data fast. Like a scroll. Why does long term data storage HAVE to be electronic or digital in nature?
I should write my important information on any available boobies? Is that what you're saying?
A learning experience is one of those things that say, 'You know that thing you just did? Don't do that.' - D. Adams
... hash tree-like structures09F91102 no, 455FE104 nope, F190A1E8 uh-uh, 7A5F8A09 that's not it, C87294CE no. Ah! 452F6E403CDF10714E41DFAA257D313F.
And I mean it literally -- why have any physical storage at all? Why not just bounce chunks of data around forever on the Internet? Presumably the 'net is going to be here for a long, long time. Imagine a mass P2P network where the data being traded is just encrypted chunks of the data of other users. It needn't ever get written to a mass storage device at all -- just received from one peer and immediately sent to others.
A protocol could be developed to allow one peer to request, or steer, the network to locate and deliver requested blocks on demand. This might be a high-cost operation, akin to bringing data in from backup tape. Or, a client could just wait for the right chunk of data to recirculate to its position in the network. But storing data is easy -- just encrypt it, format it a certain way, and inject it into the network.
A natural model for the topology of such a network, and the protocol itself, is the circulatory system. Here, cells move in a fluid, generally in one direction, but through a complex network of vessels, and in a circulatory manner. The immune system might provide inspiration for directed movement of data chunks. (See? The Internet really is just a series of tubes.)
Over time, the infrastructure of the Internet, the P2P clients, and the exchange protocol itself could evolve, as long as enough redundant chunks are allowed to constantly recirculate. Specialized clients could cache data to "long term" storage for periods of a few days or weeks, in case of large, random outages, but permanent data storage would never rely on any specific technology at all -- even TCP/IP itself. It's all just this mass of recirculating encrypted chunks of data, like cells in the blood stream.
From TFA
A 10 PB storage system could be built for about $4700 with an annual operational cost (power for running and cooling the system) of about $50.Wtf? That is 0.044 cents/GB. That's impossible! No one can do it that cheap. Sloppy reporting again I guess... Perhaps they meant 10 TB.
I agree that virtual machines are a solution to file formats becoming obsolete, but I think that emulation may be more appropriate than virtualization for this purpose. VMware can only be used on x86 computers, and even on x86 computers future processors may have subtle differences that could affect old virtual machines. An emulation of an entire computer, including the processor, can be ported to any computer, and have exactly identical behavior.
Also, it may not be necessary to layer virtual machines inside each other, if you have an emulator that that is easy to port new machines, such as by being open source and relatively simple. That is a large part of the motivation for the Macintosh Plus emulator I maintain.
I'm going to, anyway: The Web. Straight up HTTP, with HTML documentation. Fall back to plain-text if you're extra-paranoid, but if you don't do any styling, straight HTML is very future-proof and backwards-compatible. If you do anything on top of that (Dav, etc), document it as completely as you can in that documentation.
I don't really see how wireless is any more likely to be accessible than a plug -- I would argue less so, as wireless standards can change, but no one can legally prevent you from having functioning Ethernet (or Token Ring, etc). If there's a concern of making this thing outlast Ethernet and (say) ipv4, include anything you're paranoid about losing, and put a durable physical interface on the thing so your great-grandkids can read the minimum documentation they need to rig an interface, then read the rest of it comfortably in whatever a web browser looks like by 2095, as they try to code an interface to it.
Or simply include all the programs needed in the original hardware, along with full specs of said hardware, so that virtualization can be built as-needed.
I'm not entirely sure which one is easier. I suspect that an active system is most reliable, as long as you have the money to pay people to look after it. I'd imagine that if you can't guarantee that constant flow of cash, the sanest thing to do is build the most durable physical system possible -- and then build two more -- and lock them away in a vault somewhere, so that a few thousand years from now, archaeologists can reverse-engineer whatever you had. I think that'd be a lot better than hoping there isn't a political upheaval in a few hundred years that cuts off funding, and making the job even harder for those inevitable archaeologists -- now they not only have to understand the original machine, they also have to understand several layers of virtualization.
At the same time, I really, really wouldn't want to try to maintain anything in which I wasn't allowed to upgrade.
Don't thank God, thank a doctor!
Something to leave for the grandchildren.
3d crystal holography like in startrek would be cool.
With no moving disc of course.
"perfect holographic storage could store 4 gigabits per cubic millimetre"
http://en.wikipedia.org/wiki/Holographic_memory
Just think...our distant descendants may one day be able to twist one off to Nina Hartley.
Part of the long-term archival of data has to be a focus on restoration activities. Whether this is the refreshing of old hardware with new storage technologies, or reworking into new document formats while still retaining original works for the future use of scholars. It can even involve translation or modernisation of texts and images over time.
More and more our digital data will need to be viewed through the same lens as we apply to the preservation and restoration of other kinds of works, its unique properties exploited and its unique challenges dealt with. Our most important digital assets will need to be preserved by skilled professionals in library or museum -style professions.
I have talked about this issue before.
My Solution: Google. No really, find a cheap service (or two) that are likely to stay in business, keep updated, and offer online access. What else do you need? Sure it's not as secure as an offline system, but do the majority of people care about elite crackers seeing their family photos?
Isn't this a plan to find a place for obsoleted technology?
For all i know, regular HDD is going out of season soon, replaced by chips and memory crystals.
Why would anyone want to use HDD in the future?
Patents Drive Free Software as Hurricanes Drive Construction Industry
n/t
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
First, it ignores physics. MTBF can't be used in reverse. Yes, it is possible that the MTBF on a newish disc is 300K hours or more, put differently, if you've got 1000 such discs running, then every 300 hours, about every 2 weeks, one will die.
This does however:
It would offcourse if degradation in idle state was -ZERO-. If aging made -ZERO- difference and if the MTBF-rates quoted are realistic AND constant over centuries (i.e. older discs DONT start to fail more often, not even if they're centuries old)
In short: bullshit. It's overwhelmingly likely that not a single disc out of 1000 will remain functional after a millenium, even if it is powered down 97% of the time. At which point no amount of redundancy, distributed or not, will help.
Also, the exersize is pointless. As long as storage-capacities keep growing exponentially, nearly the entire cost of storing a set of data is in the first few years. If you've paid what it costs to safeguard data for a decade, you've already paid 95% or thereabouts of what it costs to store it forever.
So, storing something safely for a very long time is actually a easy task, all you need to do is:
Yeah, this -does- mean that data that nobody cares about will die. Tough luck.
For example, if you -currently- have a petabyte you want stored, you could buy 3 petabyte enterprise storage-servers, at a cost of perhaps $3million. You host these at three separate companies, say one in europe, one in japan, one in usa. For this you may pay $300.000/year. Total cost for first 5 years: $4.5 million
After 5 years you buy 3 new entry-level storage-servers. Storage/dollar has doubled ever 18 months, or a factor of 12 over 5 years. The servers now cost let's say $300K, and they're 4U-units rather than complete racks now, so hosting-costs is down to $50.000/year.
Total cost for years 5-10: $550.000
After 10 years you buy 3 new 1U "small office" servers. They cost $21K in total. Hosting is $10K/year. Total cost for years 10-15: $71K.
After 15 years you sign up for the needed amount of space on 3 separate servers and pay $3K/year, or $15K for the period.
After 20 years you put the data on 3 thumbdrives and store them however one can cheaply store a thumbdrive, total cost perhaps $1000
Or you sign up with 3 separate el-cheapo hosting-providers and pay $300/year.
After 25, you send the data as an attachment to your choise of 3 free email-providers, they all come with atleast 500PB free storage anyway, it's not as if you'll notice the extra 1PB attachment.
More likely though, you've got much MORE data to take care of in the future, so you're still paying $1million/year. Only now that buys you a storage-solution where the old 1PB-archive is a completely trivial file, taking up a so minute fraction of the array that it's not even noticeable and the incremental cost is essentially zero.
So, they are proposing Sun StorageTek 5800 (codenamed Honeycomb) as their research?
Compare article with this whitepaper, especially Figure 13 on page 28. Networked nodes with 4 disks each, grouped in cells of 16 + 1 management node. Each object is stured redundantly on disks of different storage nodes. Everything self-contained, accessible by nice API. Oh, and the software is Open Source.
:wq
It's easy to build distributed, reliable storage that theoretically lasts thousands of years if you assume that you can just keep going down to the corner computer store and buy replacement parts that more or less work like today's parts, that operating systems keep doing what they have always been doing, and that networks keep working the way they always have. But those are bad assumptions.
I'm sure that anyone who recalls the punch card first hand will cringe just at its' mention. However if you were to picture a DVD or CD as being a circular punch card, you wouldn't be wrong.
The biggest problem with archival solutions is that to create a method which is small enough to store in a few boxes, the technology will typically be required to operate on a miniscule scale. This increases the probability that the media won't be properly identified as an archival media a thousand years from now.
While it's possible we'll have simple flatbed scanner like devices in the distant future capable of simply scanning a DVD or BluRay disc and then simply applying an image recognition program to read the data, it's more likely these forms of media will not be sufficient.
Recordable CD and DVD suffer an obvious flaw which is that the method of recording requires the disc itself to be a degradable form of media. Something that can be burned through at a rate of billions of times per minute with a single laser with high precision. Obviously, even using a relatively high power laser, the material must be thin enough to support this. Therefore, it stands to reason that even in a perfect DVD recordable media, time, sunlight, cosmic radiation, and new-age music is bound to degrade the disc past error correction friendly levels in relatively short times (likely years, maybe decades, certainly centuries).
In the case of circular media of high density, it requires precisely timed devices to read a disc. The disc spins and the bits are positioned in locations that are identified by precise timing. The device to read this type of media is very complex. The bandwidth of the laser required to read the disc is also precise. An archival grade media should not have such technical difficulties, otherwise, the effort required to read the media a century from now if no device is left in existance is substantial. Especially since the plans for the reader is likely to be stored on one of these discs.
Punch cards are wonderful since they are linear, rectangular and can be read relatively easy using somewhat primative equipment. Of course, I'm not suggesting using holes large enough to push a pen tip through, instead I'm suggesting a relatively high density punch card where at least a gigabyte can be stored on the surface of a drink coaster sized card.
The card can be made of many different types of materials, in fact, it could be paper or uranium. I would suggest personally a dense metal with a half life relative to the desired duration the media should live.
On one side of each card the plans to build a reader should be stored as human readable images, although on a microscopic scale (similar to microfilm or microfiche) since anyone likely to be able to build a reader will of course have a microscope. The reader presented should be the simplest form and should clarify the encoding used for characters. The device should be able to be built using parts that have been historically available. Meaning that if they existed 50 years ago, and they're still common today, they'll be common 50-100 years from now at least.
The data on the card should be stored through a process of laster engraving or etching for example. Punching directly through the card is ideal, but could interfere with the location of the design. Of course, if you have a box of 1000 cards, you only need the design once.
The process of reading and writing these cards does not need to be incredibly fast. After all, the purpose is for archival. If a single inexpensive device can write one gigabyte an hour, that means with 20 devices, 20 gigabytes can be archived an hour. Besides, unless you're backing up film masters which are typically 3-5 terabytes each or audio masters which can usually be 4 gigabytes, all other files (pictures included) can be backed up in little time.
Every time I read about long term archival, this solution always seems obvious to me. I highly doubt that there's any new patents to be had on it. Pretty sure that pool is not only tapped, but most of the patents should have even expired by now. But if anyone actually decides to make this device from my description or something similar to it, I would love a chance to try one.
I agree with you on active storage being the future of longterm storage. Eventually it will get automated for data that has already been archived, and then the only point is to get your current data into archive. Look at how small PDA's can be. Just store any hardware device that can interface with the datastorage. Make sure its interfaces are completely documented and readable from the device itself. Continuing from that, why not just print the interface specifications of the storage device on high-quality paper an store them with the storage unit? I know, current specs are quite long, but if you design for easy storage and long-term retrieval, this shouldn't be a real problem.
At the moment, the very, very best method of long term archival we have involves the sacrifice of calves, sheep, or other animal... The UK for instance still prints all of it's acts of parliament on vellum.
http://news.bbc.co.uk/2/hi/uk_news/politics/502342.stm
Deleted
What kind of data that will be lost otherwise do we have to back-up for posterity? I mean, come on, no one is going through your perl-scripts, c++ classes, 10000 digital holiday pictures, diaries of what you had for breakfast, or IRC logfiles. You are not that important! Although it would be fun to speculate what kind of information would have been in the caveman-wiki.
Facebook employees incoming all of your data belongs to them forever
I bought a state of the art computer, with *gasp* a 1.3GB hard disk. So, that's a factor of 500 easily, compared to today. And the IDE-interface is still present on some current motherboards. So all I have to do somewhere in the lifetime of my current computer is find an IDE-cable, hook up the disk, and copy the data to my 500GB hard disk. This will take about 30 minutes maybe? So, that's 30 minutes work, once every 12 years to guarantee the persistance of all my trivial data throughout the decades.
Not a single Stargate reference (I, for one, welcome our future ignorant overlords).
Contrary to the popular belief, there indeed is no God.
Simply send the file you wish to archive to 10 of your friends.
:-
:-)
:-)
Promise them good fortune, good health, good luck, or even a good $1, if they will
a) Keep the file for one year in a safe place.
b) After one year, distribute the file to 10 of their friends under the same conditions.
Even if only 1 friend out of 10 follows your instructions, you have maintained a redundant copy in a remote location and guaranteed the propogation / continuation of that redundant copy for the following year. Cost $1 per year.
Rinse and repeat for 1000 years
Hell, if pyramid scams can work for Avon, Amway, Forever Living, etc. they can damn well work for me too
Back in the early 80's I worked for a company that put there monthly backups on to Microfilm. Yes they literally had the one's and zero's from the 9 track tapes printed out to microfilm.
I'm using Indiana preview 2 (final version comming out really soon). It boot from ZFS and I add two USB drives that get mirrored by ZFS. That's all I needed. Cheap RAID 1 that can be read by any computer that runs ZFS (MAC OS X, BSD, OpenSolaris and FUSE Linux distros).
"A CD is laughably simple technology, an engineer 100 years from now will build a player (in a way that may not look anything like our current players) in no time at all"
Rubbish. For a start CD players are not simple devices otherwise Edison would have invented them. (Just because something is now a commodity item doesnt mean anyone could build one from scratch). If you;re uncomvinced go study the maths on auto focusing an pit tracking lasers, not to mention D/A conversion, reed solomon error corection etc.
Secondly , skills are lost over time. Try finding someone now who could build a decent siege engine or longbow that would be good enough to fight a medieval battle. Hell , even finding someone these days who can rebuild steam engines is tough!
So in 100 years time, building a CD player from 1st principals could well be a tough call!
no one is going through your perl-scripts, c++ classes, 10000 digital holiday pictures, diaries of what you had for breakfast, or IRC logfiles
I'm sure that the people in the 11th century would have said the same thing about their accounts and letters, and yet historians and archeologists depend on them to tell us what life was like 1000 years ago.
The perpetual copying works very well. Look at ancient Greek manuscripts. Are any of the originals still alive. The library of Alexandria may have had 100,000 or more volumes but all the interesting stuff seems to have made it. The best philosophers, the best maths, the best art even. The best art pieces made it into themes again and again in Rome and Italy and moder art. History it self is an example of perpetual copying of knowledge.
I got some new Goat-Skin-RWs, and they work great. The smell a bit when burning, but the resolution is awesome.
I am having trouble playing them in my PS3 though.
We have been in this post-apocolyptic dark age for 1000 years now. Everything we need to bootstrap civilization mk II is stored on this "Aard Reeve".
However, the only "Comm Pewter" capable of reading this information was stolen by the evil Mordacs and hidden deep in their underground lair.
Return the Comm Pewter and we can once again wield the mighty "Loy Yer" to enslave the lowly "City Zens" and bend them to our will. We will restore "Celeb Riti" to her "Shaw Ping Mall" throne.
All the greatest powers and virtues of the golden age will return.
Who among you will undertake this holy quest?
The basis of this plan is that if you spin the hard drives less time, in theory the components will last longer. Theoretically this sounds great, but in practice this is not true. Obviously these guys have never worked in a real data centre for a few years in a row. Where I work, we actually place bets with a bunch of co-workers as to how many hard drives we'll lose, everytime we have to shutdown and bring back up the data centre. We only end up doing this once, maybe twice a year. And note that these are planned graceful shutdowns. Out of about 1000 hard drives we have, we lose about 3 on average. The last time the Data Centre was shut down and brought back up, we lost 7 drives! Hard drives are designed to run for long periods of time. They were not designed to stop, start, stop, start. Try doing that with your car and see how long it lasts! I would bet money that the hard drives wouldn't last past 3 years... 5 if you're lucky with this plan. 1400 years is completely ridiculous. And that my friends is the difference between theory and practice. So as they say....
"In theory, practice is perfect; but in practice, it is often only theory".
Adeptus
No trees were killed in the making of this post; however, many trillions of electrons were horribly inconvenienced.
I think you overestimate the difficulty somewhat.
No sig today...
"What kind of data that will be lost otherwise do we have to back-up for posterity?"
Porn.
It a thousand years it will be the height of kinky to jerk off to pictures of women who only have 2 breasts. http://www.imdb.com/title/tt0100802/
"As God is my witness, I thought turkeys could fly." A. Carlson
Because so much is digital instead of paper these days. Backups get lost or go bad. Historians hate this situation.
Just what the future needs, mental scaring from the past!
Anthropologist 1: I've finally gotten this archaic storage device working!
Anthropologist 2: Let me see! What did they store on it?
A1: It appears to be...pornography...
A2: Did human females really only have two breasts back then?!
A1: So it would seem. Look, human males used to only have two testes also!
A2: We really did look strange back then, it's a good thing we evolved.
A1: What should we do with this? We can't really show it to the overlords...
A2: Put it on G-Bay! and just label it 'possible artifact with adult imagery', some freak out there is sure to enjoy porn with only two teats instead of four.
"Quote me as saying I was mis-quoted." -Groucho Marx
I know about semi's - I don't know about any other things in the devices. How will FR-4 age? Will solder joints fail just by sitting around?
...and bury it in the desert.
It's the only thing that's worked in the past (though far from 100%).
Freedom isn't free; its price is the well-being of others.
Not the light erasing type but the type that have little fuses in them that you actually burn and only burn once, then have a the circitry that enables burning burn itself to allow no further burning as a safegaurd against accidental destruction of your existing working links in the chips. Each chip can be made with current tech to store about a GB apiece and a whole bunch of them would preserve for a very long time. Far better than Flash type memory.
Tsukasa: All I really want, is to be left alone...
I don't want naked picture of my ass floating around for the next fifteen hundred years....
Supporting World Peace Through Nuclear Pacification
The solution is simple. Store your data in an Oracle DBMS. Mirror your data on two or more drives. Swap out storage medium as the need requires. Do routine off-site backups, depending on the importance of data.
IT companies have been doing this since the dawn of time. The particular benefit company that I work at, has data archived and digitized from... the 50's!! (lifetime maximums and such).
Its not such a big deal to store data over time. The key is to always keep up with the current medium of storage. Perhaps the author of the article only has one IDE harddrive.
Hi, I Boris. Hear fix bear, yes?
The only likely successful method is carbon on acid-free paper. And even then...
- What language are you going to write information in? I'd hazard a guess that there might not be not a single reader of slashdot who could read a 1000-year-old European document. Most modern English speakers can't read Shakespeare without much help, much less Chaucer, much less something a thousand years old. How's your ancient Chinese?
- If we could, what would we want to know? What an average person did all day. What did they eat and where did they get it? What did they do for recreation, if they had any? Why did they like what they liked, and dislike what they disliked? What did they die of?
Who on slashdot would be recording such information about ordinary people?
Do you think Microsoft Office 3008 will be backward compatible with Office 2K3?
My wife doesn't listen to me either...
How long do you think it will take them to discover "books"?
Clearly you've never heard of the Usenet.
Its called a MAID array: Massive Array of Idle Disks. Copan Systems has been providing these systems for ~4-5 years now? The majority of the disks sit idle (i.e. powered down) and are only spun up when data integrity checks are done or when data is read or written. They get capacity of 1 Petabyte on a single rack...
And the data records that need backing up for 1,000+ years are all of the worlds vital statistics. Essentially genealogical information which until the advent of computers has always been kept by every government of the world throughout all of human history. The other is all the official documents we create and books we write. We need to keep a history of our people and the next generation of the National Archives are going to be digital.
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
And on the pedestal these words appear:
"My name is Ozymandias, king of kings:
Look on my works, ye Mighty, and despair!"
So 'God's chosen people' hadn't enterered the Iron Age at that point. There's lots of other signs that they were not exactly academically inclined either, like the biblical value of 3 for Pi which was less accurate than the value the competing civilisations knew.
... or maybe they just rounded the figures. It's not like they said 10.000 cubits in diameter, and 30.000 cubits circumference. It could have been 9.6 cubits diameter and 30.16 whatever circumference, which were rounded to 10 and 30. After all, the point of that scripture isn't "See, pi=3". The point is "That's a really big bathtub".
When our name is on the back of your car, we're behind you all the way!
Storing Data For the Next 1,000 Years, at least using today's technology, is not necessary. We will only need to store data for a maximum of 10 years using today's technology - if that.
By then, today's technology will be yesterday's technology; a better way will have been found to store the data more reliably, safely and cheaply.
Take for instance:
cellulose film and it's breakdown and loss of so many old movies
old photographs and their fading
old floppy disks
ANY medium you use even stone is going to deteriorate. What we need is highly trained and compensated archivists; the new breed of librarians. They must be given the tools they need to oversee the mass of knowledge entrusted to them and to effectively utilize technology to achieve that end.
Codifex Maximus ~ In search of... a shorter sig.
Silicon chips don't last a notable fraction of 1400 years even under moderate use. *shrug*
What's too bad, really, is that such great genius isn't being put towards solutions that involve harder and longer-lasting materials like stone, crystal, and so on. If you really want to preserve data, you use hard and long-lasting material. You typically don't go micro-miniature -- though that doesn't mean you have to abandon going digital.
And if you have millions of dollars to spend on it, I'm sure there's a way to store your digital information in stone chambers containing metal rods, gears, and so on. If it's done right, the entire mechanism could be powered by a nearby running water source -- mechanically powered, not electrically -- or perhaps by a small "nuclear pig" stored somewhere deeper under the stone (it could be small and still very efficient since the steam power will be used mechanically instead of converted into electricity).
Stone catacombs using reflected and compressed light and vast stores of gas and chemicals contained in large metal or crystal vaults could use large, powerful lasers to store and retrieve data crystallithographically and project it onto the wall as images or even to transmit it by pulse to a small polished-crystal port for retrieval by today's faster, miniaturized devices.
The package of the above two combined might only cost a tenth of the millions you'll be spending on tens of thousands of redundant terabyte disks and their gigabyte flash friends and other accompanying control hardware (which I'm guessing is what holds the price tag of $4700, not the drives and storage itself).
We could add on an terminals & peripherals package that includes crystal-encased viewscreens inset in stone frames made to accomodate four-inch bulletproof glass, and several exchangable panes of said glass to last the wear and tear of centuries of household use. Also includes lead-cased granite keyboard coated in gold with half-inch thick quartz key covers and self-maintaining oil system that will keep all parts oiled and working smoothly for probably several centuries to come off of a mere twelve ounce reservoir.
So on, and so on. What I'm saying is that "1400 years", in the article, doesn't realistically approach the lifetime expentancy of the actual materials involved.
Let's say people, professional shirters, are going about saying that shirts, due to normal, modern washing and drying machine treatment, can be expected to last 2 years. Five years with no rugby.
And you say, well no rugby? How about, now rugby, and I'm going to hand-turn the washer and dryer, and I'm only using enough water to cover the surface, and I'm washing it alone, folded in a dirt absorbant hanky-diaper, with only the gentlest of soaps and purest of water.
Yada yada yada...
You could claim that theoretically, your special treatment of shirts would ensure that those shirts last 500 years. But it's still bullshit that the materials will keep from falling apart or becoming threadbare before then. If they're buried with you, maybe they'll last that long for archaeologists to dig up and be surprised at the resiliency of your weaving. Well it was buried, for crying out loud.
Even if you put all this data on disks that just fucking sat there and did nothing, zero wear and tear, you know that the data won't be there 100 years from now. I don't see why slashdot keeps printing pure-hype sci-fis like these. Who's the fucking budding sci-fi publisher?
"Stratigraphically the origin of agriculture and thermonuclear destruction will appear essentially simultaneous" -- Lee