Slashdot Mirror
Large IDE Drives as Long-Term Archival Media?
"Backups are of no use without offsite archival copies so I plan to take one set of disks out of the pool, and archive them offsite on a quarterly basis.
However, I've heard horror stories about the data retention and usability off older disks which have been shelved for archival, for example disk stiction - where people try to restore data off of a 4 to 5 year old drive only to find that the disk won't spin up due to solidification of lubricants, or that they've experienced data degradation.
I'd be interested in the Slashdot crowd's opinion on using large IDE drives as an archival media. Clearly one possible problem is being able to get hold of a machine in the future with a suitable IDE interface to plug them into for restoration, but I can't see IDE disappearing within 5 years (maybe 10 though). I'm more interested in experiences and opinions on the suitability of the disks themselves for long-term archival.
- Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
- Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
- Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.
Print out all your data in hexadecimal and store it in a large vault. If and when a data loss occurs you just need to re-type all the data back in.
yes I'm being facetious
Trolling is a art,
Without normal/regular use, you WILL have problems trying to read from them in 4-5 years time. Hell, the way most IDE drives are these days (note the recent reduction in warrenty time periods), you'll be lucky if the drives last 2 years even WITH regular use.
Backing up to IDE hard drives.... That's a paddling
Not using SCSI like you should... That's a paddling
The right tool for the job is a tape drive, if you don't use it.... That's definitly a paddling.
the reason ide disks are so cheap these days is their components are substandard. have you ever wondered why SCSI disks are still so much more expensive? and the people who use SCSI disks _also_ spring for expensive tape backups?
this is because disk is a poor medium for backup and long term storage.
the problem youre running into here is simple. IDE allows you to have big datacenter type storage without the capital expenditure. however, what the IDE drive peddlers dont tell you (and what you fail to grasp) is that big datacenter type backups are very expensive. and cutting costs is not going to help you at all. good luck getting *anything* off an ide disk thats been sitting for ten years.
If you have to ask, then my guess is that using cheap fucking flea market IDE drives in lieu of some truly Archival storage will fuck you in the end. IDE drives are shitty. Newer ones are even shittier. A hard disk isn't meant to last 10 years, and you'd be a fucking loon to think it would.
Hell, unless it's some ultra common file type, like text, who the fuck says you'll be even able to open it? That shit will be packed tight in the most archival storage, (not on ghetto shit IDE) but what the fuck good will it be? Look in the non-archival slashdot archives for the archival storage story about the BBC -- their storage medium outlasted the computer they'd designed for it.
Granted, that's stupid bullshit thinking on the part of the BBC, but don't fall in their footsteps. Get tapes. Good tapes. Lots of them. And a valut.
I'll tell you why you've not heard discussion about this before: it's a Bad Fucking Idea.
Speaking from experience I can give this bit of advice for archiving critical information. Use a solid state device, don't even consider a magnetic solution, unless losing some or all of the data won't ost you your job.
Everyone is entitled to their own opinion. It's just that yours is stupid.
Hard drives are not non-volatile storage.
...on me within five years. 'Nuff said, methinks. Don't use IDE hard-discs as a backup medium. It's just... wrong.
-Mark
I think this is not such a good ide, because in 5-10 years from now, IDE may not even exist anymore... remember the old drive in XT computers?
Can't read those drives today... unless you still have the computer it was in!
any easier or cheaper than tapes?
Why not automate and optimize your tape backup system, ie; incremental backups?
As far as the manual part, you have to take tapes out just like you'd have to take the HDDs out.
And I don't believe 220 gigs of IDE space is cheaper than 220 gigs on tape.
HDD mfg's are moving towards a 1 year standard warranty. If you want to put some faith in that, it's up to you.
I don't need no instructions to know how to rock!!!!
For example, DVD.
Make sense?
Good.
...
I back up close to 300GB on a nightly basis using GraniteDigital's FIRE Vue(TM) FireWire 1394 IDE Ultra ATA Systems
I have 6 120GB Maxtor's and rotate them nightly, storing them in a fireproof safe, rated for paper storage. Granted, if a fire occurs, I'm not sure if the data storage would survive, but I think that would be the least of my worries, at that point. The Firewire works great and is very fast.
Try backing up 80TB of data to DLT and Super DLT. You thought your stuff was slow. I have backup jobs that spin 16 SDLT tape drives and run for DAYS! And the solution to the ever rising cost of tapes? 21 hour tape rotation.... go team!!
...you're walking down the hall with a 3 foot stack of drives and you trip over an ethernet cable...and all the drives take a sailing course through the air and land on the concrete floor.
I'm not a betting man, but I bet if that were a stack of DLT tape, you might still be able to read them after that hypothetical incident.
// Agent Green (Ian / IU7 / KB1JQO)
// IEEE 802.3: All 10base Are Belong To Us
What's the preferred archival medium then? CD's seem to be the best option for longevity's sake, but 700Megs is just too small. I'm not sure if I trust DVD-RW just yet for long-term bit-accuracy. Tape simply degrades over time.
The archival of huge datasets will be a big nut to crack over the coming years - especially as the "human knowledgebase" shifts from print to digital.
It's becoming more and more attractive of an idea, and I think there are some companies that sell ide arrays for just this purpose. The only think I would worry about is being able to actually connect to an 'ancient IDE drive' in 10 years. If you're just going to pack some data on it and put it in a box, longevity shouldn't be a problem.
I haven't really had any problems with stuck spindles since the early 90's with the old Quantum drives they used to stick in Macs. I have a number of Seagate Barracudas that had been sitting idle since approximately 1996/7 that I just fired up last week. All of them (about 40) worked and still had their data, which actually happened to be usenet archives that I'd been saving.
I'm certain manufacturers have gotten even better with lubrication issues over the last 7 years and I don't think I'd waste too many cycles worrying about it. With the price of large capacity DLT/AIT tape these days, it sounds like backing up to cheap IDE disk is a viable option.
Cheers,
Just Another Anonymous Coward
Quite a few companies aren't even warrantying them over a year anymore... and you're thinking of using them for long term storage? Why not just use DVD-R?
Since IDE HD manufacturers recently decreased their warranty period, I'd be *really* reluctant to trust 'em 10 years from now.
I think it would be a bad idea to rely on IDE drives as one's only source of backup. Especially if you aren't planning on using any stripping or parity. The large IDE drives are, the more prone to failure they appear to be. Ask anyone thats bought a 60-100 IBM deathstar drive lately. The added wear that would occur from joustling them around as you pull them in and out of the drive bays all the time seems like it would also make the time between failures greater. What is proposed in the story might work fairly well for a home user, but I think it would fall apart in a business setting.
People here are saying, "Don't even think about using IDE!". Well he has no choice, does he? Tape has several drawbacks as the author mentions his comment to Slashdot. He has asked for advice on IDE. If this is not a feasible option, recomend some others (besides tape). Or ARE THERE NONE?
The Welkin: Online Music Reviews
Using magnetic media to back up magnetic media isnt the greatest idea in the world, but it can work. Hard drives fail, and when they do, you want to have the data available so that you can get to it. The IDEAL way to do this is to contract an outside company or manage for yourself a backup server which does incremental backups as often as you need and periodically burns them to a more permanant media like DVD. If you cant afford this or dont like the idea, then you can burn DVDs on your own. A good program will track files for incremental backup and 220 gigs can fit on something like 50 DVDs, with maybe 1 more per session (assuming that not all files are constantly changed) Obviously a lot depends on what you have, how much money you are spending, and what you need.
People who think they know everything really piss off those of us that actually do.
with all the stories I've seen about being unable to retrieve data from just 15 yrs ago (because the format is unreadable, not because the media deteriorated) I'm convinced that archiving data using a chisel and a rock is the best way to go.
There is no reasonable defense against an idiot with an agenda
:wq
. . .lists a "Component Design Life" of at least 5 years. Call me crazy, but I just wouldn't trust them any longer than that.
If you're going to be elitist, it would help to be elite.
My experience says "choose another vendor." I have seen too many of them fail.
If you are going to use IDE as an archive, then build a system and keep that system as the one that supports the drives. Experience says again that the drive metric translations differ from one MoBo to the next. Meaning a drive readable in one system is not in another.
Good luck
I think you'd have a better chance of having the hard drives function in 4-5 (or 10) years if they're used every once in a while, rather than letting them sit for 5 years in a drawer somewhere...
Another issue you have to think about, when dealing with a 10 year time frame for hard drives, is the interface issue. After all, could *you* actually read a disk from 10 years ago, when MFM, RLL and ESDI drives were popular? Can you even buy a controller for those drive types anywhere?
There's the distinct possibility that in 10 years the standard drive interface will not be IDE or SCSI, and the new interface probably won't be backwards compatible.
If you're rotating the drives (using one for your main drive and two for backup for a month, then rotate every month) and you keep some money in the bank to replace the one that's going to fail about every 6 months, you should be ok. Well, except if something affects the whole box (fire, etc.).
The magnetic flux only lasts as long as the warranty.
----
Go canucks, habs, and sens!
What you're proposing will cost no less than a high-quality AIT drive, which, though you may need to span tapes in the most extreme of situations, will give you quite a bit of capacity. You can pick up 90GB native-capacity AIT drives now for around $500 or so on eBay. The media is affordable, too.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
it's not IBM deathstar ;)
--
I don't know about the next man but for me, I've learned from experience that IDE drives aren't reliable (why else would a majority of the hard drive manufacturers reduce the warranty periods to "save money"?). Maybe it has to do with out of the 9 IBM hard drives I've had over the years, 6 of those "Death"-stars blew up in my face within the first year of getting them. Of course, IBM isn't the only one. WhoreD and MaxWhore as affectionately called by my circle of friends/victims have their share, too. The only good thing with WD and Maxtor unlike IBM is that those two have developed better warranty and customer services to promptly deal with it. (For which I am very grateful for, BTW). While I can get my WD and Maxtor drives RMA'ed, my IBM drives still there as paperweights as IBM points fingers to some other company and they point it back at IBM. (Though IBM should be accountable for the "quality" of their craftsmanship, they deny it).
And yes, even WITH the dead DEATHSTARS, my collective data pool is over a terabyte now. I'm just waiting on C-3D and Inphase Technologies and their 100+GB per disc storage medium. It's cheaper, more reliable, and the size of a CD.
With tape, the failure of a tape drive doesn't separate your from your data (unless it catches on fire with the tape in it or something.) You can just get a new tape drive and you are good to go again.
Thus, tapes are very good because the storage medium and the read/write hardware are separated and not interdependent.
I've had customers bring me DLT tape backups of their databases, and 4 out of 5 times I can't get the tape to read the catalog.
Tape works great same system same system, but it quickly becomes an arcane science beyond that.
Used to be in the data backup biz, you should really start with evaluating what you are actually backing up. Most people backup applications and temp files that really are not going to help much. Also, do you really need to archive all of that stuff even if you are anal? Another thing to consider is, will the media be supported and will you have the proper drivers for the disk drives handy. 220 Gigs is surely still in the land of tapes, I hate them more than most, but would not suggest the use of an IDE Hard Drive. my 2 cents
Their answer? A huge RAID array starting at 180TB and growing steadily over time.
Your answer? Probably figure out which of the data is fixed and which of it changes and attempt to back up accordingly. Does all 220gb change on a weekly basis? That seems unlikely...
that disks will rot, so you can't trust them.
I counter with this: tapes rot too. In fact, any tape older than one year that I've had to go back to has been worthless (read: it had deteriorated data).
Tape is a really bad medium to trust, but we keep buying it because we can't think of a better solution. Personally, I think the way to go is just to give up and admit that disk is not cheap. You need to back up your data to a live mirror system with identical storage (hourly rsync does a nice job) and then you need to arrage a service that can back up your data to remote live mirror systems. Note that in both cases I said "live mirror". You don't want a backup sitting on a cold box because you never know the quality of it until you need it.
The remote backup part is expensive, but it's the only reliable way. You seed it by tape (full backup to tape, and mail them to the vendor) and then use dedicated lines to keep a regular incremental update going.
If one of those two backup systems fail you know about it right away and you fix it. No more tapes rotting on a shelf only to be discovered when your data goes south.
I'm sorry, but 220GB easily handled by backup tape. With SDLT and AIT tape capacities exceeding 100GB per tape, two tapes can easily handle your load.
If you have the budget, get an autoloader so you can perform a full backup in one session, or two tape drives for that matter.
Personally, i am backing up 600+GB onto tape and it works well. I've had numerous IDE hard disk failures, yet not a single data tape failure so far.
Serial ata (SATA) hard drive are emerging now as the new standard, and as the power and cabling is cheaper - these will slowly replace all parallel drives in to mid-low end consumer products. Most HD manufacturers have dropped their 3 year warranties to a measly one year, showing their lack of confidence in these newer higher density units. So I wouldn't use them for long term archival backups, but they should be OK for nightly and monthly backups. Replace the drives yearly with the newest largest available. For archives -use black CDR media or DVD. A bigger concern should be software - are you sure you'd find something 10-15 years in the future to read FAT, FAT32 or even ext2-3, iso9660 ??
Definately agreed. I just had to replace a 20Gb Maxtor that died... just before the end of it's warrantee period (1yr). These things have a short warrantee for a reason, mainly because they don't live very long anymore.
As drives get bigger, I think this problem will only get worse until they figure out a way to get more storage with less cram and degredation.
So, how is Pixar archiving it's film data? How about LucasFilm? I'd think from the amount of data they work with, thos guys would be the best at answering that question.
;-)
Personally, for long term storage, I'd go with redundant backups of differing media. Maybe hard drives (stored properly in anti-static bags with silica gel), as well DLT stored in a similar fashion. Increase your odds of support by future architecture.
For daily backups, hard drives are surely the way to go. Faster, cheaper, easy to replace, longer lasting media in my opinion. Anyone who says otherwise is trying to cover their job as a tape changer.
Someone, please shake me from this wide-awake nightmare.
Why worry? You're using several drives in turn. And of course you will all of these backup drives every few weeks for integrity, won't you? So a situation where you pick up a drive after 4 years and the data is surprisingly gone should never occur.
(I'm not an AC, but somehow /. doesn't seems
to remember my login from frontpage-to-article-
to-postreply today. Strange. Mozilla, proxy off, cache cleared, cookies allowed.)
If you're archiving old data that's not currently in use I don't recommend using IDE due to problems caused by not running the drives for a long period of time.
But for daily/weekly backups I suggest a 3 drive (or more) rotation where the current drive in use IS the backup of the last drive backed up. That way any problems after a backup are known immediately and one can roll back to previous backup drive and purchase a new backup drive.
I've been doing this for years with Linux and Windows and it works great. It's also the way to go before updating a distro. Just backup and then use that backup to install distro. If it fails, then fall back to the previous working drive. Nothing lost.
This approach will work beautifully if you take care to store the drives in a cool dry place. The drives will last longer than they would under continuous use.
If you are worried about data decay, use a backup scheme that involves something like reed solomon error correction, or simply back it up twice on twice as many disks. You could get pretty paranoid with redundancy in this solution and still come out way ahead in terms of convenience and cost! IDE drives are super cheap. Go crazy.
I bet the electro-magnetic pulse would wipe out any magnetic media in a flash. Bad idea.
store your hard drives at Absolute Zero!
and refresh them onto new disks every year. Problem solved, though at 30 times the cost (assuming 10% interest rate and that you can't sell the used drives).
If I were going to do this I would implement a few proceduress...
1) Each drive being used as a backup gets a full check disk/defrag once a month. You could have an
old dedicated machine that just sat around doing this.
2) Older drives would by cycled out with newer/bigger drives. Price/size/performance would probably make this justifiable in the 3-5 year range. I would keep the older drives for 6 months or so which would insure that the new drives had passed their burn in time. This would give you duplicate backups while the new backup drive proved itself.
3) Critical data would be moved to CD or DVD storage.
At only 4GB a disk, DVD ain't gonna cut it. 40/80GB DLT drives aren't exactly the best solution either.
Almost all the posts so far have been flaming huge IDE solutions, but nobody's suggested a remotely acceptable alternative, short of massive robotic tape libraries.
Any ideas?
Can I use my laser printer to print on Gummy Bears?
Can I dry my cat in the microwave?
Can I put rice in my car radiator?
Can I unplug all the fans in my computer so it will run quieter?
Can I run 120 VAC on the spare CAT5 pairs?
"Eve of Destruction", it's not just for old hippies anymore...
What we have is a plethora of opinions about what's wrong with IDE. Why don't you solve your problems. 1) It's unreliable over the long run. - Have multiple disks that will be unlikely to fail at exactly the same time. 2) Remote storage - Get a friend that has some space and get him to store a disk for you. 3) Jostling hard drives might reduce life expectancy. - Perhaps try remote incremental backups over the net. Do one large copy locally then locate the disk at a remote connection and backup incrementally. ( remember the local part for the first copy - I got a note from my ISP when I did it over the net.) 4) Lubricant failing in disk. - Two things, you can either replace the disks on a cycle, perhaps every two or three years. ( prices keep going down) or keep the disks spinning on a remote computer that is UPS protected. There are many other ways to do this. What we need here is people with solutions. Not more problems. Think outside the box.
In the battle between good and evil, evil has more fun...
"Print out all your data in hexadecimal and store it in a large vault. If and when a data loss occurs you just need to re-type all the data back in."
I was surprised to find this comment modded as Troll. It was obviously intended to be funny.
I'd moderate it as Overrated because it wasn't even funny enough to be used on Just Shoot me, but Troll was very inaccurate.
Surely your company can afford you one of these. It will handle all your space problems with ease. We are getting one at my work (although we are getting the single tape version since we are only backing up 100G)
0 001LW-S.html
http://rss.seagate.com/products/srssDrives/STUL62
Does the name Pavlov ring a bell?
If the tape drive electronics fails, you can get another tape drive and still read the tape. If the IDE drive electronics fail, the data on the drive is unreachable without massive and expensive intervention.
...phil
"For a list of the ways which technology has failed to improve our quality of life, press 3."
disks time life is short.
What i'm doing here is to store my datas on a 2.5 tb IDE raid-5 with a spare disk, so i can change disks when crash appens. But I must not keep the data for a long time. It can be a good idea if you keep your system up 24/24, but it seems that's not the case for you.
It's funny that this is being raised as a question now... I have six Maxtor 120 gig drives on their way to me now, to go with the two 120s I already have, and they will be connected to a pair of ATA133 cards from Promise. Total cost is around $1200.
I intend to software RAID-5 them using Windows 2000 Server (and yes, I have a separate "basic disk" 30 gig boot drive). This will give me 840 gigabytes of space on my server. My data comsumption right now is 520 gigs worth of 40s, 80s and 120s. This is in music, movies, and above all, TV shows.
I will be backing up the drives on a set of 40/80GB DLT tapes, using a borrowed drive, and putting those tapes in a fireproof safe offsite. That will at least give me a static backup, until I can borrow the drive and buy more tapes again.
The risks as I see them are: total loss of data due to theft or natural catastrophie, power supply failure, etc. The other downside is the slow speed of software RAID-5 using 5400 RPM drives with 2 megs cache.
In this case, about 10 gigs of the data is what I consider "critical." I back this up regularly to CD-R and keep a copy offsite. The rest can likely be regenerated by re-ripping my CD and DVD collections (also susceptible to theft, or other loss), and downloading the TV shows from P2P, assuming that is still an option at the time of data loss.
Why do I collect so much data? I'm not sure. I think it is because I'm a packrat; and I also fear for the day when information, data, and media are gray market materials. I want to have the bits in my hands if and when the great big lock of doom is put on the world's multimedia.
Just thought I would chip in with my anecdotal experience and say, "Go for it."
On the other hand, I could spend (as I have) US$40 on a basic (a.k.a. el-cheapo) FireWire-IDE case, US$30 for 3 removeable IDE enclosures, and (eventually) about US$70 each for 3 60GB IDE drives. Total cost: US$280.
What do I sacrifice? Not much ... one of the drives might fail. At that point I'd just replace it with another US$70 capacity drive (which would probably be larger.) If I needed to restore something from backup, I'm already looking at up-to 24-hour old data, and if that drive happened to die, possibly 48-hour ... it's unlikely that all the drives would fail at once.
The advantages? I can use the US$780 I save for something else and I don't have to worry about shelling out another US$1000 every four years just to scale to "current" requirements. I don't know what the upper limit of an IDE drive is these days (i.e. what can the ATAPI bus handle) but even 200GB is pretty big for me right now.
Anyway, just a few thoughts. The basic thing is lower cost for nearly the same risk ... tapes fail too, you know. Remember, too, that this story would be very different if I had to handle 50 machines instead of 2.
--- Jason Olshefsky
Karma: Poser (mostly affected by adding this line long after everyone else did)
Many people forget that remote backups require no on-site hardware or software and don't require you to spend hours upon hours configuring things.
Even better is that any flood, tornado, or fire at your house or business will not ruin your tape, dvd, cd, or hard drive backups. You simply connect to your remote backup location and restore your old data onto your new hardware. It's that simple, and it's cheap in comparison to spending $3,000 on a tape backup device that only stores 150GB of data per cartridge.
You may want to see if this remote backup company has services that fit your needs (I don't work for them, so it's not a plug). Basically, they state the following as the main appeals to remote backup:
Your data is continuously backed up as it changes, 24 hours a day, so it's always up to date. And it's stored electronically at Iron Mountain® data centers, where more than half the Fortune 500 protect their data.
No-Wait Recovery - Instantly recover your data to the point of failure, eliminating downtime and data loss from relying on a previous night's backup. And a unique web interface allows you to initate restores from any Internet browser, anywhere.
No Tapes, No Hassles, Lower Costs - Tape-less backup and recovery means no hardware or software to buy and a fully automated process requiring little employee time or resources. Lower your data protection costs while freeing IT resources for other tasks.
If you celebrate Xmas, befriend me (538
I know it sounds like a stupid question, but why are you backing up data? What are you trying to solve
Short term failure
A luser makes a mistake, or there's a glitch in last night's source code library, and all your current data is foobarred. In scant minutes, you can restore lost data from overnight backups, (or even hourly incrementals), and you are the hero. Realistically, you're just doing your job, and you'll never get thanked for it.
Complete Failure
In the event of a building fire/server room flood/earthquake/Act Of Dog, then you may need to retrieve all your companies data from as near back as possible. This backup should be off-site, and as frequent as feasibly possible
Long Term storage
This is for archiving of a project, etc, and should be off-site. Also for archiving source code in case your company goes belly-up, so that customers can still use and modify your software (in escrow).
Ask yourself which scenario you are dealing with, then the answer as to which media is the one to use may be clearer.
My suggestion: check out the usb removable drive trays. This way, you can hot swap/back up w/o powering down.
As for the comment about the two different types of hard drives - this can be said about anything. for example, there are two kinds of cars - those that have turned to junk and those that will turn to junk. Will that stop you from driving a car?
Tapes are fine for backups, but I never expect to pull complete and usable data off of them after 6 months. Why? Tape degrades - it's nothing more than rust on platic. As humidity and temperature change, you can end up with a solid roll which will stick to your tape drive heads and result in whole patches of magnetic coating coming off. I worked on a project restoring data from 10+ year old reel-to-reel tape, and it was a nightmare. 1 out of 4 tapes was completely unusable.
Even worse, tape drive formats keep changing - and since tape drives are guaranteed to wear out, where are you going to get a working tape drive to restore data 5, 10, 15 years from now? I've gone through 3 tape drives in the last 8 years - thank god I got a CD burner early, that data I can still read (although it's about time to start recopying stuff from 1996.)
Basically, if you entrust your data to tape long term, you have to continuously copy that data to new tapes, and or new tape formats. Where tape has traditionally shined is as a short-term backup format, although with the drop in DVD-burner drives/media, and the high-cost of high-capacity tape drives/media, this may no longer be the case (assuming you get some peon to do the big backup on DVDs, and you get to do daily diffs - otherwise, having a bank of tape drives is cheaper on staff time.)
A couple possible risk mitigaters would be to rotate your media more often (no pun intended) and use a RAID rather than single drive as an archive of any given media.
For media rotation, you should depend on any one drive for no more than a couple years and then retire it (copying its contents to a new drive if they weren't already expired). It may be a bit expensive, but if you're looking for high speed storage, it may just beat the alternatives. Depending on the size of your organization, you can perhaps "waterfall" the retired drives into less critical roles (e.g. desktops)
Backing up to a RAID would involve using (possibly) trios of hard drives together, plugged into removable bays that connect to a RAID controller. That permits one of the drives to fail while still allowing your data to be recovered without having to think about it.
What do you mean they cut the power? How can they cut the power, man? They're animals!
*I'll* be your sub, baby. Slide it on in. Whip me, beat me, stick me in front of a Packard Bell 486DX2 with a 14.4K modem.
--couldn't you get just another dedicated computer for archiving and burn the data to a cd or dvd? Maybe a little slower but won't it last longer? the blanks are cheap, last a long time, when it comes to to re-archive I imagine the tech will be even cheaper and faster. Basically, I don't trust mechanical devices to last, wheras something like a cd should last for years.
Totally unrelated though, but I am wondering why someone doesn't make a modular hard drive where the platters themselves are removable semi-easily. If/when the mechanical parts of the drive crap out, you just place the platters into a new drive. Why isn't this done? I really don't know, not an engineer but it seems at least reasonable on the surface to ask.
Sure, it can handle my load, but can I afford it?
Amazon - 110/220GB Sdlt Internal SCSI Tape Drive
Not all of us home computer users with 200GB of files happen to have $5,000 in their back pockets for a backup device.
Is there any way to do this much backup, say, for less than $500?
-S
We Apprentice Developers and Designers
Sentient beings evolved from human-created "mecha" go digging into the icefields for traces of human civilization. They can resurrect humans from a few wisps of recovered DNA, but they can't read the friggin' drives Cliff used for archiving ages ago...
The "right" way to make your data reliable is with mirroring of various sorts. On-site backups are kinda silly except when you're using them operationally because you dont have the disk capacity to do otherwise for infrequently used data. Backing up to removable media should be exclusively for offsite storage.
So get two drives and mirror your data, and you're covered in the case of drive failures. If your worried about a whole machine going up in smoke, maybe do a nightly or hourly rsync to another machine across the room.
If your home data is important enough to need offsiting (usually a home user's "important" data amounts to what could fit on a CDROM, not 220 gigs - the rest is probably multimedia fluff that you can stand to re-encode or download in teh case of a tornado or fire), then consider rsyncing with a freind at night over your DSL or cablemodems in a mutual arrangement. Encrypt the data before syncnig it over if it's sensitive.
If you're a business with large volumes of data that need to be offsite in case of disaster, then the best practice is still tape drives of some sort, and an offsite storage service like Iron Mountain.
11*43+456^2
Gee.. that's my home computer
yow!
got modded as a troll too, even though, for once, the first post was on topic, mildly funny, actually had a *point*, and was labeled as facetious in the body of the text.
Go figure. Neither you nor he deserved to be modded as trolls. Troll is overused because it gets used as a catch all for various posts that the moderator finds offensive. It's perfectly possible to be offensive to someone without being a troll or flamebaiting. A subtlty that some of the moderators haven't grasped. In fact, I'm not sure some of them have even bothered to check the *definition* of the world troll and flamebait.
So, in short, you did not serve as an example of why we need the troll moderation.
You serve as an example of why we need an *asshole* moderation.
I hope that makes you feel better, have a nice day.
KFG
If you have enough upstream I'd suggest to let
someone keep your (crypted) backups.
That's a lot of pr0n
Steve Gibson is an asshat with a product to sell.
I wouldn't take any of his advice. He's a paranoid nutter just shy of wearing a tinfoil hat.
--
the strongest word is still the word "free"
- always keep at least 2 copies of any data
- copy the data every six months to a year
Just build it into the plan.Its the only way you really know you can read it. Then when you need to shift the data to newer media, its just part of the routine.
Plan to keep the bits forever, and rewrite and/or replace the media regularly.
That said, actual shelf-life of disk drives would be an important cost planning number. I've never seen such a number published.
- "History shows again and again how nature points out the folly of men" -- Blue Oyster Cult, 'Godzilla'
There are some important things to remember when comparing price/quality.
IDE drives have gotten cheap because everybody uses them. So companies expect Joe Blow to buy their disk drive because it has more GB per dollar. They'll use cheap motors, cheap heads, and cheap platters to lower their cost so they can get Joe to buy their disk.
Tape drives, on the other hand, stay expensive because they are not subject to the normal consumer market. So they retain higher quality parts and smaller capacity. Most people don't have to do full backups every night, and again most incremental backups aren't overly huge, so non-gargantuan tape capacities are okay for a lot of the market. The drive and tape manufacturers aren't about to go ruining reliability just to satisfy the high-capacity end of the market; they can make enough money doing what they're doing.
So use tape if you want your data. Use IDE if you want to look like you're making backups. It's your money (and hopefully only your data)...
The Right Reverend K. Reid Wightman,
"I have $500 to spend on a backup solution for my 220GB data pool, and I was thinking of buying 4 120GB IDE drives along with an IDE RAID1 card and useing the array for backups, anyone have other ideas?"
"No way, you are insane. IDE is horribly unreliable and you will surely lose your data. You need a $6000 tape drive, if you can't afford it you are better off with no backups at all"
I love going down to the elementary school, watching all the kids jump and shout, but they dont know I'm using blanks.
It couldn't possibly be because the cost of materials or the fabrication process is cheaper now? I don't think using substandard materials and components is going to make you a successful business when RMAing all those drives (and most drives have exceptional warranties when compared to other consumer electronics) would cost you a lot of money. IBM just sold their disk division, and it probably had a lot to do with the fact that they were gutshot and hemhoraging(sp?) money from producing all those shit shingle drives that were failing at an almost unbelievable rate.
IDE drives have a reputation as being substandard. This is actually not the case overall. Most IDE drives are housed in machines on the office or house carpet which is one of the worst enviroments in the world for dust, dirt, grit, dog pee and heat. SCSI drives are often housed in the nice clean, cooled machine room. Further, IDE drives in personal machines go through many, many more spinup cycles than that 24/7 server disk does. Check the MTBF figures, they will also specify a number of spinups figure, since this is where the bearing wear is.
Backup tape bleeds and needs to be re-generated from time to time. It is no answer to the problem. Just ask anyone who has done the "tape salvager" routine on the 9 track. However, the same thing will happen with disks. the thermal stuff will weaken bits, etc. Personally a raid 5 array on removable ide that is periodically re-read to regenerate lost bits is the way to go.
As far as doing the same thing that everyone else does with tape & SCSI, most folks have a lemming viewpoint. They do not like to stick their heads out and put their careers on the line for something that differs with what everyone else does.
IDE's will be a very unstable, unreliable long term future option. They are not engeneered to be usable over long periods as it is and as PC components have got cheeper and more expendable hard disks are perhaps the first part of any computer which will fail (before considering issues due to shelving discs for a number of years).
Looking for long term mediums, consider DVD's and their successors which will more rapidly keep up with data volume rates than dat tapes - be more stable over time and be more likely to be accessable in 10 years time.
Using ide in short timescales might be more worthwhile - short term faliures are less likely to be catastophic for both discs and you will probably find that you can fit both backups and orginanals on the new diskdrive media that is on the market in a couple of years. So maybe the long term questions don't apply so much - rely on technology catching up?
I've already got 2 80G drives in my PC (which I just slapped together over the weekend) and I'm still a bit floored at the concept of having this much space, yet, once I get into rendering and animation, in the next couple weeks (Christmas-New Years break, Wheeee!) I'll probably also be floored how quickly it goes. :-|
A feeling of having made the same mistake before: Deja Foobar
You speak of not having tape failures, but you omit one important fact; how many times have you successfully retrieved data from tape?
IDE disks will fail from continual use, and that failure will generally be obvious, but what way do you have of knowing that you genuinely don't have any tape failures, if all you are doing is rewriting over the same tapes?
I have tried the removable hard drive w/ ide interface and after about a year of use the ide connection between the holder and the hd case stopped connecting causing a huge head ache.
We have recently switched to an external firewire drive which is working alot better and better yet, you don't have to reboot (hot swapable).
Bottom line of advice, avoid the removable ide bays, they are more pain they are worth.
My company provides mission-critical systems at the core of its customers business. Needless to say, data cannot MUST NOT be lost, so therefore backing up both transactional and referential data is an essential part of the setup.
Some 10 years ago, we weren't as good with this. We usually let our customers handle their backups themselves, maybe on our recommendations. One company had a tape backup running every night, backing up the essential parts of the database. After they'd been up and running for a few years, they had a failure and lost the disk with the database. Fine, just retrieve it from the tape backup. Well, it turns out they had run the same tape all along. It had effectively been polished blank by the magnetic heads over years, and not a trace of their database could be found.
They didn't have any other recent backups, so they were pretty much screwed. I think we lost their business after that.
Oh, I can't help quoting you because everything that you said rings true
And how do you know that? Where is the evidence that even with only 20 hours use on them that sitting on the shelf for 10 years (Assuming they aren't left in the rain) means that they won't be in perfectly good/brand new state? In fact if you had to make a guess, I'd guess that they would be! Grab a pair of them and keep a rotating/redundant backup and if at any time one ot them fails, buy a new drive and keep going.
The only real danger with keeping a current model IDE drive on the shelf for 10 years that I can see is the possibility of a hard time finding a machine that can read the aging file system or connect to the aging hardware interface seeing as Serial ATA is on the verge of completely replacing parallel ATA.
The mythological 100GB per disc flouroptical storage medium? Was it all a scam? Did it get killed by patents? Did they run out of money?
You remember Constellation 3D right? You first heard about them in 1999...try searching google for "Constellation 3D" or "C3D" or "Eugene Levich" (the CEO). I see various articles about partnering with this or that, but nothing past 2001. And their website has been offline so long that google doesn't even have it cached.
So...whatever became of this? Is the concept sound? Can someone else resurrect it? Or was it just a lie like the guy who pretended to have the ultimate streaming video codec but in reality ran coax through the power cord?
- JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
>> Not all of us home computer users with 200GB of files happen to have $5,000 in their back pockets for a backup device.
For most, the only mission critical stuff is maybe their quicken files, some documents, and whatnot. I can backup whats actually important to me on a couple of CD-Rs.
If you had 200GB of stuff that was worth backing up, you'd pay what it costs to do so.
It's just a cost/benefit thing.
Is your MP3 collection worth 5,000$?
1000$ for a redundant offsite raid-5 array?
500$ for 2 more rotated backup drives?
How about 100$ to dump it onto CD-R or DVD-R?
About the only thing I'd use IDE drives for, from a backup point of view, is as an intermediate step. Transfer to IDE disks, then use them to back up to more permanent media (tape, DVD, paper, abacus, etc.)
"Those who would sacrifice liberty for security deserve neither!"
This is probably the second-oldest problem in IT (after 'how do I get a girlfriend?').
.... * y = money
Tapes and optical are slow, and either painful manual processes, or expensive robotics. By the time you put the robotics and the drives together, the fixed disks are cheaper. It gets much worse if your goal is near-line storage -- but just archival is expensive.
Even compressing the files (which I wouldn't recommend), you're talking about dozens of DVDs for a full backup of 220GB. Incrementals make it a little easier to back up, but have costs to find each file you need to restore, and reduce your overlap in redundancy [sic].
Time * a = money
Storage space * b = money
Disaster Recovery costs * c = money
Service Agencies * d = money
Capital Investment * e = money
(Consultant to calculate a through y) * z = money
Personally, I'd only use swappable hard drives for quick-fix backups: rotate two or three for short-term backups, with my longer-term and archival going to some other media. Sooner or later, you'll need to upgrade those hard drives to a larger capacity, and that will be expensive compared to buying another spindle of DVDs.
Of course next year, you may need blu-ray DVDs to handle your backup.
OK... this is the third-oldest problem, after 'Should I buy now or later when the next kewl stuff comes out?'
Design for Use, not Construction!
CD-R and especially CD-RW degrade very quickly as well. I suppose if you store them properly (away from heat), they may not degrade so quickly. But I don't see them being much better than hard drives or tape.
Get alot of archive quality, acid-free paper. Get a printer with alot of archive quality ink and print out the data in binary. Dots or slashes would work fine for the 1's and 0's.
Archive quality paper and ink lasts for hundreds of years. Should you lose the data on a magnetic or other storage medium, you could always run these papers through a scanner with some OCR and retrieve the data.
Sure, a fire or flood could damage these if you don't have them protected against that, but at least you won't have to worry about deteoriation of the medium.
-
True. The tape drive solution is oriented towards businesses who have the money for a backup device :)
From the poster's requirement of needing offsite backup, i was assuming that it was for a business.
For home users, you can probably afford one generation behind. A DLT 8000 (40GB/80GB Compressed) drive on eBay runs for about $500. A DLT 7000 (35/70GB) runs for $300-500, so it is possible to do tape backup on a budget. It's the usual tradeoff between time and money, so you'll need to spend more time changing tapes.
On the other hand, for home use, i only archive my data onto CD since most of the data I have does not change and does not need incremental backups..
When not writable DVDs?
I currently create permanent archives (not nightly backups) with a PERL program calling MTX to manage tapes in a jukebox.
/dev/... > tape_toc.
/data -ls > disk_toc.
I write (gtar) to one tape, copy to a second tape.
Compare the second tape to the original data like this gtar tvvf
Then get a table of contents from the disk find
Write the differences to a file diff tape_toc disk_toc > difference_report examine any differences.
Send one tape off-site. Keep one tape on-site.
Keep the tape_toc.
- High Tech workers, please say NO to Union Carpenters, their Union sees fit to control our compensation.
How about copying it to an appropriately big tape and THEN sending that tape to a DVD mastering place? Don't they have the capability to easily span multiple DVD's, along with the automation to do something like this on a semi-frequent basis cheaply? That way, you've got two copies, one that can be re-written nightly and one that is truly archival in quality (20+ years). Any suggestions?
- Sometimes you're the pidgeon, sometimes you're the statue.
For my clients, I always suggest the use of stone and / or clay tablets for all mission critical data archive projects, regardless of size or scope. Bablyonian and Greek models of data retention from as far back as 4,500 years ago are (in many cases) superior to the models we commonly use today, with much of the physical meadia having survived electrical storms, tornadoes, floods, fires, and wars on every scale imaginable with a data corruption rate of zero and without the benefit of a climate controlled room, dedicated security staff, or even a closet for media storage. Imagine the elegance of a 84'3/4 STROM (Stone Tablet Read Only Memory) machine hooked up to your Slackware Archive server for performing restorations, and the ST Binary Writer you have networked to your backup systems and kept physically over by the quarry... nice! The TCO for slab is far less than that of tape archives, considering you can store the media in a pile of mud and hose it down when you are ready for a restoration.
M
All in all, there are 3 general ways of storing data in paperless formats.
1) Disk
2) Tape
3) CD/DVD
I would definitely not recommend using flat out fliud bearing IDE hard drives if you just want to just back up and then leave it alone for 4 years. If you really have to stick with a hard drive solution, I suggest SCSI based and used at least periodically.
Your "3-way" backup solution sounds similiar to a storage area network/Storage networking approach. It's logical and it works. Nothing really new here, it's been around. Check it out with storage industry companies.
Lots of other slashdotters have recommended CD's and DVD's. These work just fine, even for me. I've had CD's that got burned 10 years ago and they're working as fine as the day I got them burned. Granted, I've never found a practical purpose for those data nowadays except to bring up good ole memories of papers and such in now defunct word processor formats.
Tape is the best answer, they're tried and true and time tested. I believe that they have a 30 year shelf life if stored properly (evironmental controls are important!).
If you have an insane amount of data to back up (TB-range), I suggest tape libraries. These babies come in all sizes big and small. You can also opt for Disk libraries as well.
Now, to sum up...
1) If you are going to backup and forget, try cd/dvd or tape.
2) If you want to do incremental backups, try tape or disk.
3) ALWAYS make sure your closet or shelf or wherever you store your backups is in the optimal environmental settings specified by the media manufacturer - it will make sure that it will last its shelf life.
My first question would be whether you really need to back up all that 220 GB of data?
/etc, and a few other miscellaneous bits. That's a tiny fraction of the data stored on the disk, but it is pretty much all of the real information. If I lose my hard disk, I reckon I could get up and running again in a day or so (which for me is fine), and I'd probably find I've cleared out a load of cruft while I'm at it.
If that seems heretical, think about it for a moment. Do you have 220 GB of information, or do you just have 220 GB of data? How much of it could be regenerated? mp3s ripped from CD can be re-ripped. Someone mentioned the enormous backup requirements CGI animators run into, but in the worst case, could these rendered files be regenerated in a few days from the original script? You have to weigh the cost of waiting for it to re-render against the cost of backing the whole thing up.
And what about installed software? If you know what is installed, you only have to back up the machine-specific configuration and customisations, because you can re-install the software. It will take longer, yes, but again, you weigh up the pros and cons.
On my home system, I only back up my user data directory,
I did this *BUT* unfortunately with cheap porous rock.
Data was unreadable after 5 years out in the weather.
Always use quality granite!!!!
In Soviet Russia, articles before post read *you*!
Why don't we just sue Hollywood for keeping better media from becoming available? I have heard of new cdrom drives that can backup I thought I read 600mb on one cd with a blue laser, but you can bet the Hollywood is going to block this as just think of how many movies you could get on that.
well first of all, I've never seen any hard drive evaporate :). But beyond that technicality, there's no such thing as permanent storage. There are relative degrees of permanence and generally, the permanence is inversely proportional to the convenience. Storing a book on stone is pretty permanent, but it's a pain. Even still, the stone will slowly dissolve over time, and you could accidentally drop it and have it shatter.
You need to determine what is good enough for your needs. Will a hard drive last 5-10 years in storage and still retain data reliably? What about a DVD? What about paper? I have a computer running as my router, the drive in that dates back to my junior year of college making it roughly 7 years old. Still works fine.
This sig has been temporarily disconnected or is no longer in service
Yes. It's also a good way to pretend you're swamped with work. "This stack of paper in my inbox? It's actually Coldplay's latest single."
I used to work at a company that used DLT tape backups running on a stacker hooked to a Unix box backing up 300Gb. The software (and history files) for the DLT was backed up on a separate machine.
The backup drive failed, and during the time we were rebuilding the drive, the main unix drive was lost.
DLT tapes without an archival history are next to worthless. We were without backups for a week in a production environment.
Seems to me that you should use the most modern solution out there. You want off-site storage and you want redundancy and you might like it to be distributed.
Sounds like P2P would be the ticket here. Just upload all your files onto Kazza and Gnutella and then let nature take its course, scattering them all over the internet.
Anybody see a problem with this? Seems like a "legal" use for P2P has finally shown up.
I have had my share of IDE drives die over the past year and a half(about 25). With the quality going down I would strongly reccomend against doing this,stick to tapes. If you do drives and tapes that would be fine but for sure I would not trust "modern" IDE drives to sit on a shelf for 10 years and expect to work flawlessly at the end of the period(even testing them every year would be scary). Get a tape library if your out of space, A few years ago I was lookin at some tape libraries, 15 tape capacity, 20GB per tape(uncompressed) running about $8k or something. Excluding tapes of course. Could probably fit about 400GB of compressable data on the 15 tapes. And the tapes would have a much better shelf life I think. Sure
it costs more, but thats the price you pay if you
want reliable storage. If it's too much then cut
down on what you need archived. At my former company I found that whoever ran the tape drive was lazy and just backed up everything which included a lot of data(75% of the tape) was games, MP3s, and other data that had not been modified in at least a year or 2. Find out what data doesn't change, put that on dedicated tapes, then mark it read only so it cannot change. This may drastically reduce the amount of data that has to be backed up again and again. I prefer level 0 backups, so I only need the 1 tape to restore all the data.
I just flat out do not trust IDE disks. I do still use them but they are closely monitored and most are in RAID configurations and are backed up to tape.
If we are discussing paper, how about the perforated kind -- used to store most data less than 20 years ago.
All in all, however, this is just about:
I'm afraid, the first point would for a long time be required "for comfort"...
In Soviet Washington the swamp drains you.
And just how many tons of paper are you going to need to reliably back up a terabyte in dots and dashes?
Assuming double the standard density (160 chars per line instead of 80, 132 lines per page instead of 66), which actually works out to quad density, you get 160x132=20120, say
- 20k per page
- 50 pages = 1 mb
- 50k pages = 1 gb
- 50m pages = 1tb
Now let's assume boxes of 5000 sheets. 10,000 boxes, at, say 20 pounds a box = 200,000 lbs, or 100 tons. Man, give me the toner franchise for this!I think there is a company called FileStore what makes 200 disc DVD changers with integrated DVD writer.
Even if the machine breaks, the DVD disks would be good in any DVD-ROM drive.
and second, and third.
DON'T use a hard drive for your backups. Find something, ANYTHING more solid, more reliable, and more compatible - the gods know what hell you'd go through using one of those 10 years from now - but I'll bet you things will still read your tapes and CDs.
Hard Drives having moving parts. For long term storage, moving parts==bad. They're highly vulnerable and wear themselves out, also bad.
One thing you may want to consider is a backup heirarchy. Some things get backed up so often on such and such media, etc. Maybe some things CAN get backed up on a hard drive, but it shouldn't be anything you can't live without or recover from loss quickly.
I say this not as a hardware guy, but a programmer who's seen storage go wrong all to often - then *I* get to rebuild or find things. Or, I have to think backup because NO one else has.
"The Sage treasures Unity and measures all things by it" - Lao Tzu
...sounds crazy, but you could end up with a pile of drives that it takes something out of the Boston Computer Museum to read.
Anyone remember RLL? Or those fun little 800MB tape drives? Magneto-Optical? Can you mount any of 'em without going in your junk room?
Plus, drop an IDE drive and chances are your data is unreadable. Drop a tape and you pick it up.
DAT and DLT drives seem to be a pain to buy and operate, but they are backward compatible for the most part. Put them in a magnetically shielded storage case (like a fireproof safe) and you are good to go.
Bits rot. Under the most perfectly controlled environment the damn stuff still goes bad. Be realistic, anticipate this, do everything you can to slow it down, but plan for it and make provisions when you first put your archiving strategy in place. Tapes are likely more robust the platters as there's fewer critical parts to go wrong but nothing is perfect.
Yes they're cheap but we've far less experience with these media then we do with tape and studies are showing that they dyes may not be as stable as first thought. Heck, there's even a bug out there that eats some of these. There's also the question of long-term standards in some cases like DVDs.
Nothings worse then losing one part of an archive at one site, another part at a different site, and being unable to easily reconcile the two to get a good whole set. Make sure that however you archive things, same media or different media, that partial archives can be reconciled.
Years ago there was a big scramble to recover the US Govt's 1950 Census. It had been stored on steel tape and the required Unisys readers were no longer. (Much of the data was available but the entire raw set wasn't.) Eventually a working one was built from cannibalized parts in museum and private collections but the lesson was clear: Don't depend on the readers. The same goes for the recent BBC Domesday Book debacle - nobody could read the optical disks. Any good archive scheme will call for the material to be re-read and re-transcribed regularly in order to ensure the entire recovery-chain still works: Hardware, software, OS's, etc. If recovery becomes difficult migrate the material.
All too often folks archive everything 'cause they're too lazy to determine what is actually necessary and what isn't. Combine this with the difficulty of later having someone unfamiliar try to winnow down the material and this becomes a real problem. Even worse is later trying to find the useful material among all of the dross. Establish clear policies of what can be archived and make folks justify their material. Just as importantly make sure the costs are clear up front, even to the point of charging them a rate covering several years of storage initially. Suddenly some pack-rat deciding EVERYTHING they've ever typed is potentially a goldmine isn't so funny. Lastly, run everything past Legal: Some of this they don't want hanging around any longer then necessary.
I don't read ACs: If a post isn't worth so much as a nom de plume to its author then I wont bother either.
The last time I did a full backup, ~5 months ago, I used over 70 cdr's. Sure it sucked, but they are cheap, the format is standard, not going anywhere soon, and I didnt have to buy any new hardware. It looks like DVD formats have stabilized a fair amount, and most data doesn't change often. If I had 220 GB, I would probably write a script that would compare time stamps on files, against a prior backup, and then you are only doing incremental backups. If your data is worth more than the reliability of DVD, your data is worth enough to fork over the 5 grand for a professional tape drive.
-- the computer doesn't want any beer, no matter how much you think it does. NEVER, EVER feed your computer beer.
Check out PowerFile for a good large size long term backup option.
Pros:
* Gold CD-Rs have a shelf life of several decades so a PowerFile full of gold CD-Rs would give you 160GB of storage that would last longer than you live. I'm not sure what the shelf life of DVD-Rs are but if it's at least ten years, then you can have close to a terabyte of storage.
* You can populate the slots as you need them. Start with 50 DVD-Rs and then add more when your data needs increase.
* The data stays "online" so you can still access it (unlike with tape) so you can free up some hard drive space by moving stuff you access infrequently
* If you use CD-RW or DVD-RW discs then you can perform an incremental backup and save some time
Cons:
* More expensive than they should be...a Sony 300 disc DVD/CD changer only costs $700 yet as soon as you connect anything to a computer it is suddenly a couple grand more
* Slow, obviously since it has to spin to the correct disc to access it. However, for sequential operations like backing up (writing one disk after the next) it should work fine.
* Software sucks...they actually have the gall to charge extra for the drivers that you access the drive over a network. I'm not sure exactly how but basically you can't change the disc over the network unless you pay extra
What I would dearly love to see is someone like Linksys make a cheapo version of this, but instead of designing it as a stupid fireware component, toss in their little linux flash computer and make it networkable from the start.
OR...maybe someone smart out there can figure out a way to make one of those home theater changers into an ATAPI device that I can wire in directly to my computer, or said linux flash computer.
Is anyone aware of anything close to these PowerFile things, only at a level that is more in line with what they actually are (just a DVD drive with a little robotics to insert and eject discs into a carosel).
- JoeShmoe
.
-- I wonder which will go down in history as the bigger failure: the War on Drugs or the War on Filesharing
I have a 20 mb (yes, you read that right) hard drive from 1989 that I can still read just fine. I've hooked it up once or twice over the years just for the nostalgia.
Cogito ergo sum in Slashdot.
If you're using a good incremental backup system, that only rearchives the differences between the current backup, then you shouldn't need to copy this 220gb each time, far from it i would have thought.
This is what i do:
Have a machine with several hard disks as far away from the main ones on my ethernet as i can get them. Backup over 100mbps onto this machine. Then, every week this machine uploads the only the truely vital (and compressed) data to several FTP servers around the world. You could script this so it was just a steady stream of upload that went through the course of the week, and you probably wouldn't notice much of a spike in traffic.
It's unlikely that i'll ever have to use the servers in other countries, because with simple HDD failure, there are others in the backup box. If the whole building goes up in smoke, then i can restore everything from a server in another country. The chance of all of these servers going down symultaneously and losing my data is very slim, and so i don't have to give a crap what storage medium and such they are using.
Not only is this a very good backup system in my opinion, it also means i never have to worry about carrying tapes round etc, and it is very easy to automate this as it doesn't rely on changing any storage medium round.
Obviously, you've never had a tape physically fail.
Maybe it's just me, but after the experiences I've had the last year with crappy tapes, I'm surprised the "tape as a backup medium" idea hasn't been seen for the farce that it is.
Here's the thing. Tape is fine as a short term backup medium. It's relatively cheap (and I do emphasize the word "relatively"), which means you can throw it out after a year and get new tapes. But anyone who expects to be able to retrieve data from tapes that are four or five years old is dreaming.
Long term solutions? As you say, it's time for something new. Tape or disks simply aren't viable in the long term, and optical, while more permanent, has not kept pace in terms of speed and capacity.
Stegnographize your data and hide it in an amateur pr0n video.
To restore from backup, search with Kazaa.
Tuus crepidae innexilis sunt.
...removable hard drives. Pick one or the other (or use both, for greater redundnacy). Better yet, here's a neat drive enclosure: http://www.addonics.com/products/external_hdd/comb o_hd.asp . Add the appropriate cable to the back for USB, Firewire, IDE, or PCMCIA connection. How's that for versatility?
With the USB and/or Firewire hard drives approaching or exceeding 120 GB capacity, you'd only need a handful to get a decent set of rotating backup images. And with multiple interfaces on some of the drives, you have a greater chance of being able to read the media in the future. Good Luck
How's my programming? Call 1-800-DEV-NULL
Well, there are bad CD-Rs and good CD-Rs. I have some very old burnt CDs that have outlasted some hard drives I've met. I kept them in jewel cases in a relatively cool place (canada) and they still work just fine.
I wouldn't back any data onto a parallel ATA device for use in the far future.
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
I think most people have missed the point here.
If the backup is of data that must be archived and stored long term then it's worth sitting down and copying it onto a pile of DVD-R's or similar. They will still be readable when a hard disk will have long since siezed up.
On the other hand if the backup is simply to guard against those "what do you mean you typed rm -rf *" moments then a copy on removeable hard disks will be more than adequate.
Personally I think a combination of the two is probably best, optical / tape media for archiving, and HD's for everyday stuff. For example, I don't need all my MP3's on tape but my wife has a copy of her thesis on CD-R.
Steve.
A latent existence
Failing to have an OFFSITE backup can result in your company losing all it's assets. What is that worth to you and your company?
I bet more than the 40 dollars per tape for DLT (I doubt it would be more than say 3-4 tapes per full backup and then 1-2 per week incremental, but it depends on your data).
At $240/week or even $500/week in tapes, you can't justify NOT having them if your company makes ANY type of living off the data you are archiving
Is stiction still likely occur on newer makes of IDE drives or have manufacturers beaten the problems which caused this in the past?
My guess is that most manufactures have improved on their designs in order to minimize this problem, especially since companies like maxtor have been pushing their 5400 RPM Drives as a backup solution, but there is still a chance that it could happen and for an offsite solution it wouldn't be the best choice
Likewise how likely is bit drop-out and general data degradation over say a 5 year and 10 year period, and what do people think would be the likely maximum feasible time that a shelved drive would be usable for?
I had 30 IBM ps/2 50z in my basement at one time. I think 3 of them had good hard drives. 3/30 is not good odds considering that they are only 15 years old. Newer Drives have improved on these older mediums but they are also more dense, increasing the chance of degradation over time.
Any suggestions as to how would I need to store drives in order to minimize these types of problem and maximise their feasible life as archival media.
Keep them away from heat and humidity and spin up frequently. although some manufactures vary is their suggestions
Basicially, a Hard drive array is a good idea in a daily to weekly backup schedule. especially if they are networked on a rack or in a server and have RAID 5 reliability.
Anything beyond weekly should be done on tape backups. the ATL Tape Libraries are nice rack tape loaders, and are networked.
also make sure that the backup system can be racked. if they can you can put a backup rack in a different networked building from the servers and that could constitute an off site backup in many cases.
In Soviet Russia, Trojan exploits YOU!
RAID-5 will make your system more reliable, but not bulletproof. Two weeks ago I had a 1TB RAID-5 fail on me because a SCSI cable went haywire. Guess what, even on RAID-5 you have a few disks per bus, so you're still at the mercy of a single point of failure. Unless you have dual-attached FCAL, of course. :-)
Why shelve the disks? You brought up a couple of very important reasons not to shelve IDE drives; namely, stiction and life span.
Rather, use a couple of live backup computers, one in-house and one off-site. Put large RAID 1, hot-swappable raids in the two computers and schedule these computers to periodically back-up your data.
The RAID 1 buys you redundancy for a drive to fail and be replaced, without losing your backup.
The two computers (or three, or more, up to your heart/wallet's content) buy you redundancy for a computer to fail or be destroyed (and take the drives with it).
The live computers buy you the freedom from media and from worry. If Serial ATA becomes all the rage, you can just swap in a Serial ATA (or Ultra-New-Large-Storage-Media) based backup computer. Plus, a live computer (or a watchdog computer, which watches to make sure your backup computer is actually up), lets you know if you need to replace a faulty drive or faulty computer, before it's too late and you have three bad drives on the shelf.
So, in summary. Don't shelve. Run live redundant backup computers.
I think this is viable.
-disks will not be in constant use.
-disks can be rotated every month.
-carefully stored disks will retain integrity for quite a long time. years.
-cheaper and faster than tape.
-I HATE tape. Any and all solutions beat tape for my money.
-Someone somewhere in the company will have a system, even 10 years from now, that will be able to read the disk as long as a standard file system is used.
As long as we are on that track, the Internet was designed to withstand nuclear attack, so its obviously the best choice: archive, encrypt and have others mirror your data.
I know, I know, how do you get these people to do it? And how much will it cost? Easy, and I can get them to do it for free.
Name the backup DIVX_The_Twin_Towers.avi and put it up on Gnutella or WinMX. Problem solved.
I tried every decent and legal way I could think of to resolve the issue w/the business before I rented the chicken suit
However, if this is going to have *any* chance of working, you will need to read the drives on a regular basis. I would pop each drive in a machine and (in linux) do a "dd if=/dev/hdc of=/dev/null" to read the entire drive. I would do this monthly.
Why you ask? Because modern hard drives are sophisticated and they auto-correct errors *before* they become a problem. Hard drives will do things like correct recoverable errors and rewrite weak sectors when they encounter them. Thus if you go over every sector of the drive every once in awhile, you will use the drives auto-correction features to your advantadge (and protect against the drive fading, which would be my primrary concern, not stickage (which is easy to fix)).
Religion is a gateway psychosis. -- Dave Foley
So, like, do EIDE drives have a shelf life? If you get a new one, do some burn-in testing, use it only for backups and restore, the drive will 99% of the time on the shelf, hopefully in a safe environment. I bet it would last a Long Time but still, if the data is important, you have backups of the backup, although a rotating schedule will ensure that any single loss will lost only a certain amount of data.
For really long storage, the EIDE bus itself is going to be an issue. You could put the drive in the $199 cheapie computer from Wal-Mart but then how long will 100 mHz Ethernet last?
Yea, if you have the budget for tapes, AND wait to wait for hours to backup, and DAYS to retrieve.....
Well, maybe not days, but it feels like it when your restoring on a saturday night and have to wait for it.....
No I didnt spell check this post...
Alright, I'm just a home user, I don't have $thousands to spend on a tape loader and DAT/DLT/AIT/ETC tapes. I need a better solution. I see only DVD-R as a viable solution, but even 5-gig per disc means that I'll need close to 100 discs to do a full backup. I want a 10 TB hard drive with an attached, automatic, 200 TB will-never-fail backup medium. Ok, I dream.
Wer mit Ungeheuern kämpft, mag zusehn, dass er nicht dabei zum Ungeheuer wird. --Nietzsche
Instead of dvd-rw, try dvd-ram. Currently, you can get the media for close to $1 - $2 per gig. And the technology is actually the same as magneto-optical (MO) drives. But, since it's part of the DVD standard, the media is cheaper per gig than MO. The tech is phase-change, so it's not sensitive to light or magnetic fields.
Just another Anonymous Coward here with a slightly different anecdote to tell: I too have been saving various digital rubbish on a bunch of old 1.08G, 1.2G and 1.6G ide drives -- several of each size. I'm finding the 1.08G drives, made by Conner of all people, are retaining the data and spinning up just fine after a few years of cold storage. The 1.2G and 1.6G drives are WD brand and all of them seem to be suffering bit-rot and I'm having to try to use old Norton Utilities to get as much of the old files as possible off of them with mixed success :-/
Beware of using old ide drives for long term storage, some seem to get alzheimers after a period of extended storage.
Agreed, I've been looking into the same thing... having 230GB of data (with capacity soon to be expanded to 460GB) and no backups per se... just because the idea sends a shiver down my spine.
I'm also reluctant to get extra IDE drives to use as backup media, even if they're only use when transferring... all it takes is a magnet happy nephew/niece or an accidental drop... and the provebial shit hits the fan.
But looking on NewEgg.. you can pick up DVD recorders for under $400 (my friend recommended sticking with DVD+RW).. and you can buy a 50 pack of 4.7GB DVD-RW disks for $65 (there we're DVD+RW disks but I can't find them now). With storage capacity making CD-R disks look paltry, DVD recordable seems the way to go for our needs at least.
And for keeping tabs on what is on which disk... I've been using a freeware program called "Cathy" (I don't have any links) for all my CDR archives... and this program ROCKS! (It just rips the file data in about a second, rather than other programs that take 5 minutes to traverse and log the files) Although I don't know whether it'll do DVD's, I haven't tried.
Are you local? There's nothing for you here!
We used to always make two identical backups. The tapes would be automatically called up by the archive system and would be wound and checked. If a tape contained too many bad spots then it would be recovered by a copy from the remaining good tape. Note that bad spots were generally regarded as reoverable as each tape block was in a group protected by an XOR redundancy block which could recreate any single block dropped.
The winding/rewinding as the tape was checked got over any problems with tape settling and this process worked reasonably well with several media changes over about 20 years.
See my journal, I write things there
Dark ages technology. Burn the tapes. Tapes are expensive and they goof up too and there linear meaning you can just search it for one file easily.
Just burn it to CD's, it's cheap ! I could never understand why zip drives became so popular when the disks cost a small fortune were slow and goofed up all the time and a comparable cd held 100 times more data and cost 5 cents.
DVD would even be better but it costs more.
People using tape backup need there heads examined.
RAID is not a backup solution. RAID improves reliability in the face of drive failure, meaning that it will ensure that your rm -fr / will execute even if you lose a drive in the process.
For current drives though, I'd say "No way." The advances in drive storage size come from pressing more and more data into smaller spaces, meaning magnetic drift in time will affect them much more adversely than even older drives. Smaller and more compact also means the internal mechanisms need to be more precise, narrower tolerances for more points of failure. Older drives were more robust in many ways. 350M SCSI Seagate, read head came off one arm, wires shorting out on the platter. Took it apart, removed the platter, and the damn drive served without flaw for 3 more years in the home server until the box was retired. Try -that- with a drive nowadays...
Rotating backups on tape (with a tape cleaning & replacement schedule), off-location backup rotation, and 'hard medium' backup (CD-R, DVD-R, -not- R/W) of crit. files on a monthly/quarterly basis, and you can be covered for just about anything...
There's no wrong way, to eat a Rhesus...
....is time to restore. You back at off times and you generally do incremental backups, right? You have a failure. You lose it all and it is prime time, Big People are depending on you, this is costing the company money.
So, you reinstall the operating system and then the tape backup/restore program, you start the restore and it takes 10 FOREVERS!!!!!!!!!!! This system was installed years ago, upgraded since bit not one tested how long it would take to restore gigabytes. And gawd, help the poor soul who loses a terabyte.
The last full restore is as quick as you can change out the drive. Then you need to do an incremental restore, but most of the time is over all ready. Nice and worth the price if you are do any kind of profitable business.
You have to treat harddrives as unreliable pieces of crap that will eventually fail. Once you have accepted this fact, then yes it is possible to do backups onto IDE drives. It just requires that you keep making copies of your data such that you don't ever have one single point of failure at a given time.
Optimally, you'd have a pool of different computers networked at different sites, and you'd just have them replicate all of their important data all of the time. If one goes down, you fix it asap and continue.
It would be nice if there was a distributed filesystem that did guaranteed replication of data. Maybe one of the P2P applications could be set up this way such that you could backup your harddrive and guarantee that none of the files went away even though N different nodes failed? Anyway, good project for the future.
The computer department I'm part of hasn't taken backups for around a year and we have recently restarted the backup procedures.
:-)
Everyone is thinking their projects has been backed up regularly at least once a week or so.
I think we'll just keep quiet about that.
Beware: In C++, your friends can see your privates!
"Real men don't use backups, they post their stuff on a public ftp server and let the rest of the world make copies." - Linus Torvald
So maybe you could do it like this:
- on your backup machine, create a single gigantic backup volume
- do a full backup to it once, and again every week
- spool the full backup to tape at your leisure
- do incremental backups to the volume every few hours, or minutes, or whatever fits your schedule and makes you happy
- spool the incremental backups to tape at your leisure
- swap tapes offsite as per normal backup methodologies
Now you have superfast backups, superfast recoveries, redundancy, the best of all worlds.But on the whole IDE drives are commodity-scale junk. If you're using them as your primary, make sure you're doing at leastRAID-5, but more sanely mirrored pairs, and most sanely springing for a DAT drive.
This is not my sandwich.
Your question made me wonder, so...
I pulled out a 386sx board and booted up an old Novell server I had on a 40MB IDE drive from 6 years ago.
No problem, and a scan of the drive indicated it was in good shape.
Of course your mileage may differ
If you want to trust your data w/ Iron Mountain, go right ahead! Never have I seen a more incompetently run corporation, especially when you consider what they do, which is "safeguard" your data. Safeguard, yeah, like putting ID labels on DLT tapes so they can't be used again without removing the polyester film tag(very hard to remove) from the tape door. Or, simply leaving your data in a hallway instead of following the instructions left for them when they make drop offs. We don't mind the whole world having access to our sensitive data, no - the hallway is a great place to leave something important unattended. These are things that we actually had to deal with, aside from the missed pickup dates, endless billing problems after service was cancelled, and the absolute living hell we had to go through to get our last batch of tapes back once we decided to drop their service.
All this because there is no competition in the world of offsite data storage.
That being said, I think a large pile of IDE drives could be a good backup mechanism, when used as part of a dedicated backup machine. I would take a low-end PC, stick an IDE RAID card in it, and slap in 6 decent IDE drives in a RAID 0+1 configuration (striped mirror pairs). This gives you a large degree of redundancy, and you don't have to worry about interface problems because you are treating the whole system as a black box -- and I doubt ethernet is going away any time soon. A wake-on-lan ethernet card would be a good addition if you don't want to keep it energized 24x7. Stiction shouldn't be a problem since you'll be powering the machine up on a pretty regular basis.
You should be able to build a backup server like this for $1200 or less. Ideally, you'd have 2 of these beasties, one on-site and one off-site. Either swap them out manually or synch over the network, it really dosn't matter. I'd be a whole lot more comfortable moving around a single tower case than a bunch of IDE drive trays.
Why is it that the proponents of "one nation under God" are so eager to get rid of "liberty and justice for all"?
It doesn't give random access to the discs, but for backups and archiving you don't really need that. There are also comparable devices from Primera and others.
That said, the DVD Changer thing looks like a pretty good product except for the stupid marketing around it. I'd rather just buy the mechanism with no software and run it myself. Do you think the mechanism itself is crappy, or just the way they sell it?
The reason those disk fail is that the drive head physically rests upon the media in an out of band area. This gets "sticky" over time and damages the head or media when you finally try to spin it up. (So says our Seagate rep.)
but the cheapest way to go is to get several 100,000 or so HD 1.44 floppies. just rotate them through every month or so to keep them fresh, and put those aol disks to good use.
I have a pop3 mailserver which I maintain for our company. I'm planning to switch this to IMAP - having all the users mail on that one box.
THe backup plan goes like this:
Is this such a bad idea? Should I not use the 120gb drives for the mailstore in the firstplace?
Sure, drives will fail in continuous use, but running them for an hour every two weeks, and in this case, doing an archive (dd if=/dev/hda1 of dev=/hdb1) where the head isn't skittering back and forth a million times, isn't going to cause much wear and tear.
Huh? The head doesn't touch the media so there is no wear there. The coil doesn't touch the magnets so also no wear. And somehow I think the wear on the pivot bearing of the actuator is the least of your worries with drive reliability. More of an issue is head degredadion although this is planned for in the design of drives, and bearing failure due to overheating. Most harddrives should last through 5+ years of continuous use.
The next best thing to rock and chisel.
last time i was trying to find a (relatively) inexpensive tape backup solution, i remember that ecrix had some promising looking stuff.
anyway, it appears exabyte acquired them and since i last looked there is now a 2nd generation drive--80GB native. drives are SCSI or firewire, and are ~1000.00. not sure what the tapes cost.
anyone have any experiences to relate regarding these? how these compare to dlt? i see that ibm is selling these as an option to some of their servers now.
Given that most suggestions have been based on corporate environments, how do you guys propose backing up a bunch 20-30GB of MP3's and Photos?
With the increase in availability of hot-swappable drives via USB2 and Firewire, I've toyed with the idea myself. However, some backup systems such as Legato Networker don't seem to know anything but tapes - has anyone developed a virtual "Tape on Disk" driver? I haven't found one, and I've thought about writing it but don't have the time.
rm
Sci-Fi Storm
"No way, you are insane. IDE is horribly unreliable and you will surely lose your data. You need a $6000 tape drive, if you can't afford it you are better off with no backups at all"
Oh really? Can you back that up? Even if someone plugs up a 200GB drive once a month for a year (without dropping it) I seriously doubt they'd have a problem getting data a year or two from now.
Even if one wanted to keep the data around 10 years, which realistically isn't going to happen on the *exact same* media, he/she probably will change capacities/interface types every few years just for bigger capacities.
As with anything, there's pros and cons with hard disks vs. tape. The popular myth is that hard drives that are sitting idle for a long period of time have a shorter usable shelf life than DLT or 8mm tape. If the drive is properly stored (not in a box on a shelf) it should outlast the tape simply because of the nature of the hard disk's environmental seals.
Some experts say that magnetic media such as tapes and disks can be kept safe for five to 10 years, whereas optically etched media such as CD s and DVDs can have an average life span of five to 30 years depending on the whitepaper you reference. Since you can only put about 4.6GB on a DVD, it'd still take a few to get the storage of a single DLT or 8mm tape.
The problem isn't with dependability so much as it is usability or affordability. Hard disks are typically inexpensive in these times, unlike back in the mid 90's, when it would cost you $275 for a 4.3GB hard disk. You can get 5 times as much storage now for half as much $$$.
The reality is, you can typically store more data, especially non-critical data, on cheap hard disks. I have a Compaq DLT tape drive that I use to store all my MP3s on because it's a lot safer to archive long-term data on DLT than it is on disk. I also have an Exabyte 20GB 8mm tape drive that I use to backup audio and video with, as well as other typically static critical data. Otherwise, the two workstations I have in my basement have Kingston pull-out drive bays and I swap out between 20GB disk drives almost daily. I take a 20GB drive to work and anything I store on my workstation there gets backed up to that drive, and comes home with me nightly, and placed on my machine at home.
Additionally, I built a FreeBSD fileserver that has a samba share for the Win2K machines and that also serves as my tape server. That fileserver has a pair of 60GB hard disks in it and whatever sits on those disks gets backed up to tape nightly, using incremental backups thru the week and full backups on Sunday night.
Using large drives is a cheap and easy way of having dependable storage but I don't think it's the best for long-term storage. If you slap a DLT tape or drop it, it'll tend to continue to hold data, whereas if you drop a hard disk... well, you get the idea.
Unfortunately as the density of hard files have increased the chance of a bit flipping has also increased. Current hard drives constantly do background scrubbing to make sure the data on the disk is OK. If you left the disk unpowered for a significant period of time, the chance of getting an unrecoverable error in a sector is greatly increased. If you are going to archive on hard file, I would reccommend creating an enclosure to allow the drives to be powered and thus be able to scrub. If you are going to do that you should just buy a drive enclosure and leave the drives online.
I went to a system like this a while ago and it works fine. Hard drive costs have plummeted while backup device have slowly gotten bigger and cheaper. I recently pulled some zip drive from three years of storage and none of the data was useable. I currently backup on hard drives then with my important files I burn them to DVD. I do graphics work and have started a system where I back up nightly on a DVD-RW then after a weeks use archive that disk. Once a week I also burn DVD-R so I have two back ups and a hard drive one for every file. Shots are also burned to DVD-R immediately after they are rendered. I love pull out drives and have even gone to them for my primary hard drives. As to tape, does anyone remember what happened to the IRS? Tape is a lousy long term storage medium. DVDs are probably the best we have for now.
But is printing a whole character per bit, or even byte, efficient? I'm curious how much data a laser printer could store on a piece of paper. Is it realistic to expect individual bits printed at 300dpi to actually be retrievable? Perhaps on a good 600dpi or 1200dpi printer.
300dpi gives us almost 11KBytes per square inch. Figure 70 square inches on a letter page with 1/2" margins. That's 770KB. Print full duplex and you're looking at 1.5MB per page, or roughly a floppy disk (coincidence?) You wouldn't want to back up your MP3 collection, but for an archival method that is likely to last 100 years it's not too bad. Factor in compression and you are probably getting a 100x increase in storage density over plain text. Kind of a neat thought.
In the early 90's we spent $1500 for a 3 gig drive that we used to back up our workstations. We then backed up that drive to tape. It was infinitely faster than screwing with tapes in the night.
Right now I am backing up 53 workstations to a hard drive file using Retrospect. I then copy the file to another server and backup that server. Somewhere, I will have a copy of those backups because it exists on two machines and a tape.
If you aren't part of the solution, there is good money to be made prolonging the problem
They do exist, but are fairly expensive. I've never seen one but everything I've read says they go like shit off a shovel, not sure how good they are for long term storage though.
If you were actually going to produce some kind of machine-readable dead-tree backup, it's more likely that you'd produce a type of 2D barcode that could be scanned back in and read. Assuming an 8x10" grid at 200 dpi (the remaining area can be used for alignment and checksumming), you could get about 390K per page (single-sided...you could also double that by making it a "flippy," and you wouldn't need a notch-cutter :-) ). You're still looking at a little over 5 tons for 1 TB, but it's an improvement. 200 dpi should be well within the abilities of currently-available laser printers and scanners. If you wanted to try 300 dpi, you'd more than double your capacity and get about 879K per page (single-sided).
20 January 2017: the End of an Error.
As disk space grows, so does your family/backup.
To see examples of how this works see: Mad Max - Thunderdome, The Bible, American Indians, The Fellowship of the Ring, Aesops Fables, and the Legend of How the Great Nog Vomited the Earth and Heavens in Ancient Times, Before the Oceans Drank Atlantis.
I have heard rumors that this is how Google archives.
But I think most /. readers would agree with you, most hard drives should last through 5 years, but most don't/won't/can't be depended on, certainly not for archival purposes, where it has to work.
Most the people I talk to, and reading many of these replies don't understand the difference between archiving and backup. Before hard drive prices became so cheap both archive and backup were handled the same way, but now you have look at the difference.
Archiving data is usually for business reasons, usually legal like for IRS. So this is data that needs to be able to be restored at some time in the future. For IRS I believe you have to keep business records for ten years minimum. Last place I was at tapes were on a one year rotation cycle, and we made a yearly archive set never touched.
Now backup is usually for safety reasons. So the dumb ass marketing jerk who wipes out the new ad copy due to the printer tomorrow can be saved. This is usually for short term and hard drives doing snapshots are good for this.
I worked for one the major ISP's and email is all stored on NAS storage and they keep about five days worth of snapshots to do restore a mailbox if necessary. Same with web content. This stuff to too dynamic to waste time and money doing tape backup. Tape is only done on business data.
So today to save money and backup-window time you need to design storage systems dividing data in to what needs to be archived vs. backed up.
Also when designing new archive systems don't forget to plan to archive your old backup server and tape library. What good are tapes if you don't have the software and hardware that can read them to do restores.
I have three computers with up to 120 GB between them and need a backu solutions that will:
t ml ?Enclosures/IC1394USB2.js
- Backup an raid 5 linux
- NT2000 box
- Laptop with NT and Linux
I had decided about 3 days ago to go to IDE drives because:
- 30 GB of tape runs around 64.00 (usd)
- I have three computers to backup
- My releatives want me to back their systems up.
So I decide to go the USB/Firewire route because I
have different plugs on different boxes. The
current enclosure I'm looking out is below.
http://www.centuryglobal.com/Product/product1.h
I can stick a 3 1/2 inch drive in and it supports
USB 2.0 and firewire. Which also means I can use
it on Macs.
Problems:
- I need backup software to break across multiple
drives.
- The enclosure has an external power supply.
(pros/cons)
Hey just my 2 cents.
L. Donaldson
A home user with 200GB of files should drop the Jergens and Kleenex long enough to get a DVD-R and some blanks and burn that pr0n onto DVD so they free up all but 10GB of that drive space and get to watch that pr0n on TV, where it belongs.
I tried every decent and legal way I could think of to resolve the issue w/the business before I rented the chicken suit
Don't back up the porn. Put everything else on 2 floppies.
You souldn't drop anything onto the floor anyway... The floor is used to walk on. No to drop things on it... If you cannot take care of your things, well, don't care try to backuping-it.
Have you ever drop a pc on the floor? No? Why you could drop an harddrive? Did you ever burn a tape with a cigarette? No?
All of these two event could append, but the question: Why would this append. Anyway if you drop the harddrive or burn it... you juste have to buy another... But if you do ether of these when you where trying to restore data...
I say shame on you!!!
220 GB is two SDLT tapes, with room to spare. Just use SDLT.
Most of the storage technology R&D money is going into hard drives which is why they are so inexpensive and will continue to be. Tape storage technology R&D money is simply unable to compete.
I've predicted for a long time that tapes will become obsolete and hard drives will be used to back up hard drives. A very interesting example are firewire hard drives. A small premium on the cost of an already cheap dist drive, you have a fully plug and play high performance solution. For the cost of a tape drive and 10 tapes you can buy 10 firewire hard drives and rotate the drives instead of tapes.
The advantages are enormous.
Fast recovery time
Fast seek time
Proven reliable technology and much more reliable that tape.
Inexpensive and becoming less so
Easily networkable (just mount the drive on the network) and fully supported by any self-respecting OS
When you're done with the backup, you have some spare storage for anyone..
There may be some work to do to make some of the backup software talk to a hard drive, but there are probably so many different solutions you can use, you probably don't need to worry.
I bought 2 firewire drives a while ago and have never looked at tapes since. Love it....
There are only two types of women in the world: The one's we've slept with, and the ones we haven't slept with yet.
The difference between an asshole and a troll is that an asshole is being, in some way, well, an asshole. A jerk. Being an asshole means that you have done something negative that * didn't need to be done that way.* Like swearing a lot making a point which didn't require the swearing for the point to be made. Your content was valid, even perceptive. But you were an asshole in the way you said it. You were *gratuitously* abusive.
The original poster who wanted to mod you troll was at least right in that respect in respect that some trolls should moddable UP though.
Some trolls are art and deserve to be respected and recognized as such. Right now we can only use funny for that, which isn't always accurate. A true troll has a *point.* To elicit response. Sometimes that response is baited in a way that isn't just art, it's *high* art.
Read the letters and short essays of Mark Twain. Many of them some of the finest trolls every penned.
Swift was a master of the troll. All of Gulliver's Travels is nothing but one massive, and brilliant, troll, and was treated as such at the time.
By the way, the difference between a troll and a flamebait is that the flamebait is a post specifically designed to illicit responses from assholes.
KFG
But how about a 600dpi laser printer, 8"x10"?
For good readability, we can use:For (1,0) which gives us 3 dots per bit, or 200 bits per inch. A square inch would then give us 40,000 bits, or 5,000 bytes. A sheet of 8x10 then gives us 400,000 bytes. Or if you tweak the margins, 400k per page. So that's already 20 times your density. Increase the resolution to 1200dpi, and you can increase the data density to 1600k per page.
We can also use different encodings: Right now we use 9 bits to encode 1 bit of information (really, really, redundant). We can probably safely use the following encoding to double our data density:So this further gives us 2 bits of information in the same 3x3 square, which increases our data density another 2fold: 800k or 3200k per page. At 1200dpi, that's 3mb per page, so that 1gb == 333 pages, and 1tb == 333k pages. 67 boxes, or 134 pounds per terabyte.
There are more variations of course. We can increase density to 4 bits per 3x3 square. With a bit of thought, we can also increase the density up to the theoretical limit of 2^9 values in a 3x3 square, but we want to include some leeway for data redundancy...
So by doubling to 4 bits per square, we require only 70 pounds per terabyte. By doubling again to 8 bits per square, That's down to 35 pounds.
That much (little) paper... is actually lighter than a terrabyte of digital storage!
GPL Deconstructed
Unfortunately, the so-called "archival" papers, while "rated" for 100 years, won't last anywhere near that long without some degradation. Then, if you're going to store it that densely, you've got to make allowance for putting the data into "tracks", so you have to leave spaces between each row. Cuts your 300 dpi down to, say, 100. Add check-summing data, so that you can recover from dirt, toner falling in the cracks, etc. And now, let's make the dashes twice the size of the dots. Cuts your storage by another 50%. Now, let's put spaces between the dots and dashes - otherwise, you get one LOOOONG dash. Your 11kb per square inch is now less than 0.5kb per square inch. Oh, and don't do duplex printing, you'll have transfer of toner onto the drum from the previously-printed side. Net result == about 30kb to 50kb per page... Oh well, maybe we should try microfiche ... or bit-encode the data into fake avi files and record them on VCR tape - cheap media for sure.
2000 sheets of 8-1/2 x 11, 20# laserwriter paper weighs 20 lbs.
First of all, this changes your estimate of weight from 100 tons to 250 tons.
Typical yield of paper: 125 lbs per tree
250 tons (500000 lbs) divided by 125 lbs per tree gives us 4000 trees.
440 trees per acre :)
This, after division, gives us 9 acres of trees destroyed for backing up 1 TB of data. Seem worth it?
FireWire would be better in this capacity...it's faster, it doesn't bog down your computer, and the controllers only run about $40 or so (though if all backups will be done through one computer, you can get by with just one card). USB's OK for low-to-medium-speed devices, but it's not up to handling high-speed devices such as hard drives.
20 January 2017: the End of an Error.
Any such software?
Ouch! Even worse! :-)
We do need another method for tapes...I think a hard drive backup system would work but I think you should consider creating a new server with a large RAID set. Then at least your backup set will have redundancy. Use TAPES to backup your raid and take offsite.
If properly stored, hard drives last at least 15 years. Then again, maybe older drives had more built-in tolerance...
I don't worry about backups- I'm sure Adm. Poindexter won't mind sending me whatever my disks lose! Especially if I let him have some of my backup savings. I'm sure he could find a use for the money....
Samsung still offers a three year warranty standard with their drives.
First, crappy backups are better than no backups. I think someone already said it, but that is definitely the truth. Second I would ask, have you thought about using a SAN. Depending on how often you do backups, hopefully not nightly if you are doing it manually, and also how much demand there is for restoration, you could build a Linux server configured as a SAN. Move the data to drives configured in RAID V, and then put the drives into storage for archival, I would really recommend rotating drives so that no drive sits longer than a few months, but that isa cost issue. With a Linux SAN, you could automate your backups and also give yourself a great restore system in the process.
----- "It's all fun and games 'til somebody puts an eye out, then it's just funny."
...the Beowulf cluster imagines you!
This space left intentionally blank.
To do a restore, just file a FOIA request!
Put it on a web page, make sure Google caches it, and voila, a back-up you don't have to maintain and it'll probably last longer than a tape drive or any given hard drive. Also slightly faster than the hammer and chisel method suggested by others.
Truth be told, I've thought and thought and thought about how to back up data reliably over long term and I have yet to find a good solution. I used to think CDs were great until I moved to Mexico and had my CDs eaten by a fungus. Now I don't trust them at all.
I also encountered a mold that attacked VHS tapes and I'm sure it attacks other types of tapes, so they're of no use.
Short of storing your media in a vacuum, I have no idea how you store stuff long-term without risk of loss.
Well, I say they should do nearly the opposite -- write bits more strongly, further apart so that the data can be read 10-20 years from now!
Let's look at a fictitious tape-technology timeline.
3 years ago - write resolution: 250 strength units, read: 150 units
Meaning: a fresh tape is written, reading it back yields a signal strength of 250 units of some sort, which will probably degrade over time; and as long as that strength is 150 units or more, the data can still be read properly.
This year - write resolution: 25 units, read: 15 units
The problem is that the ratio of write/read sensitivity is the same in both cases! The "ability to accurately retain data over time" has not improved over the years. We need the equivalent of:
write resolution: 250 units, read: 5 units
Meaning: we write the data super-powerfully, and as long as a tiny amount of signal is retained over time, we can still read the data.
If 1's and 0's blend into each other, then write a whole bunch of 1's to mean ONE, and a whole bunch of zeroes to mean ZERO. When you go to read it back in 20 years, the middle-most 1's and 0's will still be readable, to tell whether it's a ONE or a ZERO.
Sacrifice density for resiliency.
make sure your shit is worth it.. then go to your friend and if he likes the "shit" too.. he'll backup (copy)it... etc..etc...etc... if your frind refuses.. your shit is not worth being backed up... uhm.. uhm.. damn this comment is too informative and will get positive votes... hmm.. ok.. the usual insulting of americans: YOU SUCK! WTC SUCKS! YOUR PRESIDENT SUCKS! YOU CANT DISTINCT BETWEEN ARROGANCE AND WISDOM...
Personally I use a firewire drive for backing up my system and it works well. I don't know why people here keep saying tape is a better solution because it really isn't. It costs a lot more, is generally slower and tapes break too. Long term backup should be done on more solid media like CDs / DVDs. However day to day operations can easily be back up on HDs. If you're paranoid use more than one external HD. It's highly unlikely that they'll all fail at the same time.
Why not just use punch tape? Or better yet punch cards! Hey, it'd give the geezers a chance to reminisce... Umm, just don't have it punched in Florida.
The fact is you cannot trust backup data to sit on a shelf, no matter how reliable the medium might be. For very mission-critical, long term data storage, I'd recommend at a minimum once every 2 years, preferably more, you recover the data, verify it (through the use of checksums), and re-record it onto fresh media, and verify it again to make sure it copied right. Make sure you have multiple copies of each, stored in physically different locations, so if one goes bad, you have several others you can trust to work good.
More important than that, though, you need to decide what data you want to back up this way. Unless you're a large multi-national corporation, no one's going to want to spend a week doing this to a stack of 100GB tapes. Decide what's essential to have long-term digital copies of -- things like financial records, sales reports, product plans, stuff there's a chance you'd need to access 10 years down the line, and would need quick. The rest, if they really want, buy some archival-quality paper, print out a few copies (use high-density barcodes if its binary data), index them properly, vacuum seal them, and put them in a bank vault.
Why doesn't someone just make a "rent a backup"
Base it on a LTO drive from HP with 200/400 GB capacity, or AIT2/3 from Sony.
You rent it for a few days/Week and make the important backups, the rest is handled the traditional way, either CDR's or lots crying.
As a customer you would only have to buy the media instead of the drive (up to $4700)
These drives are somewhat expensive(~$400), but are hot swappable and you can easily take them offsite.
I work in a school, and our main backups are done onto firewire drives that are rotated throughout the district. These drives are then replaced every ~1-2 years.
What, me worry?
And what is the media used in the backup box? A cheap removable IDE drive. Stickem or not, if worst came to worst the media is still far more recoverable (and I've never had to resort to a recovery service, so the cost is obviously low even if it does happen occassionally). I plan on upgrading the backup box to use a cheap IDE RAID controller in a few weeks, mainly so I can expand the storage and stage out the drives. Nothing expensive or fancy.
Problem solved.
The question is not how much data you have, but how much data changes every backup cycle. Good incremental backups can do wonders!
And good lists of files that one doesn't need to back up, like caches.
Read about this years ago: Send a laser transceiver way out in space and have another in earth orbit (always able to see the far satellite). Encode your data in a continuous stream to the far end, and it sends it back, which is then sent out again. The further the distance, the more you can store in the stream. Although you have to wait until it comes around again if you want to access or change it. You can use radio instead of lasers and you might want a bunch of them in case of a failure at either end.
WOW!!! even if that is /. (ie.. imaginary) Math
that is a Beautiful post..
well writen.. kudos to you sir..kudos
The More Knowledge you have the Luckier you Get- J.R. Ewing
Disk drives, even in sealed plastic with moisture absorbing desicant are not designed to be stored for years on the shelf. We see 10% failure rates after 1 year storage within 1 month after initial use.
Disk drives, after use, when stored in a normal office environment pick up moisture (you do know these things have breathing holes) and that is bad for thier health.
Disk drives are designed to be used quickly, to run hot not sit cold.
My guess is a 20+% failure rate after 3 years.
Even tape has to be renewed for really long term storage. Say over 5 years it needs to be read and rewritten and the tape retensioned.
I'd rather change out 50 blank plastic disks once
than endure the need to redundantly and repeatedly
spin up multiple copies of a single 200GB IDE drive.
I do hope you've optimized the *algorithm* first.
Archiving deltas, etc.
Another approach is to just keep the backup always
hot, and geographically distributed, e.g. using
WebRAID.
-I like my women like I like my tea: green-
Also, I find that using the older drives that become to small to be great for storing kernel data or images for emergency boots, etc...
Works for me.
You need to get a bunch of Teletype 43 with Paper Tape option. Just punch and spool.
Or you could get card stock and punch.
Both have shown years that years later you can still read them.
Wow, how appropriate that this came up today. I have a directory called "Utils" that I keep a bunch of dos programs that I've hauled around from computer to computer, job to job, for quite some time. I was going through it today looking for a program and started noticing dates on some of these. I've managed to haul this directory around since 1986!!
:-(
My oldest program is from Sept. 9, 1986 - DIRERASE.EXE
I also have a DOS editor that a buddy of mine wrote and I still use called, simple "E.EXE". 3/21/1987.
I have a dos de-tar (as in the unix tar utility) program from 1988. And, anyone remember this: Spinrite.com. It allows you to adjust the interleave on your hard drive. Well, it doesn't seem to work too well under Windows 2000
Funny that I just noticed all this today.
I have been wondering if a CD/DVD Burner Carosel is available, like the kind you can get fro your stereo that hold 200+ discs. It would be great to load one up with about a Terabyte worth of optical storage. The software that controls it could track the dics, and the space left, and you just keep feeding it new discs. Incremental backups this solutions would probably be pretty good, if it ever exists. I envision something in a 4U rackmount case. It could have a seperate "restore" head, so that it can read and restore data, while performing a backup. Anyone know if anything like this exists? I tried a quick google search, but all I found were audio units, and tape carousels. -ms2k
Just encode your data into a pr0n video and share it on gnutella. That data will never be 'lost' !
Why are you using dashes and dots? Dot=1, no dot=0.
Gamingmuseum.com: Give your 3D accelerator a rest.
Well, I didn't write the comment, but I can be very sure of my tape backups being good. I regularly have to go back and restore backups. I once had a project where I had to restore backups that were 3 years old. I had to merge every single tape, and of those 30 tapes, zero had any read errors to report. There was one that was from 1994, and even that one was good. Your post is implying that tape backups aren't any better than IDE, yet you don't back that up. Sure, you'd notice if an IDE drive failed quickly if it was installed in an operating system. HOWEVER, the original post was about archival mediums. Tapes work much better as archival mediums than HDD's do. I can still listen to my 8-tracks today, but my old 20mb RLL hard drive is toast.
My workstation has local copies of what I'm working on. This, my friends, is the data.
My laptop gets everything that I work on when I go travelling. Thus, it has not-regular backups - a last ditch line of defense.
My server is where the actual 'normal' backups are located. These can change anywhere from hourly to biweekly, depending on how much work I actually do.
For my critical data, everything is encrypted and then placed on a remote server, the then-encrypted filed only readable by my user account.
For uber-critical data, an additional frequent backup is made with cd-rw. Less frequent backups are made with cd-r for longterm storage. But be it r or rw, it's thorougly tested after it's burnt.
CDs are not an ideal storage solution. They are easily damaged and eventually rot. They are, however, far more resilient than any hard drive.
I worked for a tape drive company in the late 1980s. They were bought out by a competitor, mostly just to kill their product line and eliminate a competitor.
About 12 years later, I get email from a guy in South America. He said that he found a decade of his country's seismic data stored on these tapes and they couldn't find drives to read them. He found me from an old Usenet posting.
Luckily, one of the other engineers that used to work for the company still had an old drive working, and he restored the data for them.
Tape is not safe, but nothing else is either.
I have a terabyte plus of data at home. My usual backup method is to have a backup server with enough disk space to mirror all my data and using frequent cron jobs to keep the data on the backup server synced with the machines it backs up. Critical files I also back up online and now and then to external media (dvd-r's kept in a safe at a different location). If you have a safe room in your building put your backup server in there. A room safe from theft, fire, flood, and tornado is an ideal location. Offsite secure locations are a must.
Any storage medium will die eventually. Count on it. I'd suggest keeping a live copy of the data that you can verify is correct using checksums on a backup server and making new backups every so often. If you don't recreate known-good copies of the data on a regular basis eventually that data will just degrade.
At what price learning? At what cost wisdom? The price is a man's peace of mind, and the cost is his life.
The original poster specified dots and dashes.
Well, according to this, their parent company Singapore Technologies filed bankruptcy in late '97, and rather then try reorganizing under Chap. 11, they just liquidated the company..
There's no wrong way, to eat a Rhesus...
However, for long-term archival backups, IDE might not be the best idea. Drives do tend to get corrupted, and if you're not careful about letting them spin down completely each time before you remove them, or they get a little too much shock in transit, you could lose all your data. If you're looking for 1-2 year archival, IDE should probably work, but not much longer than that.
The next option would be tape backups. If you have the money, I would spring for one -- an autoloader if you can -- mainly because it will be more reliable than IDE. Recent experience suggests that tape media (at least the old TRAVAN kind, and some older 8mm DAT) has a shelf life of about 4 years after you write it. I recently tried to recover, for a client, some monthly non-incremental backups for the period 1995-1998 (they are the subject of an IRS audit). The tapes were a mix of older tape (TRAVAN and compat. earlier standards) and DAT media, depending on which of two servers they came from (and how old). Only one (a DAT) of the 24 tapes from 1995 (12 DAT, 12 TRAVAN) gave us 100% of the backed up data. 20 gave us partial, and 3 were completely unusable. The 96 tapes were a little better (12 complete data, 1 unreadable, 11 partial). 97 saw 20 complete, 4 partial; in 1998 all the data was fine. After 1998, the company switched completely to DAT; all the tapes later than '98 worked fine. So DAT isn't the best long-term storage medium. It is also worth noting that the tapes weren't kept in climate-controlled conditions, but instead in an shielded box in a cabinet in a manager's office. The office was air-conditioned, and the temp rarely got above 74, but company is in NYC and it can get pretty humid in the summertime. From talking with some colleagues, don't expect more than 5 years from tape unless you've got it in a climate-controlled environment.
One solution, however, is to backup to tape and then restore and backup to new tape once a year. After their debacle with the IRS, the aforementioned company is going to start doing that in the future.
Another good question is, what are you backing up? If its documents, and you're looking for long-term storage, the best solution is to print the documents out on acid-free paper, put them all in a box, and archive them at a storage facility. Although with 220 gigs of data it sounds like you've got quite a bit more than just documents...
Statistically speaking, there's a 99.998% chance that my IQ is higher than yours. Get over it.
This raises a good question about the digital age: how do we deal with long-term archival? Ancient documents have come down to us on paper, or through papyrus. What civilization 2000 years hence will be able to read off of a CD-ROM? How can we preserve our Rosetta Stones, scraps of papyrus, Dead Sea Scrolls, and such for analysis and discovery by some far-off human culture?
Statistically speaking, there's a 99.998% chance that my IQ is higher than yours. Get over it.
And if you're really clever, you would take advantage of the fact that levels of greyscale are easily discernable. Leave a seperation space on all sides of each dot ( so they're more easily decoded ) to form a grid system. Yes, your storage capacity will drop by a factor of 4, but you can easily encode 8 bits ( a factor of 256 ) into the dot.
Most laserprinters can do 8-bit greyscale.
But for redundancy:
- Make two dots for each 8-bit piece of data, the 8-bits and it's complement. This is only good at error detection, although theoretically you could add error correction at a capacity cost.
- Add 256 calibration dots every few inches to make up for aging of the ink and media. We can assume that the cameras will have much higher resolution than the printer, so they can tell the difference even if the levels have faded together.
You could pack a whole lot of data on paper if you put your mind to it.
Man is the animal that laughs.
And occasionally whores for Karma.
A copy of a copy of a copy of a copy is... the original! Sync it to as many harddrives as you need to have at least one working, and preferably online/availible so you'll know of failures (scripted CRC self-verification for instance). RAID 5 (checksums), RAID 1 (mirroring), multiple backup/offsite machines if you need to. If it's important enough I'd go for security in numbers and replace as they fail. Will it have decades of lifetime? No. But unless you let it decay to the point where all copies are lost, you'll still have your data.
Kjella
Live today, because you never know what tomorrow brings
Point 1.
Make sure you select a very well-made drive, don't cut costs there. Example: I have a 20-year old Mountain HardCard that still works fine. However, I have had cheap 3-year old drives fail.
Bringing up point 2:
If you try it, make sure to use an "exercise" schedule for all the drives in your backup set. For example, once a week for each drive, plug it into a spare box and ensure that it spins up, spins down, and the read/write arm travels its full sweep. Maybe do some read/writes at various places on the platter surfaces, just to be sure.
It works for me, so I hope this helps.
C|N>K
We currently backup 200GB - 300GB of data each night. we recently switched from DLT tapes to a LTO tape loader which has turned out to be very nice... but... as a single backup unit pulling datafrom all our servers it has also approached the 'takes to long' category. So what we have just started to test is a dual setup. We bought a cheap IDE RAID computer which we racked out with 7 120GB IDE drives. We put 2 1 GB copper nics into the computer and also put the loader on the computer. So we are planning on doing a complete Electronic backup over night from all the servers to the cheap IDE RAID drives. (in theory this should be alot quicker) then once that is done our LTO tape backup can kickoff and backup the cheap IDE RAID for us. it can chug away on the backup all day if it needs to not affecting the servers if it takes to long. this also keeps our offsite storage on LTO Tape... has anyone out there tried this type of setup before and if so what kind of problems /results occured with it?
its a Idea we feel is worthwhile to test...
From http://dvbackup.sourceforge.net/
Basic Explanation of the working principle of dvbackup.
As you probably know, current digital camcorders can save approximately 13 GB of data in SP mode on those tiny DV cartridges at a speed of 3.6 MB/second. That's fast. Very fast. It's faster than most DAT streamers which only work at 1 MB/sec or less. We can not use all of the data, but 10 GB should be good enough for everyone. Theoretically you could use LP as well, but then you need some sophisticated error correction mechanism, for instance rsbep by Guido Fiala.
That's nice, but how can we use this to save data on it? And here comes the fun part: If you read the DV documentation carefully, you will notice that the AC DCT coefficients of the video data blocks (8x8 pixels in size) get a fixed amount of space in the DV data stream, but can be terminated earlier with a certain code sequence. So let's have some fun: We terminate the AC coefficients immediately leaving only the DC coefficient for a fancy penguin picture and use the rest for our backup data. Future implementations could easily add a little picture showing the currently written file or something like that.
Not really. If you just keep tape sitting on the shelf, magnetization from one layer will transfer to the next and the tape will become unusable over time. The oxide will also start flaking off after a while and the carrier will become brittle.
While you can be lucky and read tape after a few decades, you can't really rely on it for more than a few years.
I still think somebody will make glass MO archive media, with gold as the reflective surface, but if you're going to use paper, use 2d barcodes... about 1.1K/in^^2, for around 9.5K/side.
Oh, and to be sort of on-topic for the actual story, My friends at Seagate say that modern drives should start up fine after many years proper storage. I still don't trust them (the drives, not the friends).
Dropping a tape can easily destroy it. Or rather your tape drive will destroy it when you try using it and it creases the edge of it because the tape is no longer on it's proper track. Whoops! Bye bye. I hear that dropping a tape into a rain puddle on your way out to the car isn;t too good for it eaither.
There are a million ways to destroy your backup device. I don't see any inherent danger of damaging a drive over a tape.
Yup, IDE is under a buck a gig. This is half the cost of tape, without the overhead of the expensive drive.
The problem with tapes that I've seen is that people get rid of their old systems, AND the drives, but keep their tapes. There's a whole industry in collecting ancient drives to restore data from tapes and strange disks, assuming it's even good.
I decided to switch to hard drives recently, but what do I put them in? A removable enclosure is not a solution to that same problem.
However, an old Pentium is FREE! They've got tons down at the junkyard for a few bucks, same yard that has all the old tape and syquest drives. I've been putting two 80gig drives in the old suckers and putting them in the attic! (yeah yeah fire.) Done five so far, for me and some friends. Well, four. I shoulda had the fifth one done last month.
I figure, stiction won't hit me for at least ten years because I've got some drives that old that work. Also, sticktion comes from heating the lubricant over time, and I'm not running the drive. I don't know if they still have that problem anyway. Ask me in ten or fifteen years.
Also, I've spun up some stuck drives in NeXTs just by tapping them on the side really hard when you power up.
So, I've got backup boxes that are a little large, with two drives each. All they need is power, and you can get into them with ethernet after they boot slackware. Might be tough to find an analog monitor or a serial terminal but I'm not too worried. I'm not worried about the CMOS batteries either, I tested that.
The other question is, how long does eeprom last anyway? It's not permanent, you know! This worries me most. The firmware may rot before the media!
But, $1/gig, no tape drive to break or find, or BUY for thousands, I like this solution alot!
When something better comes along I'll copy it all to that. Or somebody else will, if I'm pushing up digital daisies. But, Digital media just isn't permanent! You have to copy it at least every 10 or 20 years or it's gone....
=Rich
StorageTek, the company that makes those nifty tape-backup robots, has a new product based on IDE RAID called "BladeStore" that should suit your needs.
I use a mirror hard drive to backup my data, which I do on a weekly basis. If either drive goes bad, then hopefully I'll know about it. If the data is more was important like as in your case, then I'd either make my backup machine either redundant raid, or have two seperate machines that do the backing up. The idea is that both dataset won't go bad at the same time. :) And since both are in kind of active use, I'll know when pretty soon one goes bad.
:) For my purposes, having both machines next to each other is pefectly acceptable risk, besides, it's MY data. If the room catches on fire, the data is toast, along with the rest of my belongings. If there's a huge electrical surge, I'm not sure what's gonna happen, both machines are plugged into a UPS. I'll just pray to God in this case.
I used to have all my data on a single unbackuped up 80 gig hd. A while back on slashdot there was an article on incremental backups via rsync. So I looked at it's feasiblity for a bit and I ended up buying two 120 gig hds, one drive resides in my linux workstation/server that I'm using now, it holds the live, in use copy of my data. And the other drive is in a dual boot xp/linux machine that sits right next to it. It's usually in xp when I want to play games, but once I week I fire the machine up to backup my data.
All in all I'm backing up around 90 gig to a 120 gig drive, plus I keep one previous copy of the data on the drive. I also use the same machine and hd to back up a server I have in maryland. I keep 3 incremental copies of this data since it's smaller. I'm counting on most of the files not to change, or else I wouldn't be able to store all of this on a single drive.
While you won't be able to back up 220 gigs of data with a single hard drive, you could easily have a mirror machine with software raid in linux that runs rsync on data.
You also wouldn't want to have the machine sittings right next to each other obviously.
Anyway, here's the scripts that I ended up using, I used scripts from the previously mentioned slashdot article as a guidline.
http://pimpbot.qooqle.com/gid/backup/
pimpbot.sh is the actual script that invokes rsync, then we have the exclude file, plus there's backup.sh which get executed nightly on the server that's being backuped up, it backups up the database, and other random files that's more machine specific.
Mind you, the absolute surefire way to back "something" up is to stuff "it" full of toilet paper the day after stuffing yourself with cabbage rolls. :-)
If you want a medium you can keep on a shelf, properly stored optical disks or optical tapes are probably your best bet. While nobody knows for certain, they are much less likely to show the kinds of degradations you get with purely magnetic media. You can buy special archival CD-Rs (and, presumably, DVD-Rs) that should last for decades.
Magnetic tape is probably the worst long-term backup medium. While you can get lucky and it can last a few decades, data on it often becomes unreadable after a few years through a variety of mechanisms.
Instead of basing backup policies on how long the backup media may last, why not just assume a maximum "shelf" life for the media, and have a policy of backing up the backups at regular intervals.
For example, say an IDE drive may have an expected "shelf" life of 5 years - then, if you need to keep the data indefinately, why not define a policy that, say every two or three years, all the backups are copied to whatever technology is available then.
Not only does this solve the shelf life problem, but , as capacities increase with new techonolgies, so your physical storage requirements reduce as well (plus this kind of policy has built-in verification of backup data)
This is possibly the coolest thing I've seen in a while.
You are one anal guy, buddy.
Try polycarbonate and a industrial high-speed CNC router.Heck if you design the software to convert it right, you might be able to make printing plates.
This needs to be modded up and I don't have points!
7 November 2006: The day Americans realized corruption and incompetence weren't addressing 11 September 2001
Endura paper
When properly stored, this paper will suffer no degradation for 200 years. And that's even with complex color photo development. Something like storing binary data or barcodes should be readable on such paper for much longer than that.
-
I agree with you. The post you were responding to was me trying to show stupid the argument looked to me... thus the quotes. Sure, tape may be far better than IDE for backup purposes, but the idea that you either use tape or you use nothing at all is absurd- if you can't afford tape, IDE is a great backup method, and certainly far better than not backing up at all.
I love going down to the elementary school, watching all the kids jump and shout, but they dont know I'm using blanks.
Doh.....sorry....posting frenzy, didn't see the quotes.
there's always the back of the paper
maybe you could write checksums on the backs
karma: ouch!
Not to be a naysayer...but I will anyways. What happens in 30 years when a massive electromagnetic field wipes out all digital machines (possibly in conjuction with some attempt by humans to wipe out the robots taking over the world...those damn robots!)? By then 15 years of scientific publication may be more or less completely digital, and all gone, gone. Better hope we never lose access to that handy-dandy resource electricity....
only infrmatn esentil to understandn mst b tranmitd
(First off, it's 100Mb/s; and in fact, it's a 125MHz carrier on the cable which a far cry from 100 MILLIhertz.)
All magnetic media has a shelf life. Video tapes, DAT tapes, DLC tapes, floppies, and even hard drives (etc.) will slowly lose their magnetic charge. Hard drives, however, are far more susceptible to degradation. The largest problem with a hard drive on a shelf is variations in temperature and ambient magnetics (the earth's magnetic field fluctuates.) A drive left on a shelf for a few years may be perfectly functional once it's been re-initialized ("low level formated") but the data will be toast.
Get one of those USB 2.0 external drive enclosures with an open front, into which you can insert an IDE drive. (This is basically an external USB 2.0 drive using an IDE drive, but WITHOUT the drive.) Use those removable drive cages that let you remove an IDE drive through the front panel.
Cost of external USB 2.0 case: <$100.
Cost per backup media...
Drive: $80
Removable Cage: $20
Spending $100 per removable backup media might seem a lot, until you compare with high end tape. Plus you don't need a $1000 or more tape drive.
Advantages? Backup is fast. Restore is great -- just mount the drive read only, and interactively hunt for the file you want and drag it back to your drive.
Each backup consists of a bare drive with those removable rails attached to its side.
Those who would give up liberty in exchange for security and DRM should switch to Microsoft Palladium!
Tape is the best way to go. It's a little expensive to set up correctly, but once done, it can be a simple procedure to switch tape sets once a week. Just need to get yourself an autoloader. 220GB @ 20 GB/Tape == 11 tape autoloader. Get the next largest size autoloader, and you'll have plenty room for expansion.
If your the data that can be compressed, then you can half the requirements, but generally speaking it is better to not count on this ability.
have managed the fine art of being both funny and insightful at the same time. My congratulations to you sirs.
.NET? I'll tell you what you can do with your .NET buddy. My OS and *all it's apps* take up less than one percent of of my HD space. HD's are getting bigger in quantum leaps. Spreadsheet programs are getting bigger in tiny, tiny little increments that are slowly getting tinier. I'm not going to run my apps across the net. My data stays home. The only cache of my data is on *my* drive, which I can physically destroy if need be.
I have in my house, not more than a couple of feet from me right now, these things called "shelves." On these shelves are these things called "books." A few of these books were written by people such as Einstein and Eddington and printed going on 100 years ago. In fact, I have a few other books nearly twice that old. The data on all of these books is still completely "readable."
What's more, I have every expectation that they will continue to be readable by my children's children and *their* children's children. Amazing, isn't it? I can pass a magnet over these books, I can douse them with water, I can infest them with mold and mildew and the books themselves may be destroyed as integral objects but the *data* will still be recoverable. They also have no EULA attached, and although I can't copy them and sell the *copies* if they are below a certain age, I can treat the physical object as my property.
What I can't do is burn them. If my house goes up in flames so do my books. They arn't perfect, but they won't just "fade away" or go "poof" in the middle of the night for no apparent reason.
All electronic and magnetic media are inherently highly volatile. That includes tape.
However, I do have, sitting right here on my desk next to my shelves, machines that can copy these books to volatile electronic and magnetic forms and *back again* to print or nonvolatile digital form. In fact, if the book is old enough, I often find that the work of converting them to magnetic and electronic media has already been done for me.( if we all did just ONE unique book that would be millions of books commonly availble in electronic form. Think about. Just one. Maybe even a short one)
In an odd twist of fate the old fiction of the Xerox commercial comes true. The best "backup device" known to man is a monk with a copy machine.
I once got modded as flamebait ( and I honestly don't know why, troll maybe, but not flamebait) for pointing out that the best PDA I knew of, and the one I personally used, was a pocket sized spiral bound notebook I got from CVS for $.69. It's cheap, nonvolatile, easily replacable anywhere in the world, has an infinite "battery" life, is fully pen compliant, can go through a metal detector without a qualm, can transfer its data to *any* computer in any data format and I've even run it through the wash and still recovered it's data. Hell, I can throw the thing against the wall as hard as I can as often as I want and the damned thing doesn't even get *scrathed* let alone lose data. I love the thing.
Pen and paper is still the optimum solution to many problems, except perhaps having a fun new toy to play with.
On my shelves, since I once worked a few days with an archeological doctoral student in Mexico, are a few bits of stone and clay with hyrogliphics on them several hundred years old. These bits of meaningful earth have been buried, rained on, trodden on, smashed, earthquaked dozens of times, burned and god knows what else, and their data is still recoverable.
Let's see your tape drive do that. I'm serious. There's a lesson to learned here.
Here's the best way to permenantly backup long term digital data. Take a giant titanium platter 5mm thick and punch 5mm holes in it for each "on" bit. On the first "track" of the platter make sure to make a code key easily decipherable by anybody with any sort of mathmatical knowledge.
There ya go. A backup that will last longer than the pyramids stand. You could even shoot it off into space, unprotected, with reasonable assurance that it would survive the trip until "someone" picked it up.
Ok, so that's not very reasonable for the average Joe who just wants to protect their mp3 collection. What has any of this got to do with you?
Plenty actually. In the first place, don't store anything in a computer format that can be more effectively stored in nonelectronic form. Except perhaps as a backup of *that* media. Books are a prime example. If your computer goes down and you need to refer to a manual to get it back up again that manual is worthless if *it's on the computer.* ( And did I mention that books have no EULA? That's a *feature*)
That isn't to say that you shouldn't have ebooks and CD based manuals, but that those are convieniences and backups. The *books* themselves are the primary data.
Don't rely on your PDA to keep critical phone numbers and such. You WILL drop it. It WILL get stolen, or have its batteries run out, or go through a *hidden* metal detector.
*Write them down* and carry the written copy. If anything happens to your PDA you're still good to go, and restore the data to your PDA/computer when time permits, like when you're just bouncing around your Days Inn room wondering what the hell there is to do in this God forsaken town anyway.
What to do about all that data that is inherently digital, or at least more viable in digital form?
Did you ever hear the phrase " The net is the computer"? Just for this once take it seriously. I've been a strong opponent of storing data on the net. Drives are big. That's why we're having this discussion in the first place, isn't it? They hold a LOT of data. Why put your sensitive personal data, data that, perhaps, could someday cause you considerable grief if it fell into the "wrong" hands ( and this doens't *necessarily* mean the FBI. Ever get divorced? Trust me, you don't want some things to be recoverable from an outside party) on the net? It makes no sense. The net is a *party line* capable of permenantly recording everything that goes across it. Certain documents belong ONLY on paper, or your own *secure* machine, or in your lawyers safe. Period. Please, take this advice very, VERY seriously. You may well have reason to thank me for it some day no matter how doofey you think it is now.
Do likewise, and it wouldn't hurt to encryt your whole bloody drive either. Just in case.
But all of that being said, do I *really* need a local backup of my files downloaded from the Gutenberg Project? They're already out there on the net. They're mirrored all the hell over the place. Someone else is already taking the trouble to swap drives, swap tapes and transfer the files to up to date media. It only takes a few minutes to download the whole bloody lot. If you lose your net connection, even permenantly, you know 20 other people who can grab it for you, and a half dozen locations where you can use a public computer to get the stuff. Even if all the mirrors to down at once, and *stay* down, you can always go on usenet and ask around. You'll find what you're looking for.
For this kind of noncritical data plastered widely across the net that you can, in all probability, recover faster from the net than from local backup ( remember, it takes *time* to find physical media, load it up, find the files you're looking for, and transfer them), simply *don't bother.*
It isn't worth it. Even if it takes slightly *longer* to do it this way than from local backup. The time you save in making the backups and keeping them up to date more than makes up for any trouble.
And remember, this sort of full backup is *disaster* recovery. Like if your house burns down/your entire system gets infected with a virus/your HD crashes/your dog eats your homework. In this sort of situation the small amount of time it takes to recover from the net is peanuts compared to the other things you're going waste time and money on. So don't sweat it.
If you take care of your system properly most of these things aren't even likely to occur,except the new HD that goes down after only one week. For the most part a *full* backup is mental insurance that your data can be recovered * somehow.* If it's on the net, and it's something like the Gutenberg Project, it's going to be there. Relax.
So, what about your mp3 collection? That took you years to build, and would take you weeks to rebuild, and maybe the RIAA will win out and the stuff will become unavailable. Well, for these you want to back them up to CD anyway, so you can play them in alternative devices, make copies for your friends, etc., right? Well there ya go. Don't bother putting this stuff on tape or anything. You've got it already. Make two copies, give one to a friend. That friend now has copies to play, and you've got off site backup. Have the friend do the same thing with HIS collection and you're BOTH protected. Well looky here, in certain cases and situations socialism DOES work ( much to the RIAA's distress).
The same goes for image files. It took you years to build up the pr0n collection. Some of it from pay sources that have gone *poof*, or that the FBI closed down. All of it from all over the place in any case. It would take you weeks to rebuild it, if you could do it at all. So sure, back this stuff local, but again, do it to do some form of digital disc ( CD or DVD). Find a friend with the same taste in pr0n and do the same as you did with the mp3's.
Repeat if you have video material.
Now take all of your system discs. OS stuff, Quake, Photoshop, etc., and copy them. Store the *original* offsite, maybe with a friend ( you do *have* friends don't you? I forgot to ask. A lawyer's safe or safe deposit box can substitute) or family member. Don't forget to write any applicable install codes *directly on* the copy.
Now if the worst comes to worst you can simply reinstall everything. You might even find that half of this shit you never even *bother* reinstalling. Go figure.
So now what are you left with? Surprisingly little in all likelihood. You're going to want to back up all of your system settings. Sure, it won't take you that long to reinstall the system itself, but the settings took you ages to do and you don't even remember them all. Back 'em up. Obvious data like your resume and thesis, software projects, financial data, Quake player data, etc..
Ok, what's left? For the average home user, and even the not so average, you may be surprised to discover you have now * done it all.*
Wait a minute, what happened to the traditional full backup?
Well, the thing you have to remember is that a *home* computer and *business* computer are inherently different when it comes to backup. When a business computer goes down it has to go back up NOW! And it has to go back up *exactly* as it was when it went down. This need makes all of those tapes backing up everying in full, right down to the most common thing they could just grab from the web, absolutely and completely necessary.
A *home* computer isn't like that. Your house has just burned down or your HD has crashed and you need to be sure you can recover everything, or even only *most* of it, within a reasonable time frame at minimal trouble and *expense.* Tapes, tape drives, and full backups all the time are, in and of themselves, trouble and expense. All against a future disaster that might well never come. You don't need to do it NOW. You just need to be able to rest assured that you *can* do it.
To hell with the full backup. Reinstall your OS and apps and restore just your settings. Sure, it takes a little longer, but that time goes against all that tape swapping and storing. You might actually come out ahead.
Now just download the stuff you can as time permits or as you need them. Get your stuff back from your friend if your local backups went up in smoke with your house ( you can live without Houses of the Holy for two days, honest, and even the jpeg of Natalie Portman's head Photoshopped onto Pamela Anderson's naked body) and you're back in business.
Now you're just left with incremental backups. For some people this might have to be done every day. For some they can legitimately wonder if every month is really just being paranoid.
Ok, so you really, really, REALLY just don't feel comfortable without a full backup? Maybe you're just the sort who really thinks he *can't* live without Houses of the Holy for even one extra minute he doesn't have to?
Ok, NOW the idea of mirroring drives for a backup makes sense because you aren't relying on it as your *primary* means of backup. You're only going to use it if your house burns down *tomorrow*, not in five years and in such a case it could save you some time. Hey, if it rows your boat, go for it Sparky.
I might even end up doing it that way myself.
KFG
I think this is a bit of a ridiculuous suggestion.
/. last week. (Though the
Why not just save up and buy a DVD burner? Spend
a little time to sort out and 'archive' some of
your data so that you can back most of it up once,
then only back up the parts of it that you 'take out' of the archive.
Yeah, I realise at 4G or so (?) it is going to take a few DVDs and a bit of time organising that 220G, but probably less than the time you would
waste getting your hard disk solution going.
As for stiction, in the 5 or 10 years you mention that taking, there will be larger optical formats for data storage & backup, ie like DVD burners. There was an article on one such format that can store 120G or so on
capacity for a burning media will probly be less). In a few years or so, buy one of them and
transfer your backups to that format.
Alternatively, if you can afford now to buy 160G drives to use as backups, then in 5 or 10 years
when you might need to worry about 'stiction',
similar capacities will be much cheaper and you will be able to buy shitloads of extra disks to
transfer your backups to.
still, if you have 120GB now, in 5 or 10 years
you could have a few TB...
Several years ago I heard about some FMD
technology that supposedly held quite a lot of promise for huge capacities... no actual product seems to be materialising though..
here's a few lines quoted from some article written in feb 2000:
Hello FMD-ROM -- Bye-Bye DVD?
By Andy Patrizio, Byte.com
Feb 21, 2000
"Constellation 3D is in the final development stages of its FMD-ROM drive. "
"The capacity potential for the first-generation of FMD-ROM is up to 140 GBs of storage, almost 15 times the capacity of a dual-layer DVD-ROM disc."
Man, I wouldn't worry about your ~200GB of porn... those women won't look good in 5 years.
And to top it all off, I back it all up to a DDS-4 DAT autochanger. Yes, those six tapes will only hold 120gb, but the amount of important data on my disk drive is far less than 120gb (it is actually less than 20gb, including the original 44.1khz .wav recordings of all my original songs, and fits onto one tape easily).
Do you *REALLY* need a backup of your .mp3 collection?! Probably not. Do you *REALLY* need a backup of all those ISO CDROM images that you downloaded for fifty versions of Linux and a half dozen versions of FreeBSD? Probably not. But that's the sorts of things that are taking up 80gb plus on my hard drives -- i.e., utterly disposable cruft. Which is true for most personal computers.
Send mail here if you want to reach me.
For long term archiving, you should use a Storage Area Network. When one of the drives dies, take it out and put in a new one, and the SAN will rebuild the data (Assuming it was configured with a RAID1 or RAID5, or is a fully duplexed SAN)
So one way would be to both preserve a general specification of how to read the data, and then the data itself. So not only would you need a method of encoding the song onto paper, but you'd need to include the details of an algorithm - simple enough that people whose language may be very different from ours - can recreate it using their machines of the time. And then they can feed the data into it, and replay the music/video/whatever as we intended it to be seen.
-
That much (little) paper... is actually lighter than a terrabyte of digital storage!
I doubt it, 200GB hard disks are out now, that's about 5 hard disks per TB. 320GB will be out in a couple months, that's a little more than 3 drives per TB. In another year we will have 500GB+ drives most likely. All ATA of course, I doubt SCSI will keep up with the size increases, it's been lagging further and further behind over the last year. (Those huge 181GB SCSI drives don't count, they don't fit in most normal drive bays).
I've had enough abrasive sigs. Kittens are cute and fuzzy.
Hooked it up to what? I didn't think modern motherboards supported MFM drives...
Solution 1: RAID5 IDE cards are available. Buy large disks, and budget for replacing the RAID unit every two years, and for spares. Additionally, if you have the bandwidth, backup the system to an identical setup at another office location or a 3rd party. After an initial large backup, incremental backups should be manageable overnight for small to medium-sized companies.
Solution 2: Contract a 3rd party to do the job. If they lose your data, sue, win, retire, and stop worrying about it.
I am forever amazed by the market segmentation resulting in product offering gentrification, being taken to indicate that the products (which are identical apart from the electronics) are in fact different.
No 10k RPM drives on IDE for a real simple reason, and it has nothing whatsoever to do with technology. It has everything to do with marketing. Few people will see benefit in going to 10k drives, and the IDE folk are not likely to pay a premium*. So 7200 RPM max for them. The folks that will have this mistaken delusion that SCSI >> IDE (its just electronics in most cases!!) are more than willing to pay the premium for the knowledge that they are "getting a better drive".
Economics of manufacture force the drives to be made the same. Costs more for seperate bare drive processes. Gives the company a way to make money off of drives on all sides of he market.
* The recent trend towards adding larger caches on IDE drives shows that people are willing to spend more on the drives, so there are now market segmentation forces going on in the IDE market as well. You see performance drives (at a premium), and slower drives real cheap. Price difference isn't all that much (not like IDE-> SCSI), so people voluntarily pay more for these drives (for marginal extra performance). The drive company just raised prices on the lower end and no one noticed.
And of course we still have the fools who claim one is better than the other. Isn't segmentation great? Gets otherwise intelligent and savvy people to shed their desire to seek accurate and complete information in favor of convienent stories they can tell that make them feel better telling them.
Gotta love good marketing.
I often work on firmware build systems that are 5 to 10+ years old. The answer is SCSI hard drives with SCSI backup hard drives. AND keep them running. No mechanical device performs well when left motionless for an extended period of time.
Currently I'm working on a ten year old system that has the main SCSI drive and a mirror drive encase the main unit fails. Then a nightly backup is done of the changed files to a main server.
I hear they're cheap now
Actually, Moses DID crash the original stone with the ten commandments, when he saw the people dancing around the golden calf and all that.
Of course those were just 10 commandments, which amount to less than 1 kB of text (it was even less in Hebrew writing), so writing them again was straightforward.
What wasn't straightforward was to keep the same word meaning and the separation between each commandment. However, the spirit of the commandments is still there.
Acknowledging the Lord as our God and no worshipping other gods but Him, no taking His name in vain, keep a day of the week holy, honor your father and mother, no murdering, no committing adultery, no stealing, no false witnessing, no covetting anything that is your neighbor's. Yeah, it's all there.
Too bad the knowledge of our civilization is not as straightforward and lasting in the time.
Luis
--- Sueños del Sur - a webcomic about four young siblings
Interesing project at SourceForge... DVbackup. Might be worth checking out... especially if you already have a DV Camera.
.:diatonic:.
Are the "*"s round the "is" to highlight the fact that the grammer is wrong? You should have said "Tapes are the right media for long term backup" or "Tape is the right medium for long term backup".
Use carbon-based ink on leather scrolls, put them in pottery jars, and store the jars in an arid desert cave. The Dead Sea Scrolls were preserved this way for almost 2000 years.
Perhaps the ideal solution isn't here yet. I've read that optical discs on the order 100GB are just about a few years away. Below is my list of the shortest to longest shelf lives (estimated) and their problem(s) 1. IDE Drive (2-4 years) - chemically,magnetically,mechanically dependent - lubricant gets old - may have degradation - cheap price and cheap drives 2. SCSI Drive (5-10 years) - Chemically,magnetically,mechanically dependent - same as above, but better stuff (Seagate cheetah 1.2 million MTBF) - expensive for large capacities 3. Tape Media (15-20 years, in good condition) - magnetically and mechanically dependent - the tape falls victim to temperature but the vxa drive combats that the packet technology - drives are expensive and tapes are somewhat affordable 4. Solid state (50? years) - magnetically (and electrically) dependent - The electrons can move over time and cause data degredation - expensive and only in small capacities 5. Optical media - optically dependant (thermal negligible) - comes in small capacities. The best of this list is to put them on cd's and put them in a box. we just have to wait to larger capacities to come around. Disk drives and solid state devices are meant for online storage. Tapes are meant for backup and does somewhat ok (I think that the F-14 flight recorders use DLT). I right now, I'll probably use tape (vxa) for backup until the large capacity optical drives come out with smaller beams and such.
People are all paranoid of loosing Email and the like now, but in 1 year they don't care about it any more. In 2 years, it's just wasted space. In 10 years, they won't even know who or what they were talking about..
I used to keep all my email (except junk mail). Now I have a loose 250MB limit. When I go over it, I start randomly hitting delete.
Funny thing is, there's not much worth keeping in the NEW stuff.
Well, by now there is probably no way that anybody is bored enough to read this far down, but I'll add my 2 cents on tapes.
What is it that people are doing that degrade tapes media so badly that they are unreadable after a year? I've restored from tape far older then that, and I've got old reel to reel audio tapes from the 60's that are in perfect condition, and I've not done anything special to take care of them. Has the quality of tape degraded since the 60's?
How much of this data do we actually NEED to back up.
.ini / .cfg / save files/ registry entries, mail address books - the WHOLE LOT that's important to a 600mb cd - (I kid you not) the rest is all files from the installer from the game / app which are on the original CD in the first place!!
I've got a tonne of cd's I've "backed up" over the years which I feel better knowing I have a copy of, but in reality I never use the data or it's not super important.
I mean sure I've got 24gig of porn, 4gb of movies of great car chases / crashes - and a shitload of mp3's I don't listen to also, but do I really *NEED* to back it all up to some kind of cd / dvd / whatever when in reality, I don't REALLY need it.
I think stuff like mp3's / movies would be nice to dump on say an 80 - > 320gb drive in one of those loungeroom based Tivo (or similar) units to play back funny / interesting movies to friends but really never to bother dumping on some kind of storage medium
Also interestingly (yes I'm a Windows user) I've found of the data on my primary drive, (approx 25gb of apps / games / whatever) I can backup all the relevant
Something to think about.
Divide all your data into 10K chunks. Create a username like mybackup-20021210 that has the backup date. Paste each 10K chunk into a separate journal of its own. To backup, read the journals sequentially.
Laser printers do gray scale by dithering, you lose resolution. Good idea though. Better storage medium would be black/white photographic film like microfiche.
-Yarn - Rio Karma: Excellent
If you're looking at long term storage, tread carefully. The MTBF (mean time between failures) that you will see for a drive of 100K hours is during the NORMAL lifetime of the drive, typically 5 years.
Rotating or not, once that life span is up, all bets are off...
While I don't have any size or mass estimates, I think we could compress it quite abit farther by using one of the newer versions of the bar code. I think they call it '3-D' and it looks like a scatter of dots all over the place within a given recatangle. You can put a lot of data in a small place with this.
--Begin #! /bin/sh
mv $1 /dev/null
End--
Benefits:
1. No worrying about media
2. Saves space
Drawbacks:
1. May be difficult to get your data back
2. No GUI (yet)
I don't know about you, but my ASUS motherboard still has 2 ISA slots...
Anyway, I would guess the same goes for my old IBM XT/286 locked away in the basement, which has a 20MB harddrive that worked fine when I last checked it 5 years ago.
Problem is, things are not made to last today, like they were back then.
-H
Of any kind will degrade fast, well not so very fast, nut I have pristine floppies(remember those) from 12 years ago, kept apart in wooden boxes with 4 inches apart from anything, and they have gone to shit in 12 years. 720K floppies of some of the best DOA stuff on the planet. All are trash. The Earth's magnetic field killed them. Electromagnetic fields helped. OPtical is the only way to go, IF you never use it except in extreme emergincies. Optical does degrade, but much slower. Go for the optical and find a wat yo make 30 copies and disperse them intelligently. Who knows what stuff effects optical, other than optical. Be paranoid, but remember, the IRS only needs 7 years of data for an individual, maybe more for a corporation. Blow me!
now lets figure in the groth rate of trees and
figure out how often we can back up that terrabyte
of data assuming we replant those nine acres.
The point was that at 6gb a pound, a paper solution would outlast a magnetic media solution because it was optically readable and be able to survive much harder conditions.
GPL Deconstructed
Vmyths ought to steer this OP right.
Mr. Gibson does about as much good for computers, as Symantec Antivirus does for JokeADay.com.
Saskboy's blog is good. 9 out of 10 dentists agree.
however, i use an 80 gb firewire drive for medium-long term storage. it was cheap ($130) easy to start with, and only turned on once a month for backup, and to keep it warm.
keeps the drive in ok shape, without exposing it to harm. works for me . . . .
"You never want a serious crisis to go to waste." - Rahm Emanuel
The funny moderates you
[sa]
Ha, that would be cool. If they'd store something cool or useful on the jewelry. The Bible example leaves a bit to be desired.
DLT drives are available with stackers/autoloaders (sometimes they call 'm "tape library"). Plop in a set of tapes, and you're set for a week (or more, if you use incremental backups). Also, your puny 220 GB will take maybe 2 hours on a fast DLT drive, much less if you use incremental backups. There's another tape format called LTO that has similar specs.
I wouldn't use hard drives for back up in any scenario. They are really unreliable. I'm sure we all have a few drives that *just died* lying around. That's why you need something that has the storage mechanism separate from the storage medium.
So, use tape drives for short term. Don't worry about massive data as it is likely to not be changed the next backup around. There are many apps out there that will backup only what actually changed. After the first backup, you can just leave the tape in the drive and automate it.
Then every year copy your tapes to DVD's, as at least these will last few years and put them in a place outside your house, like a safe box or something. Burning is a hassle, especially with lots of data, but once a year is not that bad.
This will only last you a little longer though. So you will need to copy those DVD's onto other media/formats.
In 10 years, chances are that it will be a challenge finding a device capable of reading your backup format/media. Even DVD's will not last that long. They will be replaced by drives 1/10th the size and 1000 the capacity. Not to mention the file systems will likely go through a big change as well.
USB 2.0 has a higher speed than FireWire.
(1) You need a true hot-pluggable interface to ensure there is no hardware damage or data corruption when you remove the drives. IDE would be a poor choice. SCSI (in commercial RAID systems) usually has expensive, proprietary hot-plug solutions. Flame me if you will, but on a limited budget I'd go with USB and live with the bandwidth limitations.
(2) Modern ECC systems built into hard drives (all types) do a great job of handling your drive's bit error rate--until it deteriorates below its functional limit and suddenly becomes junk. The threshold between working and broken is very steep, so you can have a false sense of confidence in a backup drive that "checks out 100%" but really is on the verge of major data loss. The same thing applies to tapes, too (modern ones with advanced ECC, at any rate).
(3) ESD straps and ESD baggies: use them. Religously! ESD damage is cumulative, so drives handled frequently are at very high risk of ESD-induced failure. Yes, a data salvage specialist can remove remove your platters from a damaged drive and may be able to retrieve the data from them. But that service is expensive, and the down time will be longer.
(4) Any time a drive is "on line", it is at risk of data loss: software errors, brainware errors, power surges, you name it. Get the backup drive off-line and off-site as soon as the backup is verified.
(5) You DO verify all the backups match the original before idsmounting them, right? At least that they're readable?
Nice analysis, by the way.
>> Assuming double the standard density (160 chars per line instead of 80, 132 lines per page instead of 66), which actually works out to quad density, you get 160x132=20120, say 20k per page >>
I remembered reading something about this; dataglyphs use slashes to encode data to present a grey background to a page, if you don't look too closely. I quote: "At 600dpi, DataGlyphs can offer up to 1KB per square inch of data." If you want, you can encode less densely, and make it faxable. This strikes me as a cool way to avoid hardware obsolescence, which I worry about.
Just because we have the means to digitize every bit of information that comes our way doesn't mean that information is inherently worth anything.
I'm not talking about the libraries of congress. I'm talking about those photos you took last summer that look like shit, but you figured you'd keep 'em cause, hey, it's digital, no penalty. The Star Wars script. Linux source code. If it's your own stuff, fine. If you use it, fine. But if you're just a digital packrat, consider chucking it all.
Learn a poem and recite it. That will last for millennia.
There is no difference between a text file and a video file, except in how the voltage pulses are acted upon. Thus, an explanation must be given as to how to properly interpret the strings of binary digits. Suppose you compress the data, why would they be using the same compression algorithm, or even know how to detect it? If you want someone to be able to read it, wouldn't you want to make it as easy as possible for them?
I was reading an anecdote the other day about early NASA data tapes. The machines to read them have long since disappeared, so in order to get the data off they pulled some 80 year old engineers who built the things out of retirement to interpret the data.
-
Still having trouble understanding the Mayan's last backup. Man what were they trying to say about 2012?
-- taking over the world, we are.
This reminds me of a guy that worked for a company the specialized in data recovery. The company had a contract with a large goverment research facility in the area that had many drives that would fail due to the bearings going out. The solution to this of course was to put a couple drops of wd-40 into the drive, recover the data, then follow up with a couple drops of superglue so as to make the client thought you really were something. They also liked the facilities method of making sure the old drives didn't become a security hazard, they had a large machine that would turn the drives into a metalic confetti. offtopic but still one of those things that popped into my head when i read this
Timber!
We do it at the National Archives everyday.
I remembered that The Long Now Foundation had done some work on problems like this. They have an interesting solution.
Storing your data in a format you can scan back in, yet it lasts for hundreds of years seems like a neat idea.
hi Pohl!
CD-Rs are the way to go0 10719S0 003
Check out this article for more info:
http://www.informationweek.com/story/IWK20
Its cheaper nowadays to buy 10 IDE drives than a single DLT drive.
A typical letter-size paper is 8 x 11 inch. So it can contain only 88 KB per page. A bit of an improvement to the 20 KB of the original scheme but still not high enough for Terabyte of data. In addition, this data pattern probably requires compatible hardware/software to read it back. Considering it is non-open source, how do you know it is still available 10 years from now?
The real problem isn't usually hardware rot - it's data format rot and interface standards rot. IDE is a bit tired, but it'll be supportable much longer than N-1 of the current DVD-R standards, so you probably want both. In my attic, I've got a probably-good 9-track backup tape of some of my early work, and some newer probably-good Sun cartridge tapes, and I might still have a Sun tape reader, if I can find a copy of SunOS 3.5 for my Sun-2 diskless workstation to bootstrap something to read it with.... Not a high priority - anything useful I did back in those days is probably on a Usenet archive somewhere.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
It only improves the density by four times, but it might be capable of doing even better with, say, 1200 dpi. Or maybe you'd need special, heavier paper. It's obvious they've thought about it a bit; it copes with faxing well, for example.
The reason I mentioned it is because it only needs a scanner. The longevity of the software right now is admittedly questionable.
is to chant "It'll never work. It'll never work" while rubbing blue mud solemnly in your navel just before inserting the bounced tape into the drive.
You are an insane person..
If you are thinking long term, do not be cheap.
... and so forth)
Be lavish in the number of drives you purchase
(whatever kind that may be), and also set aside
a few complete systems (hardware and software)
that can read/write the media.
How many of you have tapes or floppies from
yer college days that are worthless because
you don't have the 1/2 in tape drive to put
a tape on? (or the floppy drive, or the DLT,
or the Exabyte device, or the
Hey - don't forget the source for the software
to read/write the data!! Lots of people have
tapes full of worthless data because they can't
decode the bitstream, even if they can read
the media just fine.
Do you suppose this worked because it was an old, low-density drive?
Do you honnestly want me to believe that you have 220GB of "valuable" data?
Right!
Remove the MP3 collection, sure it's time consuming to create, but you can archive it once to CD, the songs don't change.
Remove your movie collection (see above)
Remove your software (It's less trouble to reinstall them, usually. Exceptions are either custom (make a CD with a doc on how to install) or pirated (You could make a CD. Is it worth it?)
What do you have left? Probably your e-mails, bookmarks and the 16 odd documents from the likes of word and excel, most of which are no longer relevant. Possibly your tax/accounting data. Like 500k of it.
Hmm, looks like that fits on a 2GB DDS-1 tape.
So yes, use some kind of IDE RAID system for your backups, and be sure to save some backup computers and operating systems every year or two to read the old disks.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
I'm doing what you're talking about. No, it's not "perfect", but my budget comes from taxpayers, so I go for cheap yet sufficient. I use 3ware controller to create a RAID-5 array. I backup everything on my network to it. 120gb * 8 @ raid-5=840gb (approx). Then I have a 2nd box of the same that mirrors this data. I have all changed data for the past 60 days plus a full monthly backup. At the end of the year a full month is about 200gb and fits on 2 disks. I've noticed that thus far (2 yrs now) that the IDE drive capacities keep up with my requires of space required. So next year, I copy ALL of my previous archives onto the new disk. This accomplishes a few things .NET, I just copy all the data to that drive next year!
1. My data is good and fresh. I know the data is good as it's all copied onto a brand-new disk. Unlike tape, I trust a copy from disk to disk (/v)
2. It is all in one place. next year, I'm sure I'll be able to buy 200-300gb IDE for the price of my 120gb this year, or my 40gb last year...
3. It's accessible. I have data that originally came from NW 3.11, 3.12, 4.11, and NT 3.51 servers all on a folder on an w2k Drive that works on ANY w2k ws. If I migrate to MS
4. As mentioned by others, it's cheap, fast, and accesible. And after working with computers for 20 yrs I WOULD trust a hard drive (that is mirrored and updated) more then tape drives. I can read IDe from a few years ago (although I don't need to) much easier then trying to find the backup program, tapes, and drive parts of a tape drive from the same timeframe. So when we move to serial ATA, I just copy the data to a serial ATA drive. The next technology? Just do it again. And if my drive does crash? So what, I've got a 2nd verified copy off-site.
Works great and is cheap. Easy, too.
Presumably, one way would be mathematics. 2+2=4 everywhere in the universe. Therefore, a system based on some fundamental math properties would be useful.
The book 'Contact' briefly dived into this, as the contacters sent diagrams for the machine, along with a primer on how to make sense of them by using various universal things to describe the concepts used in the schematics.
For example, to describe what the symbol for pi is, you might try drawing a circle, a line halfway across the circle and a couple of dots to describe "2piR", which is the universal equation for circumference.
-
Just don't connect the data cable to it. I don't know if the lack of head movement is a problem. I think they would move a little every time the power was cycled on the machine.
mhack
Building a better ribosome since 1997
Is anyone else reminded of IBM's punch cards with this talk of printing for archival purposes?
Who says you can't use double byte charactors anyway?
Just doubled the storage capacity since one charactor = 2 bytes.
Your data will not only not be useful to you, it'll probably only be useful to someone suing you.
In 100 years, you'll be dead. Nobody living at the time will care about you or your precious data. Get over it.
If you're worried about losing your MP3 collection or your Porno videos (what else do you have taking up 200+ GB?), share them with friends. You won't all lose your IDE drives at the same time and it won't be very difficult to get copies from your friends.
My company adds another 2tb/month, every month, like clock work and the rate is increasing.
Currently up to 55 TB and must scale to infinity.
And I know better than to think I'm big time. There are much much bigger sites than mine.
Just divide your data into "static" data and "dynamic" data. Backup your static and forget about it. Backup your dynamic using ghost or something.
Relax, it's a Computer.
Records of the computer age will be a blank in the future.
Better now we print them all out on vellum, roll them up and put them in a wooden box. Sure as eggs are eggs, our computers will lose them all eventually.
The reliable, safe computer is a myth. We have tampered with nature and gone too far. If you lose a piece of paper in the filing cabinet, maybe it's fallen out of its folder and is lying in the bottom of the drawer. You can find it. If you lose a document in your computer, forget it. Imagine that old filing cabinet had a button on the front, one press of it and the whole contents take a one way trip to oblivion. Would that button be accidentally pressed one day? You can bet your back-up it will. If there were two buttons on the cabinet which had to be pressed in a certain order, would it still happen eventually? Goodbye.
The computer experts, who are professionals in the arcane art, say any problems are all the fault of dopey users. A snooker professional would also think we should be able to put together a break of at least fourteen, but we can't. The cry of the PC Pro is, 'A computer can't make mistakes.' I wish to advise them that computers are temperamental, infuriating, unpredictable and as difficult to use as a one-wheeled bike. Maybe the top five per-cent of computer users can cosset their particular machine in such a way as not to upset it and give it a grudge against humanity. Normal people can't. The fact is that a lot of the time spent using a computer must be devoted to it's wellbeing. E.g defragmenting the hard drive! If I had to defragment my grill as often I would give up toast.
Hard drive! We needed that. A disc of magnetism of unimagineable capacity, going at incredible speeds with a pick-up arm floating a millionth of a hair's breadth from the surface. I call that soft, an anvils hard. And this robust piece of indestructablity is the heart of it all. Everything saved is gone if it breaks. Have two perhaps? Trust it all to two butterflies wings? You can worry about both then.
I am a law-abiding citizen until I use the computer. Then I commit more illegal acts than the Mafia. I wish it told me 'Not a good idea.' Or 'Whoops.' Instead of having the quickest, most one sided trial in history and giving the inevitable sentence like Judge Jeffries. And what's with this 'fatal error'? Nobody's died yet.
When all the butterflies wings won't do anything, for reasons known to no-one, a screen just stays, and all mouse wheeling and key jabbing does nothing. We have to become a shamefaced felon, look over our shoulders, and when the coast is clear, switch it off in a way of which it does not approve.
If it never works again we know it was our fault and it serves us right.
Regards to all
Peter Perkins
www.comedygenius.fsnet.co.uk
the real key is to not get attached to your data in the first place. everything decays. everything will be lost. you will die someday. let go.
Anyone have any experience with building a Firewire RAID box with the $90 2 drive enclosures available at Fry's (and elsewhere)? I'm curious to hear reports on that.
J
Check out this Patent
Basically it's 450kB per duplex page (assuming a generous 0.5" margin). So a TB costs 2330 pages, not bad for 5 reams of paper costing what a few $ each.
oh, wait.
nevermind.
there can be hours between the so and the what of the so.
And you ay be able to reclaim obsolete data storage mediums for decades to come!
.
(David Bowman, EVA near HUGE Monolithic Win-PC in orbit around Jupiter) "My God - its full of Malware!"
Only theoretically...in practice, my understanding is that FireWire is still faster (delivers something closer to its maximum speed) and still doesn't bog down your computer as much. (Think about it...Apple designed FireWire to work properly, while Intel designed USB to drive demand for faster and faster processors.)
20 January 2017: the End of an Error.
www.paperdisk.com claims that they can get either 660K or 1MB depending on resolution on a sheet of paper. How long a piece of paper will last when encoded with this density is unknown, but with good paper I'd bet it's a hell of a lot longer than any disk. Furthermore, even at that density, there's a huge ammount of physical redundancy in the data storage. If the paper gets to be fifty years old or so, I would imagine that the technology would be available to cheaply scan at ultra-high resolution to compensate for any degradation.
A little off topic but I have an old 40Mb Miniscribe hardrive which was removed from an IBM XT around 1987, been in storage ever since. A few weeks ago I got curious and decided to plug it in to a spare box I had lying around to see what would happen. Fairly easy since the original MFM controller and cables had been kept with the drive.
I was amazed to find that not only was all the data on the drive intact but the thing booted up straight into MSDOS 3.1 with no problems.
MSDOS 3.1 and Word for DOS 5.0 really scream on a PIII 800 !
Anyway, curiosity satisfied I put the drive back in storage. I figure I'll try it again in another ten years or so... if I can still find a motherboard with ISA slots
I've never had a hard drive fail that was in storage, not counting the one that rattled around on the dashboard of my car for 6 months. I still dont think I'd recommend using them for long-term archival media though
I'm still running the first HD I bought in 1996 (well actually, it is not the original since that one had bad blocks when I bought it and I got it replaced like a week after the purchase). And also all other HDs I've bought over the time is running fine in any of my computers.
Maybe it is how you handle the HD that is important? I have them screwed in my box from day 1 and most of the time with space beteween them so they won't get unnececary hot. And I haven't had any problem with them ever except for the first HD with bad blocks.
On the other hand, HD that have been lying around for a few years are not very likely to function, I got several disks from like 3-4 years ago recently. Half of them wouldn't start at all and the rest is having a high pitch noise coming out of them... The data on the disk that started were intact as far as I could see.
My situation is perhaps similar to yours, and perhaps my solution is also helpful to you.
I have 200GB of data, spread across several IDE disks. I'd love to back it all up, but money is tight, and I can't justify the cost of removable media. If I could, I would probably look at a CD jukebox with CD or DVD writer, since this form-factor looks to me to have some life left in it, and I could put my DVDs in there too. My compromise is to consider only a portion of this data to be valuable enough to backup, so I periodically copy it from one disk to another.
I'm doing a find|cpio, and I'm not pre-erasing the target directories, so I expect that over time the target directories will (slowly) grow to be bigger than the source. It's a compromise between having an accurate recent image, and having a good long retention time.
Since I find that any time I buy a new disk, it's bigger than my old disks, I'm not expecting disk capacity to be a big problem! Also, having these disks online all the time will allow me to react quickly to problems, although this is only running for about a year now, and "Dan's Data" has some disturbing things to say about the lifetime of these cheap disks.
If you're worried about the reliability of powered-off disks, what about leaving them online? If your computers are already distributed, you could install disks in several of them, and copy data between them. You could also exercise the disks regularly, to get advance warning of media failures...
So after a brief look at hardware RAID I realized that the software RAID support in Linux was all I really needed. Since this is my own machine, I didn't really need the hot-swap capability of a hardware RAID controller.
I bought two 100GB Western Digital drives and set them up in a RAID-1 configuration. A month later, I bought another drive, replaced one of the drives in the machine with it, and put the removed drive in the safe. A month after that, I bought another drive and repeated the process, this time moving the drive in the safe to an off-site location.
Every month or so I repeat the process, rotating the second drive of the array through my various offline storage locations. The real beauty of this (especially vs tape) is that I only need enough downtime to swap the drives and reboot the system; the mirror reconstruction runs in the background as I use the system normally.
The use of RAID-1 gives me complete protection against data loss in the event one of the online drives fails (though I've had no failures yet with the WD drives). If both drives are somehow ruined (e.g., by a fire within the computer), or if I accidentally delete something important, I have my first offline backup, less than a month old. If that's also ruined (e.g., my whole house burns down and the fire-rated safe fails to protect the drives it contains) I have my off-site drive, which is less than 2 months old. Obviously I could easily extend this process with more drives and more offsite storage locations.
Because the backup drives are regularly rotated into online service, bearing stiction should be less likely to occur. And if an offline drive were to fail when I bring it back into service, so what? It was about to get overwritten anyway.
Naturally, I also continually back up especially important files (e.g., email, work projects, documents, etc) to various machines over the network, as that's the easiest and most effective way to protect small amounts of data. But when it comes to periodic full backups of big disks, nowadays I just don't see any practical alternative to disk-to-disk copying. And RAID-1 is the easiest way to do that copying.
Oh and never assume that because your backups are working your data is safe, I've seen a tape drive fail that rendered all of the tapes created by it unusable as the heads had been slowly drifting and none of the 3 replacement tape drives would touch the old tapes :(
According to this page, the expected realistic life of "high-quality" CD-R media can range from 50 to 100+ years.
Apparently, since DVD+R and DVD-R use higher quality versions of the same materials and read/write process, the expected shelf life for them is also from 50 to 100+ years.
(A quick search on Google will show you all sorts of estimates, but the 50 to 100+ year life expectancy numbers are quotes from TDK and Kodak. The question is, do you believe them? I guess I do...)
P.
55Tb ? Pah! Our current on-line storage is 218Tb ... and we don't even have a single customer yet ;-)
Cue the next "Mine is bigger than yours!" :-)
Mark
Other than periodically transferring to the to ensure long term readability, does anyone have any serious suggestions?
printing out on paper isn't that dumb a suggestion as a belt-and-braces backup for long term storage.
Finally, what format are you going to store that data in? Is Word 2000 going to be around in 30 years? ASCII or XML? All important choices...
$500 to back up 220 gigs?
your data must not be worth dog shit.
I suggest you should change the medium you are using to backup from time to time. You currently have 220GB of data to backup. I would create a RAID5-Array of IDE disks which can hold upto 300 .. 400GB of data. As soon as the ammount of data is increasing and coming close to the maximum ammount of data your can store/backup, add an extra disk to your RAID.
I also want to suggest to re-evaluate your
backup system every 3 to 5 years to see if is still fits in the backup strategie you have.
And if required changing to a new backup system.
Whilst I agree that Canada is a pretty cool place, if you are serious about your data retention, you would store your CDs in Paris, sitting outside a cafe with an espresso and a pair of dark sunglasses.
Lately I've been testing amanda as a backup server. It's quite good. You can do backups of your PCs to the server via network.
It stores data in a "holdisk" and it even can be configured tapeless. Or you can configure to always keep as much as possible in the holdisk.
Add RAID, and you can have cheap backups on IDE. It requires the disks always on-line, but then you'll sure the data is still there, because when a drive fails you notice and can replace it without loosing data.
You can always do rm -rf / and screw it all so tape backups are always good to have.
Paper too (acid free or not) can turn brown and is susceptible to bugs eating through it. The best way to store data is punch cards made of plastic. Maybe this can be shrunk to use needles to pinch holes into the plastic, but plastic should be thick enough to withstand time, weather, bugs etc. Of course it shouldnt be bio or chemodegradable.
Perhaps CDs made of gold film (or platinum) will work. The plastic of ordiniary CDs is good enough, but the metallic layer isnt. Certainly shouldnt be silver- or aluminium based. I'm sure it doesnt take much gold to plate a side of a CD and shouldnt need too powerful a laser to burn it.
Then again I'm assuming the laser will burn the plastic thats in contact with the gold, not oxidise the gold itself since that wouldnt work, else we can try building cdRW with more powerufl lasers. By all means the CD should be readable in normal drives since the company that makes these new CDs and CDRWs could be gone by the time someone needs the data.
I'm thinking centuries not 10 years of retention.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
The storage conditions are important too - there is a tract of Landsat data that is forever lost because one of the store rooms flooded and mildew killed the tapes.
Whats even more frustrating is where you have the tapes, but not the readers and machines to understand them. I understand some of the Apollo data may never be recovered because no-one thought to archive some of the tape mechanisms and computers at the time - something that is now done semi routinely at NASA
Ive tried looking for IDE drives focusing on stability.. mayb single-platter, 5400 or lower speed, low power, low density drives which can be guaranteed for about 5 years. Companies have reduced their warranties from 3 to 1 year, I'm sure a niche market will take drives with 5 yr warranties even with higher perMB costs, lower speeds etc. I remember seeing a thin maxtor 10GB drive inthe store the other day, single platter and not too fast, and this was manufactured last year (me thinks). Its relative simplicity should make it more stable, a mirror RAID of these will be perfect for servers. It would be nice to see a company step forward, offer maybe larger 5.25" single platter 4-40GB low-power drives that focus on stability and provide wannaties of at least 5 years. I never felt comfortable with the new ATA133 700mA 7200 drives which along with a geforce3 card and athlon processor needs a 400W power supply and extra fans. Think of running this in Pakistan where the temp touches 50C in summer.
For a longer term solution, gold-plated CDRs should work for a long time. Anyone has ideas for mediums that will work after being stored for 200 years at STP? Should be made of plastic, maybe gold plated, but I dont know what glue can be used to bind them together. Wouldnt the gold peel of after the effects of several seasons and humidity levels? Can it possibly oxidise in contact with plastic? Should we add glass? Or maybe cover the entire thing in glass making for a heavy robust CD??
Has to be some company out there that sells medium to write messages on for your great^5 grandchildren.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
*raises hand* I still have the tapes, and 5.25" floppys, BUT I still have the drives- and OS's- to go along with it all. Working systems, working software. I even made disk images of the DOS 6.22 and Win 3.1 install disks, and trust me, it SCREAMS on a Pentium 200 with 128 MB RAM..Windows startup time? 2 seconds flat.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
Buy yourself an array of radio telescopes, and beam all your data into space. Perfect data recovery (in 50 years time) will be assured through the use of alien technology[1].
This follows the best practice of always storing critical data offsite in case of disasters like, er, the sky falling in.
-Baz
[1] of course you'll have to escape the slave labour camp first.
Bought one two months ago.
I had purchased an 80 gig HD to replace my failing SCSI drives (Bus was flaking out.) around that time.
Guess what I just bought this week? Another 120 gig HD with an external Firewire/USB 2.0 enclosure.
retrorocket.o not found, launch anyway?
http://bay4.de/DataBeholder/
It's small and dirty but works and is efficient at what it's meant for. And it's free. That's the most important part of it. *g*
Stiction..... This brings back old memories of the time my company, a high end Unix vendor, found out that the scsi drives we were using from a well known drive vendor has this problem. At that time a TB of space took up a full rack or two of space, so anyone at the high end had say 50 or 100 drives. Our Fortune-500 customers were not happy when after power cycling an array, a handfull of drives didnt spin back up.
:-)
Well, I learned more about how lubricating oil gradually makes its way down the arm, to the head, and lodges at the interface to the platter than I ever wanted to. But on the other hand, I also learned how a failed disk drive can make a good frisbee. It turnd out that pretending to throw a stiction-ed drive like a frisbee caused the head to dislodge and fixed the drive about 20% of the time. ("Of course wait until the customer isnt looking before trying this onsite...."
We ended up replacing all the drives with stiction potential (failed or not) at customer sites in the field and sending the original disks back to the disk drive vendor. Word was that the vendor then resold them thru their retail PC channels, i.e. where you and most PC users probably get your drives from. This wasnt as horrible as it sounds, because at that time PCs typically did not run for the 3-5 consecutive days (ask me if you want to know why) that it took for the potential of stiction to manifest itself. (This was in the Windows NT 3.5 and 4 days.)
So remember, you get what you pay for. Someone out there likely has drives that have a higher level of reliability and are "guaranteed" not to have stiction at any significant rate. We were definitely told at the time that the drives were not supposed to fail like this and that there was a particular flaw that caused it in this batch of drives.
But obviously the mass market wont pay for this higher level of reliability. If you go this route, my suggestion would be to contact the disk vendor directly to try to find out if you can get drives of a higher quality than what you typically find on the street or in any mass marketed PC. You would certainly want these if you go this route for archiving. But of course you would have to open your wallet just a little bit wider.
Oops, sorry about the units snafu - but I an thinking of 100baseT, isn't it baseband. I didn't think there is a carrier with baseband.
As for the shelf life problem, I guess one remedy is to "re-fresh" the backup every year - i.e. grab another drive, format it and then copy the data from drive A to this drive (B). Then rotate drive A from drive C, etc.
This is increasing the cost, but if the drive is worth it... I have seen some people say the procedure is need to for tape. I think that even if tape has a longer shelf, the increased speed of hard drives would be it worthwhile. Especially if you have mission critical systems that could need to be restore quickly in prime time. After a failure, the time to restore can get one HELL of a lot of attention....more than you could ever want.
Fujitsu HDDs!
...and he grinned, like a fox eating shit out of a wire brush.
And if you want to be REALLY fast, you can RAID stripe across tapes.
I had a Dell PowerVault 130T; 4 DLT4 drives, 28 slots for tapes, and a roboarm. There was just something indescribably nice about having four separate backups going at once, all nice and zippy.
Vintage computer games and RPG books available. Email me if you're interested.
See what happens when you try to edumacate people?
You get modded down pretty quickly.
It's a shame that anyone expressing an opinion on slashdot is a Troll.
The only other option is to backup from here to eternity, putting your data on the latest available media. Don't worry, in about 10 years one CD will hold all the data you could ever produce. After that we'll have crystal data storage in which you'll be able to put oh, say, 1000 TB in a one centimeter cube crystal.
-- Taibunsuu
Of course, (220GB / (8GB/disc)) ~ 28 disks. Hmm... on second thought, maybe that's still not a good enough idea..
"I was a apprenticed to a man who was living in the eye of the hurricane to know despair. He knew all the dreams by heart..."
I can't understand why everyone thinks IDEs have to be low reliabilty. One year warranties are a result of the fact that OEMs who buy 99% of these drives have warranties = 1 year, often far less and won't pay a penny for anything not essential to make a sale. A warranty is a very expensive item.
a pplications/index.htm
Maxtor builds drives with a mean time to failure of 1,000,000 hours --- 114 years - is that enough?
See http://www.maxtor.com/en/products/ata/enterprise_
In nearly a decade of using 8mm tape I've written a few gig nightly to 2 drives I'm approaching 100 TB of data written and nearly 100% successful restore operations. I have exactly one instance of data-loss in that time and many instances where the rotation scheme saved someone from a poorly timed deletion or other accident / error.
Total failures, 4 dead tapes, 1 drive refurbished. I don't have a controlled environment or especially perfect storage conditions, yet I've had no problem reading data that's 8-10 years old.
Just pulled the july 8, 1995 tape and read it just fine (nothing older than that happens to be onsite just now).
Sure, it's a pain when something breaks, but honestly I put more time into working out the methods to not be surprised than I've ever lost due to lost data i.e. we don't rely on any one tape ofr anything critical.
Perhaps more importantly planned backups have saved entire systems when the o/s got wedged, and if ( - it's not happened yet) a system gets rooted we can restore known-state with minimal disruption.
Ohh and btw I've seen the spindle bearings of out-of-service hard-drives freeze when left off for extended periods -- those are not odds I like at all.
Linux is Linux, if One need clarify their dist: <Dist>/GNU Linux
bsds are of course just BSD
It's not so much an issue of restore time but one of archival longevity. There are a great number of instances where various data must be kept for a number of years (telco call records, court documents, financial transactions, tax records, etc., etc., etc.)
Any "mission critical systems" should have sufficient redundancy and resiliance that their failure would be beyond the scope of any backup planning -- like, say, airplanes crashing into the data center, fires, wars,...
Okay, magnetic tapes suck and degrade after x number of years... What if we use a mylar material to make a tape in a similar format, but us a laser to change the state somehow. I don't know, black spot or a hole, whatever. I think that the only drawback might be a write-once limitation.