Slashdot Mirror
Might Flash Memory be a Viable Backup Medium?
General Books asks: "Rather than fuss over mechanical failures and damaged media, why not use flash memory for backups? We maintain about 100 servers distributed to customers' sites. Each night we copy a backup of critical data (generally less than 128MB) to removable media in case the hard drive fails. We have experienced high failure rates with CDRWs and so now I am considering some sort of flash memory like a USB key drive. They are solid-state and you can get a 128MB device for $20. They seem ideal to me, but I can't find solid evidence. One question is how would they endure a lightning strike (perhaps not as good as an optical medium)? Admittedly, there is a wide variety of CDRW drives and media but don't they all seem risky compared to a solid-state device? More info about my circumstances: We have no network for backups. A second hard disk is not viable because it could not be rotated offsite. Tape drives are relatively expensive and overkill for our volume of data."
You mentioned that CD-RWs weren't all that good, and I have the same experience, but why not try normal CD-R discs? They're cheap and pretty reliable. You can even make multiple backup copies if you want, or a multi-session disc to store several backups on one CD to cut costs.
I suffer from attention surplus disorder.
A USB or Firewire external drive would store far more data and could be rotated offsite.
When that place burns I want to be sure all the evidence goes up in smoke.
134340: I am not a number. I am a free planet!
So let me think this through... you are trying to store small amount of data on individual items. Why? Once in storage, are they kept apart fro one another?
I think a RAID will suffice. Locally. If you need to keep it on the cheap, pump your small amount of data to 3 other simple boxen offsite. I mean, for small amounts of data, there's no reason to muck around juggling the physical medium. One can duplicate that data faster and more reliably than boxes of little memory cards with scribble on them.
If you need to go cheaper, try floppies! W00T!
mug
If you carry it with you when you aren't actively backing it up, you don't have to worry about lightning, fire or whatever. If you are struck by lightning, burnt in a fire or in some other way destroyed, you won't really miss the data.
Or why not get yourself a few of those neato caddies that hold a HDD, and allow you to swap them out (internally; I am not talking about external enclosures) they are available for IDE, and the more expensive ones (claim to) allow hot swapping, even (I cannot personally verify how well the hot swap feature works or doesn't ...). I have seen them that even allow you to lock them in place with a key; how cool is that?
... i.e. just pull a drive, and put a fresh one in, let the card rebuild it for you; backup your whole system if you like. Restoring doesn't get much easier than that, either.
Much cheaper in the long run, in terms of media costs, at least for large quantities of data. Especially if you score some inexpensive smaller drives (like a surplus batch of 10 GB or so)
Hell, if you went all the way and just put an inexpensive RAID controller in there, it might pay off in the simplification of your backup procedures
Here is one made by 3ware
Here is one made by Promise
There are plenty of other, cheaper ones out there, too
I cross my fingers (no, not literally) every time I've inserted a piece of Flash Media into my camera, PDA, or USB drive. I find that about one in every 20 times the disk comes up empty. I take good care of them (three different media types), but I don't find them reliable at all. I'd sooner use CD-Rs (though now I might start backing those up every 2 years on new CD-Rs.
Alex.
I work for a university as an educational content developer. We are in the same situation in that our whole codebase doesn't break the 100 meg mark but needs to be kept extremely safe given that it represents months and months of man hours.
We had a lot of bad luck with CDRW's and ended up dropping that idea and moving to a dual backup system. We do intremental backups to CD-R's and make two copies. Secondly we push the content to an FTP site that is elsewhere on campus. The FTP site is backed up seperately onto tape as added an precaution.
Just my $.02
Before you give up on CD-R/CD-RWs, try this:
I don't see any reason why USB flash media wouldn't work for backups if the OS supports it. The only problem is that USB flash media is more expensive then CD-Rs. $20 will buy you enough CD's for a monthly 'archive' (12 CDs/1 per month), plus a weekly backup/incremental daily backup (4 per month), even if you don't reuse the weekly backup media (personally, with the cost per CD, I wouldn't). However, to implement such a system with USB flash drives (assuming $20/drive), would cost $320 dollars.
I love my USB flash drive, but its not cost effective for backups.
I can still read CD-R and CD-RW discs that I recorded 4 years ago without any problem. The only disc read errors I've had stem either from horribly scratched media or dying CD-ROM drives. Maybe you should just experiment with different burners and media and see what happens.
On the subject of flash media, I haven't heard of any tests saying they have any real reliability. They were created to share files easily and quickly, not for long term storage as CD media (in theory at least) was.
Two things are infinite: the universe and human stupidity, though I'm not yet sure about the universe. - A Einstein
one of these perhaps?
No TiVo and no caffeine make me something something...
Is the "generally less than 128MB" before or after compression? A nice compression package like 7-zip might get the files down to a size that can be emailed off-site each night.
-One question is how would they endure a lightning strike (perhaps not as good as an optical medium)?
Given that when you aren't actively moving data onto / off this thing it is supposed to be in your pocket or on your keychain, if it gets hit by lightning then 'data retention' is going to be pretty low on your list of concerns.
Glonoinha the MebiByte Slayer
You say there's generally less than 128MB of data which needs to be backed up; how much of that is new each day?
If most of that data isn't changing, you could use those antique things called phone lines to transmit the differences.
Tarsnap: Online backups for the truly paranoid
What operating system are you running? How long do you need to keep each days data? Does it need to be stored offsite? Is this data security sensitive?
.bat files on a Windows server would work similarly. Make directories on the backup HD corresponding to day of week/month and automate a job to copy the data to the appropriate backup directory. Automate a job to tar/gzip or zip said backups. Automate a job to FTP those zip files to a remote server. Hard drives are dirt cheap, and with the tiny amount of data you are talking about you could hold a whole years worth of backups on a $70 dollar hard drive. Need redundancy? Do the same process to yet another machine with another $70 hard drive. Never buy a tape or CD-RW media again. Never worry about backups again. Set it and forget it. Have the backup machine(s) email you the results of each night's backup just to be sure they happened. Make a test file on the server and delete it and restore it every so often to ensure you can recover last data. Burn monthly to a CD-R for even better sleep at night.
An extra hard drive and cron works wonders on a Linux or Novell server. I assume the task scheduler and a set of
Rather than fuss over mechanical failures and damaged media, why not use flash memory for backups? We maintain about 100 servers distributed to customers' sites. Each night we copy a backup of critical data (generally less than 128MB) to removable media in case the hard drive fails.
Both of these would be my recommendation. I use flash media to boot firewalls, routers and embedded servers that run from RAM drives (nearing 100 deployed at customer sites and in our network). But I automount a partion on CF modules for logs. Flash memory is very reliable; it's rated at about 100,000 destructive writes. Read that as wiping it out, reformating it, not as I wrote to /var/log/messages for a week and the media toasted because somebody's machine caused the firewall to log crap every 2 seconds for a week. If it wasn't reliable, Cisco wouldn't use it for non-volatile storage (neither would I).
The way we handle server backups is for servers to backup via a script to a tar.gz file over a private T-1 for servers. Granted, this amounts to a lot of GB for us but if you use something like rdiff-backup or a more simple script that backups up your files across the net through an SSH tunnel, you should be in pretty good shape. CDRWs are a poor choice if you can't or won't rotate media routinely. Especially since their lifespan for writes is low. You or your customer will have to rotate if you use CDRWs.
you can always Print out the info. If you're really worried about lightning you can use a laser printer and marble
that's only one part of the story.
most of the restores i've ever done have been because some clueless user has accidentally deleted a file, three weeks ago, and only just realised they need it. if you're continually wiping out your old backups then you're fucked.
don't think "what do i want to backup?" think - "what do i want to be able to restore?"
You can read the Compact Flash FAQ
A quick google search returned these links, that may be interesting to you
IDE to Compact Flash Adapter
Flash Storage Solutions
Read all this thread if you will be storing sensitive information
How Compact Flash can keep your data safe?
This guy has an opinon different from mine. He says that, all of a sudden, he lost hundreds of picture. Well, I've been working with Compact Flash for more than one year, now, and the ONLY time I gost corrupted data was when I took the card off the camera while it was writing. Then the camera could not read any picture. They seemed to be lost. But later I put that CF in my CF reader, and ran a chkdsk. It found lost chains, that I saved as files. And recovered ALL pictures except for the bottom half of the one it was writing at the very moment when I removed the CF. It probably corrupted the FAT (same way as hard disks, when the computer is not properly shut down).
And I do think CF is more reliable than Microdrive.
-
Roses are #FF0000, Violets are #0000FF, find / -name '*base*' |xargs chown -R us && mv zig greatjustice
Forget the DLTs. Look for a college friend who is working at another company in the same city and is also assigned the task of backups. Give her a user account on your firewall get a user account on hers. Now setup perl scripts to encrypt the backup data on your site, bzip2 it and then transfer it to her site... and she should do the same for her site... swapping offsite space and gaining even a few GBs of space is quite enough for good backups. Just be sure your Internet connection does not come with download or upload caps.
"Give orange me give eat orange me eat orange give me eat orange give me you." -Nim Chimpsky
1) Slowest speed possible doesn't mean best burn. You may actually have to experiment. Frequently higher speed drives have less-than-accurate servos and do better when they aren't trying to keep the speed down. Pick a slower integer factor of the highest rated speed and use that.
2) Buy a lot of CDs, and not only back up the CDs, but have a schedule where you regularly duplicate older copies onto new media. CD-Rs can have short lifetimes (and can be damaged during handling). Check your md5sums after each burn, and maybe write the last 8 digits on the disc itself for quick verification.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
http://www.newegg.com/app/viewproduct.asp?descript ion=20-167-108&refer=pricewatche .yahoo.com/digi4me/mem12usbflas.h tml
http://shop.stor
The 256 is about $50-55, 512 is avail for less than $100.
It's a pretty good deal they have going now adays.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
I set up my powerbook 1400cs with two Flash PCMCIA cards. One is used as a boot disk and the second one is used as VM (virtual memory e.g. swap) both of them are 256mb and work fantastically. I removed the internal HD now I get about 6-7 hours of charge. While researching the effectiveness of using Flash in this manner I found some reports as to its low number of writes before breakdown but so far I've had no problems with the card I use as VM (swap).
NarratorDan
"If you're not confused by quantum mechanics, you really don't understand it." - Niels Bohr
You could try M-Systems's products like DiskOnChip or even their IDE/SCSI Flash Disks.
I'm working with embedded systems (Linux) in my company and I'm very pleased with the DiskOnChips despite their half-proprietary driver with which you can only generate kernel modules, not compile it into the kernel for legalese reasons. The DiskOnChips work way more reliable than any other flash chips I've used so far, with no defects yet (and we use them just like normal hard-disks currently).
Granted, it could be just that we had bad luck with the other flash chips we tried before, I don't know enough about the flash market / flash chips in general.
Disclaimer: No, I didn't get paid for this "advertising", but I wouldn't mind if someone would do it nevertheless ;-)
You're right that optical media, or anything that rotates only when being accessed, is very secure in the event that the machine takes a hit while the media is in the drive. For this reason, you might want to keep the CD-RW drives in the machines even if you're using a different technique for nightly backup, and do weekly backups to an RW disc or something.
A USB flash device, left plugged in to the machine for days at a time, would be connected to the PC's power supply during such a strike and might sustain damage. However, there are USB surge protectors made by several companies, they're just not super cheap. (Personally, I'd throw a $2 16V MOV across VCC and GND, and call it a day.)
Look at the costs before selecting a keychain drive. You might be able to get a USB-CF reader, and a large CF card, cheaper than a straight USB keychain drive.
Why exactly are you worried about lightning anyway? I'm sure you've got a surge protector on the AC power input.. are these machines connected to a lot of weird interfaces that would be impractical to put protection on? I'm imagining a warehouse environment where RS232 links and stuff might make things difficult, but B&B makes surge protectors for just about every interface you can imagine. APC has a pretty wide selection of surge protectors too.
If the threat model didn't include "massive jolt frying everything in the box", you could get away with some much cheaper backup solutions. Get a batch of surplus or refurb hard drives. They're cheap in single quantities, and practically free if you buy by the crate! Set your software to spin the drive down except during the nightly backup run. This reduces the chances of all sorts of failures, including data corruption risks caused by power glitches during operation.
I don't know if it would be of any use to your application, but having a ten-gig backup device means you could keep old backups for quite a while before reusing the space.
I think this was one of the most compelling reasons for getting a zip drive back in the day. Quick, cheap backup of 100 megs. Or with modern drives 250Megs. The media is cheap ($20 per disk), the drives are relatively cheap ($100) and they've never failed me. Except for the paralell drive that got the click of death, but the media didn't get corrupted.
In my personal experience, all of them have been more reliable than optical or flash media.
Since you're talking about less than 128M per backup, I'd seriously consider a Zip-250 drive. They're fast and
You should also be able to get an inexpensive tape drive. The old DAT/DDS-1 standard stores 1.3GB on a cheap 60m tape, which more than enough storage.
Heck, even an ancient DC-600 QIC drive will do it (600M per tape) although the tapes are a bit too large for me to find convenient. You can probably find one of these for free at a flea market (but I wouldn't trust critical corporate data to a drive I got for free at a flea market.)
There should be an icon with a green arrow with all the others in the tray. If you click on it, select the option that says "safely remove hardware." It will then display a bubble which states it's safe to remove your hardware.
Sent from my iPhone
Compact flash is addressed much like any other drive, but often the card's own logic distributes writes so that all portions of the card are used evenly. This is called "wear leveling" and extends the life of the card. The problem is that cheap no-name manufacturers don't implement it, so do a little research before picking one. Also, I hear that some of the nicer cards remap failed sectors and do read after write verification, though at the expense of speed. Again, check into the specs of the more respected brands before buying.
Now consider that a good consumer level CF card can do 300,000 writes on a sector before failure. With the benefit of wear leveling this means a 64 meg card can have rougly 64 x 300,000 megabytes written to it before failure. Now this seems fine and dandy, but you still have two potential problems to worry about.
Frequently accessing the card causes the OS to write information like last access time very frequently. Each time this happens the data is moved to a new sector on the card, using up one of the 300,000 writes that sector is allowed. In some scenarios this can wear down the card very quickly. With backups, the data usually sits unaccessed, so take it as a warning. Alternatively, pass the 'noatime' option to mount. This tells linux not to update access times.
One last pitfall - CF has a bad habit of dying if power goes out mid-write. For this reason I'd buy 2 quality CF cards with more capacity, and thus lifespan, than I need, rotate them, and not worry about it.
Would backing up to an offsite FTP account or two provide the protection you need? You could host it yourself or have an ISP with its own backups host it for extra protection. It may even be straighforward to automate depending on your setup. You can use GPG to encrypt the backups if you are concerned for their privacy in transit or at the backup storage site.
Since your backup is 128meg, you should factor a backup medium that has some room to grow. What do you do if one night your working late and your backup is 135meg? Skip it and wait for the next day to buy another key? So lets up your USB key to 256meg.
CDRW's, lets say they fail every 20 uses. 365 days a year, 20 uses per disc. That will be ~19 discs per year. @ $40 per 25 pack. $40 per year.
USB Flash, Lets say you go through 1 a year. @ $190 per 256meg of storage. $190 per year.
Now, it is quite possible that the USB key will last longer than a year. So your purchase can be spread over a number of years. After a few years the cost difference will even out. But there is a downside. What if you loose or break the USB key. What happens then? Your fuxored, plain and simple. At least with the CDRW's you have the previous disc to fall back on for historical data if you rotate the disc's before they phisically fail.
You can even use a new CD-R everyday and keep the daily disks for historical protection for just a little bit more than a USB flash key.
But it all depends on what you are looking for in a backup, temporary daily backup that is gone a day later, or an actual backup with historical protection.
Being called a dork on Slashdot must be like being called the retard in special ed.