Slashdot Mirror
Best Redundant Storage for Home Use?
Brad Mace asks: "Despite my hard drive's dedicated service, I'm aware that it will someday fail. I'm not really interested in burning 100 CD-Rs to backup my hard drive, so I've been looking at RAID solutions. Obviously I don't need the best or the fastest stuff out there. What would be a reasonable setup for personal use? Have people had better experiences with internal raid arrays, external raid towers, or networked storage such as Snap servers? I'm primarily interested in low price and ease of use."
Its not an ideal solution (what is?) but drives get bigger and cheaper all the time; just upgrade a bit more often than you would, keep a couple in the computer and rotate the others as offline backup... one or two may die, but they won't all die
If you buy a RAID card with on board cache make sure the cache is battery backed up otherwise a power cut may corrupt your array. Some of the cheaper cards don't have battery backup.
Despite my hard drive's dedicated service, I'm aware that it will someday fail.
In terms of storage efficiency, nothing beats the naggy girlfriend:
The downside, though, is the insanely high maintenance fee. Noisy too.
raid only provides availability. if your fs gets hosed, you delete a file, or whatever you're unable to undo the damage. ditto if your pc gets burned or stolen.
if you want a real backup, make a real backup. if you want to do it cheaply, buy another drive, copy the contents of your data drive to it, and store it someplace safe. buying an external usb2/firewire enclosure will make this a lot easier.
I run a 6x120Gb software raid here. The redundency is nice but when you get that much storage in one place you start having other concerns.
Drive failures are scary. Note, if you're going with a hardware highpoint 1540 controller: forget raid 5. The array takes forever to rebuild (like 4 days) and sometimes fails midway though causing total data loss. (Other people may have different luck though, my friend is doing hardware on 4x160Gb and not enjoying it). I've had 2 drives fail on my software raid and the rebuild went well enough. I suspect the failures were due to inadequate power from the UPS causing a "brownout" condition when my power went off, since both failures happened after a power outage.
So other problems: thats 600Gb of disk space. Nobody in their right mind wants to sit through an fsck, so a journalling filesystem is needed.
but, all the journalling filesystems are new and untested. New your saying? They're a few years old.. and yes, thats true but they're new in comparision to all the other filesystems. And when you're talking about 600Mb of data on one filesystem it really starts to concern you.
(Even so, I've been running raid 5 with journalling on 4 to 6 drives for the last 1.5 years or so and haven't had filesystem corruption)
A friend of mine is running mixed drive sizes in his fileservr and doing no raid. He occasionally has failures and loses stuff, but at least he doesn't lose everything. Still, I'd be pissed if I lost an 80 gig chunk at a time.
The problem with raid is, you gain some redundency but you completely lose the ability to make sensable backups. Unless you're a corporation and can afford $3k for a tape drive (and an additional $500 for tapes) you're faced with the idea of mostly redundent, but no backups. Offsite backups become really appealing, but then you need to shell out the same money for drives that won't be put to use (until something happens)
I still don't have a solution for offsite backups.
As an aside, I started playing with encrypted partitions (not raided). That has the same sort of scaryness. One filesystem screwup and you lose everything. No backups unless you have a second hard drive. Then you're faced with raid 1 which would corrupt both drives, or copying the encrypted volume from drive to drive each night.
I've got no answer for that one ethier, except that encrypting a tape backup would probably be good in a corporate situation.
OK, my own experiences in this area:
I was looking for a basic, cheap RAID system to give me some redundancy.
I got an Adaptec 1200A RAID Controller card, and used two drives in RAID1 mode. This served me well, until I needed more capacity - RAID1 has 100% wasted space.
So I looked at RAID5. I got a Highpoint RocketRAID 454, because it was cheap. BIG mistake. The write performance, on my P4 2.8 Ghz, with three WD1200JB drives, was a terrible 9 Mb/sec, with 80-100% CPU usage. AVOID. I returned it then next day.
Now I have a Promise SuperTRAK SX6000. It's very nice, 25 Mb/s write with only 20% CPU thanks to RISC processor, but expensive.
In summary:
If you want RAID1 only, a nice cheap Adaptec will do fine. If you want RAID5, you will need a reasonable card. Promise do some cheaper ones than the SX6000, with less channels, you could look at those.
Hope this helps!
You need to determin what you want to accomplish first. I've taken the stance of creating paper copies of everything on my computer that I care about (until my printer broke...), which works because it is only a small amount of financial information. Everything else can be re-constructed, or just forgotten about. OTOH, I want my grandpa to have a better backup system because his computer holds family tree information. (He also has a lot of historical information that isn't in the county archives but should be)
So first you need to take inventory of what you have. All those illegal mp3s can be downloaded from the net again. Your OS can be re-installed from CDs or the net, as can all your applications. All those jokes that you are saving can be found on the net. All those short stories that you have created can be published on your website at your ISP, and re-downloaded from there. (Make sure you keep this up to date two way, ISPs don't always backup websites) Usenet is a good palce to publish that, and google will archive. Family pictures belong on your website for the family to see.
That leaves a small amount of private data. Is it still an unreasonable amount to burn this to a CD/DVD? If so invest in tape backup (which is expensive, but holds more data).
Do not forget off-site storage for all this, a fire will destroy your home including the computer and backup. This is the biggest problem I have with RAID.
Seriously... I got on eBay and bought an older server (dual PII), with RAID and 5 hot-swap SCSI bays and a tape drive. I bought 6 SCSI drives the same way. Got it all for less than the cost of a new desktop.
I keep all my data on the server, and do backups. Anything short of a house fire or a stupid robber willing to lug a heavy old server and I'm good...
I'm sure that the flames will now commence as many Slashdotters seem religiously opposed to tape. In fact a recent Slashdot artice cited some tech magazines prediction that tape was obsolete. But, tape is still an excellent storage medium.
The fact is that tapes last for a very long time with a shelf life of at least decades. Most especially when they are not used much, as is the case when tape is used to backup a hard drive and is only accessed again if the hard drive fails and a restore is required.
Tapes store much more than any other removable storage medium. CD-ROM and DVD can't hold anywhere near as much as most tapes can.
Tapes do not cost too much, contrary to what many people say. Right now, on eBay, Travan 8GB drives start at US$5.00 and DLT 30/70 GB drives start at US$50.00 and go on up to >US$300.00. The tapes themselves can be more expensive especially when one chooses the preferred new tapes but, again eBay has DLT tapes available for anywhere form US$2.00 to US$100.00 per tape.
You suggest that you may choose to use RAID. But, tape is still a better alternative. RAID is effective in protecting you from a single drive failure but, it does not protect you against accidental deletion, rogue applications, viruses, versioning or controller failure. On the other hand, tape and the right backup strategy can protect you from all these issues.
Almost always tape backups are the best alternative.
A RAID array is a good backup for hardware failure in an environment where 99.99999% uptime is the goal. However, many failures that would blow your drive are software based... For example, if you are running under Windows XP, one of MS's specially formatted "updates" could wipe out your system. And with no widely available NTFS repair solutions such recovery can become very expensive very quickly. While this may not be as much of an issue on a non-proprietary file system, even Linux sometimes requires a full system re-install. Having that 2nd drive on an ATA chain will save you the few hundred for a good raid card, and will save your data in the event of, for example, a virus that deletes all local data... or accidently typing "apt-get upgrade" twice in a row.
Just set a cron job to mount, copy all, and unmount every week. Short of a rapidly moving brushfire, you should be all set. All this for less than $150 dollars.
The ______ Agenda
I've been dealing with mirrored RAID cheap harddrives for a couple of years now on medium duty systems (in harsh environments), and I can tell you two things:
1) I haven't yet seen a situation where both hard drives failed simultaneously so badly that I couldn't recover most* of the data.
2) RAID is not a replacement for a good backup.
No matter what, you should keep a seperate copy of the data at minimum separate from the computer itself, ideally offsite. You should also have a mirrored setup so a small failure (one drive, or fried computer with still working drive) won't set you back to your previous backup data.
What I've done temporarily is use a HD caddy (but now I'd go with a USB 2.0 or Firewire drive) to take an occasional snapshot which I can take offsite, storing it in a different place.
You should plan on expending at least $200 for a decent backup solution.
Further, I suspect you overestimate your backup needs. Compression helps with most everything. If you are backing up movies, burn them to DVD and keep them offsite. If you are backing up programs, burn those to disc and keep them offsite. If you are backing up pictures, burn them the DVD and keep them off site. You only need to continually re-backup items that change over time, and that data is vastly smaller than what you think you need to back up right now, and generally can be compressed up the wazzoo.
But you're looking for the easy way out. So, do mirrored drives, and try to get an external drive at some point later in time.
-Adam
*I did have lightening take out a server, which fried everything in that computer. I'm glad I had a regular backup. For kicks, I tried anyway, and one hard drives still worked, but a small portion of the data was corrupt, and it wasn't worth fixing since I had a known good backup from the night before.
Right now, hard drives are the cheapest data storage medium bar none. Even an external USB or Firewire drive is cheaper per gig than tape/DVD/whatever for mortals once you factor in the cost of the drive (datacenters with terabytes of data are a different story).
For most of us, we have a lot of data that we keep "just in case", and that collection grows pretty slowly. So, once you've made the initial backup, the incrementals are pretty small.
There are three types of failure to worry about: drive failure, stupidity ("oops. I just deleted everything."), and major physical loss (theft, fire, etc.). Any second set of data on different media handles the first problem. An asynchronous second set of data (i.e. not RAID) handles the second problem. Off-site backups handle the third.
So, for off-site backups, I made an initial backup locally with a Firewire drive, then I shipped it to a friend's house. He leaves it plugged in to his machine, mounted off of my home directory on his machine. I make nightly incrementals and send them off to his place. Every few weeks, I grab the drive, take it back to my place, and do a fresh full backup.
If you have no friends (this is slashdot, after all), you could probably take the drive to work, so long as your boss doesn't object.
Forward, retransmit, or republish anything I say here. Just don't misquote me.
Software RAID
You are just looking to ensure that if one HDD goes down, you still have most of your data. Software RAID mirroring incurs large performance penalties. Oh, and do note that many companies produce "Hardware RAID" cards that aren't hardware raid. Its hardware ASSISTED raid. These systems can't, or have great dificulty booting root of a raid system.
Hardware RAID
Either SCSI or IDE. I've changed my tune about IDE RAID. If you do it in hardware, it is a pretty good and valid option. It provides increase in speed and ease of setup (in most cases, litteraly plug and play). 3ware are a good company for hardware IDE raid controllers. Also they are very well priced.
Big hunking hotswappable SCSI RAID
If you have money to blow, here's the best place to blow it. Here the RAID system (whatever level) is done with hotswapping drives. 99.99% of the time, THIS IS A WASTE OF MONEY. For most small companies, and a lot of midrange companies, hotswap is throwing money away. Unless your server NEEDS to run 24/7 with 5 nines reliablity, hotswap is a waste. Hotswap only offers the ability to replace a bad drive without downing the system. But if you want to spend the money, or like the look of the flashing lights on the cases, go for it.
Ok, next, is RAID the answer?
Yes and no. RAID (mirroring) protects against hardware failure. It doesn't stop anyone from typing `rm -rf /` or anything similar. So if you want backups against mistake, RAID doesn't offer this.
Backups
Unless you have money to burn, tape backups are expensive. Most of us have 20 or 40 gig HDD drives. If you try and buy a tape backup at this level, you are talking a significant ammount of money (eg possibily the same as the cost of your system in the first place).
It is possible to do tape on the cheap, but not recommended.
CD/DVD burners
CDs hold 700MB of data (maximum), so doing a 20gb backup would require 29 CDs in total. This is probably not viable. So lets put that asside for now.
DVDs hold between 2 and 4GB (depending on various hardware, etc). That becomes more viable. You wouldn't do a full backup each day with this mehtod (still requiring 6 dvds to do 1 full backup), but you can do it using a base + data method.
By backing up the entire system on 6-7 dvds say once a month, you can then procede to do incremental backups over the month on 1 or 2 dvds (depending on how much you change). If you also are careful how you store your data, things that you want, but aren't likely to change. Eg your mp3 collection, you can backup to a seperate dvd, saving space on your incremental and full system backup.
Removable HDDs
The final option is, imho, the best. By having, preferably, 2 external HDDs (either using USB drive holders or the old caddy system), you can backup most, if not all of your system onto the removable drive. If you get removable disks that are the same size as your internal HDD, you can backup the entire system.
The pros are obvious. The cons however may bite you. First, in theory, you can drop a tape on the ground and have no problems. Do NOT drop your remoable HDD. So you'll need to be careful. HDDs are also rather more volatile than tapes or CDs when it comes to keeping its data. You'll need to ensure that you store the HDDs in a good location.
If one HDD fails (either a removable, or your internal) you'll be able to restore from one of the working ones. I said you should have atleast 2 removable HDDs because when you do a backup, the first thing you do is delete the old data from the removable. if the system failed then, you are screwed.
I prefer this system, and for some SOHOs I offer it as a viable option if they need data backups.
Summary
Either way, in the end, it comes down to how much you are willing to pay. Backups are still NOT cheap. Its something that should change, but in the short/medium term, it won't.
I use to have a funny sig, but slash cut it off, and I forgot what the punchline was.