Slashdot Mirror
Why RAID 5 Stops Working In 2009
Lally Singh recommends a ZDNet piece predicting the imminent demise of RAID 5, noting that increasing storage and non-decreasing probability of disk failure will collide in a year or so. This reader adds, "Apparently, RAID 6 isn't far behind. I'll keep the ZFS plug short. Go ZFS. There, that was it." "Disk drive capacities double every 18-24 months. We have 1 TB drives now, and in 2009 we'll have 2 TB drives. With a 7-drive RAID 5 disk failure, you'll have 6 remaining 2 TB drives. As the RAID controller is busily reading through those 6 disks to reconstruct the data from the failed drive, it is almost certain it will see an [unrecoverable read error]. So the read fails ... The message 'we can't read this RAID volume' travels up the chain of command until an error message is presented on the screen. 12 TB of your carefully protected — you thought! — data is gone. Oh, you didn't back it up to tape? Bummer!"
12 TB of your carefully protected â" you thought! â" data is gone. Oh, you didn't back it up to tape? Bummer!
If it wasn't backed up to an offsite location, then it wasn't carefully protected.
There are shills on slashdot. Apparently, I'm one of them.
When HDD's move to bigger sectors - there should be better error recovery reducing the probability of unrecoverable read errors. Right? Ok, I'm moving to ZFS.
Prediction: The real iPhone killer is going to be sex robots from Japan. Think about it.
RAID is not, and has never been, a substitute for backups.
I mean, WTF? Many people regard RAID as something magical that will keep their data no matter what happens. Well ... it's not.
Furthermore, for many enterprise applications disk size is not the main concern, but rather I/O throughput and reliability. Few need 7 disks of 2 TB in RAID5.
The Raven
The problem with Raid 5 is that the more drives you have the higher probability you have that more than one drive dies. That's why you have multiple raid 5 arrays of 4 disks maximum instead of one array of 7 disks.
If you use RAID to 'protect' your data, you clearly don't value your data at all.
While the interesting bit of this article is the coming demise of RAID 5, what you should be bringing away with it is, if RAID is all that stands between you and data loss, you're a noob.
That's what RAID stands for. It's a nice idea in theory, as long as the disks remain cheap, but I've never trusted them to work properly and had more than one break on me. "All you have to do is unplug the bad disk, plug in a good one in its place, and in a few minutes all will be hunky dory." Bzzt. Wrong. Thanks for playing.
Backup every day to tape, to another disk entirely on a diffrent machine, to R/W DVD, twice a day if you have to, or all of the above--anywhere else but the machine itself. RAID: the accident waiting to happen. Yeah, I'm paranoid. It comes from experience.
How about a moderation of -1 pedantic.
If you have one RAID5 box, just build another one that replicates it. Use that for your "hot backup". Then back that up to tape, if you must.
Storage is so cheap these days (especially if you don't need super-fast speeds and can use regular SATA drives), that you might as well just go crazy with mirroring/replicating all your drives all over the place for fault-tolerance and disaster-recovery.
A RAID 5 setup is only a precaution in case of an hardware failure. It serves as no excuse for not having backed up your data.
And the topic is also flawed - RAID 5 doesn't have any self destruct mechanism.
This story is just ridiculous. It clearly states that this doesn't affect Enterprise users, as their URE rate is lower and unless they're idiots they use smaller drives. What home users will have 7 disk RAID 5 arrays of 2TB disks? Is this really a large enough percentage of RAID5 users to call for the death of it?
If you put 7 disks in a single RAID5 without backup then its called bad design and bad implementation.
This has always been true regardless of disk size/speed.
As above posters have pointed out once you get past 4 disks the non-ZFS way to go is multiple blocks of RAID-(whatever number is appropriate for your scenario).
Though ZFS is awesome and if your OS/hardware supports it 100% there is little reason to stick with RAID
This is trivially testable. Any slashdotters have experience rebuilding 7TB RAID 5 arrays?
You'd think, if this were really an issue, we'd be hearing stories from the front lines of this happening with increasing frequency. Instead we have a blog post based entirely on theory, without a single real-world example for corroboration.
What's more, who even uses RAID 5 anymore? I thought it was all RAID 10 and whatnot these days.
I'm going to troll ridiculously old articles and post them to Slashdot and hope the editors don't notice... oh cool, they didn't here either!
I just wasted your mod points! HA!
I can see a lot of people getting into a tizzy over this. The RAID 5 this guy is talking about is controlled by one STUPID controller.
There are a lot of methods, and patented technology that prevent just the situation he is talking about. Here is just one example:
RAID is not perfect, not by any stretch, but if you use it properly it will serve it's purpose quite nicely. If your data is that critical, having it on a single raid is ill advised anyways. If you are talking about databases, then RAID 10 is more preferable and replicating the databases across multiple sites, even more so.
What is this article about?
They say that since there is more data, you're more likely to encounter problems during a rebuild.
The issue isn't with RAID, it's with the file system. Use larger blocks/sectors.
Losing all of your data requires you to have a shitty RAID controller. A decent one will reconstruct what it can.
The odds of you encountering a physical issue increases as capacity increases, and decreases as reliability increases. In theory, the 1 TB and up drives are pretty reliable. Anything worth protecting should be on server-grade hard drives anyway.
The likelihood of a physical problem popping up during your rebuild is no higher with new drives than it was with old drives. I haven't noticed my larger drives failing at higher rates than my older, smaller drives. I haven't heard of them failing at higher rates.
Remember, folks, RAID is a redundant array of inexpensive disks. The purpose of RAID is to be fault-tolerant, in the sense that a few failures don't put you out of production. You also get the nice bonus of being able to lump a bunch of drives together to get a larger total capacity.
RAID is not a backup solution.
RAID 5 and RAID 6, specifically, are still viable solutions for most setups. If you want more reliability, go with RAID 1+0, RAID 5+0, whatever.
Choosing the right RAID level has always depended on your needs, setup, budget, and priorities.
Smells like FUD.
The whole argument boils down the published URE rate being both accurate, and a foregone conclusion. Will disk makers _really_ make drives that have a sector failure for every 2 terabytes, or will they improve whatever technology is causing these URE's to be much more rare? (if the rate was real in the first place).
AccountKiller
How many times does this have to be said.
RAID is not a backup. RAID is designed to protect against hardware failures. It can also increase your I/O speed, which is more important in some cases. Backups are different.
Depending on what you are doing, you may or may need a RAID, but you definitely need backups.
RAID 5, as well as RAID 6 is nothing more at an attempt to add some amount of redundancy without sacrificing too much space. Go RAID 1 instead with the same number of disks.
As far as I'm concerned, RAID 5 really has no redeeming features (it's slow, not particularly safe, but lulls people into a false sense of security).
From a data integrity perspective, though, RAID6 is a better solution than RAID1.
Given arrays of equal sizes, with RAID6 your data can survive the loss of *any* two disks; with RAID1, if you lose two disks which happen to be a mirrored pair, then you're hosed.
But, as you point out, RAIDn doesn't really qualify as "carefully protected"
Not everything that can be measured matters; Not everything that matters can be measured.
In practice, this means that while your array is rebuilding, your performance SLAs go out of the window. If this is for an interactive server, such as a TP database or web service you end up with lots of complaints and a large backlog of work.
The result is that as disks get bigger, the recovery takes longer. This is what make RAID less desirable, not the possibility of a subsequent failure - that can always be worked around.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
It's implied in the Slashot description that ZFS solves the problem of drive failure. It does not. Just want to make that clear. In fact, I'd argue that there is actually more risk inside of ZFS with regards to the actual problem presented here.... for those that believe all is doom and gloom with regards to RAID.
Spell it out for everyone.
RAID won't save your data if there is a fire.
Or if you delete a file.
Or if two drives fail.
Or a thousand other scenarios.
All RAID does is prevent the system from going down when a single drive fails (except RAID 0). Thus giving everyone in the office time to finish up their important work and log out for the day so you can swap the drive. Or, if you're brave, swap the drive during regular work hours.
For the home user (not working on huge graphic files) RAID 1 (mirroring) should be sufficient. As long as it is paired with another EXTERNAL hard drive that you copy your important information to. And leave with your brother or something. I'm talking family photos and such. Your tax information should be small enough to fit on a USB drive.
If your computer completely failed TODAY what would be the really irreplaceable files on it?
Back those up. Then store them with a friend or someone in your family.
There, problem solved.
RAID 5 isn't a false sense of security. It actually DOES protect you from a disk failure.
I made the decision about two years ago that all disks at home will be either mirrored or RAID5. Disks are so dirt cheap that there's no reason not to.
RAID doesn't prevent you from having to have some sort of backup solution, and if you can't trust yourself to do them unless you're being risky with your data, I'll happily avoid dealing with restoring data and all that bullshit from a single disk failure and you can sink your time into doing it all manually.
- It's not the Macs I hate. It's Digg users. -
Scrub once a week, or once every two weeks.
RAID6 isn't about losing any two disks, it's about having two parity stripes. It's about being able to survive sector errors without any worry.
It's about losing ONE drive and still have enough parity to replace it without any errors.
RAID6 on 5 drives is retarded, tho, because it leaves you absurdly close to RAID1 in kept space. RAID6 is for when you have 8-10 drives. At that point you barely notice the (N - 2) effect and you have a fast (provided your processor can handle it all) chunk of throughput along with an incredibly reliable system. Well, N-3 with a hotswap.
Personally, I think I'd go RAID-Z2 via ZFS if only because it's a little bit sturdier a filesystem to begin with.
Wow, how incite-ful. Doesn't matter what the discussion is, some geek is bound to weigh in with all the shortcomings of any idea.
Newsflash: there is no perfect backup! No method is foolproof, especially when it's bound to be boring as hell, and you've got an inevitable human factor. You get lazy moving the tapes offsite, you put off fixing a dead drive because there are 4 others, you wipe your main partition upgrading your distro and forget that your CRON rsync script uses the handy --delete flag, and BOOM wipes out your backup.
Shit happens. Pointing out what we all already know doesn't do anything helpful.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
Sure expected the "editor" to actually look at the article is excessive. But:
"Disk drive capacities double every 18-24 months. We have 1 TB drives now, and in 2009 we'll have 2 TB drives."
Is an obvious indication that this article is old since 18-24 away puts you in 2010 now...
>>Seriously, you're kidding yourself if you think RAID is protecting you.
It's a question of what type of faults you care about. Right now, I have a 500GB RAID0 setup to boot and play games off of, an internal 250GB hard drive that mirrors my important folders every night, an external drive that both mirrors my important documents and is used when I shuttle between my two home offices, and an FTP server in a different city that I back up most of my small and important files to (I ain't sending my Snoop Dogg discography over my DSL line). Each has a different failure mode and risk factor. The RAID0 is most vulnerable to hard drive death (though it's been running since '04 without any problems, and my HD health monitors show it in good shape), the second internal drive (which would have been a RAID5 drive, except RAID5 sucks ass on my controller) is vulnerable to theft (steal the RAID0, steal the extra drive too), the USB drive is also vulnerable to theft, though perhaps a different kind of thief (it can be stolen from my car, for example), and the FTP server is slow as shit, but relatively secure.
I'm relatively lazy about doing backups, but having automated stuff to handle all of it makes it not much of a hassle.
RAID-10 ftw? Expensive I know, but at least you have a full layer of redundancy rather than just a parity drive.
RAID doesn't protect against your worst enemy
rm -r *
nor is it supposed to. not being a moron seems to have protected me from "my worst enemy" just fine. RAID has protected me from random disk failures. seems to be working as designed
TIAEAE!
Well what the author is saying is not true by my experience..
When I configure a new server with RAID, before handing over the box, I will test every single RAID 1/5/10 sets by pulling one drive out at a time for a couple of minutes then I would put it back in until the rebuild is complete.
I worked with multiple Dell PowerVault MD1000 using 15 1TB SATA drives and never have I ran into an issue of not being able to complet the rebuild because of the issue mentioned in the article.
I can tell you that drives will fail but if you have the right monitoring in place and catch when a drive fails or about to fails and with the right RAID solution in place, you can consider your data pretty safe. RAID is not a backup not 100% but chances are that a RAID solution will surly save your butt when a disk dies and you just don't have the time to rebuild / restore a box.
My observed error rate with about 4TB of storage is much, much lower. I did run a full surface scan every 15 days for two years and did not have a single read error in about two years. (The hardware has since been decomissioned and replace dby 5 RAID6 Arrays with 4TB each.)
So, I did read roughly 100 times 4TB. That is 400TB = 3.2 * 10^15 bits with 0 errors. That does not take into account normal read from the disks, which should be substantially more.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
I'm in the process of building a new 8x 1T array. I'm not using any fancy raid card. Just a LSI 1068E chipset with a SAS expander to handle LOTS (16 slots in the case, using 8 right now).
I'm not putting the entire thing into one big array. I'm breaking up each 1T drive into ~100GB slices that I will use to build several overlapping arrays. Each MD device will be no more than 4-5 slices. This way if an error occurs on one disk in one part of a disk I will have a higher probability of recovery.
I may also use RAID 6 to give me more chance of rebuilding.
Disk errors tend to not be whole disk errors, just small broken physical parts of a single disk.
SMART will give me more chance to detect and replace dying drives.
Seriously - what's the problem with RAID 5? It's not a FALSE sense of security: It actually DOES prevent data loss or down time on a single disk failure. If you're a moron, you're creating 14 disk arrays. If you're smart, you keep it to 7 disks at the very most.
RAID 5 is great. It's fast, unless you have a shit controller without enough cache. It's going to prevent down time on a single disk failure (which is overwhelmingly the most common type of failure) and it doesn't cost you too much capacity.
Usually I'm more concerned with a fire or flood than a double-disk failure.
RAID 6 is good, but you get the same (actually worse) performance hit over RAID 5. More parity calculations. You can lose any two disks, which is nice, and if you can spare the space, go for it!
I don't see RAID 6 as being all that much more of a big deal over RAID 5 and actually it shouldn't really have it's own number since it's exactly the same technology and parity system as 5. It should be RAID 5.1 or something. Or maybe RAID5+1. The only reason it's become more available now is because controllers have gotten fast enough to deal with the additional parity.
- It's not the Macs I hate. It's Digg users. -
In communications design, such as cell phones or digital TV, channels with low reliability (fading, burst interference, etc) are tasked with getting much better overall bit error rates than you'd think you could given all the crap spewed into the RF spectrum. I'm kind of confused by why the same techniques of forward error correction, interleaving, and such aren't employed more aggressively for hard drives (maybe they are more than I am thinking, maybe that's how you get to 10^-12 in the first place?). 10^-12 bit error rate is phenominally good compared to what most digital communications devices deal with.
Typically you throw away 20-30% of your available channel with extra bits due to the encoding (imagine hardware encoding by the hard drive as it writes the bits), but you are guaranteed that if you can get most of the bits right (I'm talking 99%, not 99.9999999%) you can get the original data back, or at least know that you didn't. Interleaving spreads the bits around, so one dead sector (or 10 in a row) can easily be rocovered automatically.
I use Unraid for all my data storage. The parity and data isn't striped across all drives, only one drive is parity. If I loose two disks, I only loose the data on those two disks. I've got 4TB of home storage at the moment, but will eventually scale it. Best thing is that it runs on most hardware, and boots off a memory stick. It happens to run Reiser FS though, so when a HD dies, it really makes your life hard to get any data back. But thats why you have backups, right?
The problem with RAID6 is that it's not supported by most controllers, especially if it's a year or two old. My new EMC AX4-5 doesn't even support it. I'm doing RAID10 on it, though 6 would have given me some more usable space. I just set it up with two hotspares so I feel alright about the possibility of an error.
Check out my sysadmin blog!
I guess you should be considered a new age Luddite?
Are you the same guy that always waits for SP1 before using any software? I thought so.
RAID is a proven technology and it's use in nearly all business IT systems from big to tiny.
RAID isn't meant as a replacement to backups. It's one PART of the entire system of preventing unnecessary data lose, and more importantly, down time. You can keep on running your server while the failed disk is replaced and rebuilt.
So, while I eat cheeto's and surf Slashdot while that RAID array rebuilds itself, you can go ahead and recover your old data from last night all day long while people bitch at you for not using the technology that's been around since the inception of the hard drive.
If you actually did have the experience you claim, you'd slap yourself for such a stupid fucking post.
- It's not the Macs I hate. It's Digg users. -
You get your first RAID controller from a trusted friend. "Here" he says "try this" and hands you a Mylex board. It has a 64 bit bus and 3 SCSI LVD connectors. Oooh. That looks fast. So you start ebaying drives, cables, adapters, more controllers, the inevitable megawatt power supply and you mess around with raid 1, raid 0 raid 1+0 and raid 5. Suddenly every system falls prey to RAIDMANIA; eventually for yourself you build a system with 3 controllers, with 3 busses each and a drive on each one of 9 busses. With a controller for swap, one for data and one for the system will Windows now be fast? Yeah, sorta. Those drives sure are quiet - from a click-click busy noise perspective, NOT from a "sounds liks a jet airplane when running" perspective. Heat is an issue, too.
http://rs79.vrx.net/works/photoblog/2005/Sep/15/DSCF0007s.jpg
But oh my are the failure modes spectacular.
I just use a laptop now and make several sets of backup DVDs or just copy to spare drives. I love RAID to death. But it's really only marginally worth the effort in the real world. But if you need fast, OMG.
Need Mercedes parts ?
I have read and written about 50 Tb with my single, non-redundant 160 Gb hard drive and I have never lost so much as one bit. This is just a store-bought hard disk without even a hint of longevity. It was the cheapest drive in terms of bytes for the buck - even cheaper than used drives, which tend to be slightly overpriced for people who want to spend next to nothing. I verify all my reads and writes because I handle some large files - no data loss has ever occurred, even though I do not treat my drive lightly. I transport my drive from place to place, run it from various power sources, and use it fearlessly during thunderstorms. Zero information has gone missing.
Therefore, with a RAID system of redundant 50 Tb disks, I could fear nothing.
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
Is this the way it's going to be from now on? Always in full 'crisis" mode? OMG! Y2K! No more IPv4 addresses! Spam! Spam! Spam! Spam! Spam! Great for job security, I guess. This whole fear factor thing is getting out of hand.
What?
See here and Sun's silly response as to why they neglected to provide fsck support for ZFS in their OS.
From a previous mini-thread on the matter. See the full thread for plenty of other examples in defense of fsck utilities on "robust" filesystems.
"ZFS has checksums and will find errors, but only will be able to self-heal the errors in a redundant configuration. On a single disk, ZFS will find the error thanks to checksums but will not be able to recover your data. Since ZFS was mainly designed for systems that will use redundant configurations, it may have sense there, but desktops are not never going to do such things. IMO the ZFS people were a bit elitist here - "let's going to build a filesystem so good that we won't need a fsck". But in the real world you _are_ going to need a fsck util. Only in excepcional and very rare cases, but you're going to need it."
I've set up an OpenSolaris raidz2 array with 9 disks at home for my backup (that is, I backup my main system that is JBOD to the raid array, so it's my extra copy). I'm not that versed in Solaris, and slowly learning (it's built in SMB was much easier for me that SAMBA has been), but I really know Red Hat better. What's coming to replace it (zfs) that's better?? Is it going to work on CENTOS?
Opera, Proxomitron-Grypen,GPG 0x0A1C6EE3
Nobody is saying that RAID is a backup. RAID is there to keep you up and running in a business environment when a drive fails, which is, as the author puts it, inevitable. Then he goes on to statistically prove that, while rebuilding an array of currently relevant size for a large business, as in many TB of data, that you will almost certainly not be able to recover your array to a healthy state because of an unavoidable, highly probable read error on one of your "healthy" disks. Of course you have a fucking backup of your production 12 TB RAID array. He said what he did about tape backups to drive home the point, which is that your shit will be down, out of production, thereby making the fact that you had your data in RAID 5 completely pointless. The author has a good fucking point, RAID 5 is statistically useless when dealing with disks that large.
RAID 0 is only redundancy, and doesn't do anything to protect data in any way beyond what the file system might do. RAID 1, or mirrored drives, is costly (2x drives). RAID5 permits one drive (min 3, but more is desirable and effective) to fail. Above RAID 5, the ACM paper that initially described all this, sayeth not, but it's generally a hot-spare.
We have several T; we back it up on an occasional basis to a slower drive array and it takes time. Subsequent backups are only delta, and it's surprisingly a small amount of data that needs backingup. We keep ISOs ready to go, too, in case we need to burn local laptops or servers-- online-- ready for PxE boot.
If someone steals the SAN, we have a backup and an insurance agent. Fortunately, it's not AIG.
---- Teach Peace. It's Cheaper Than War.
Which is why I only run Windows, so this evil command cannot be run. Problem solved!
By that same logic can we also say that you're kidding yourself to think you're safe if you're using anything short of a nested RAID spread across 3 different locations?
You just got troll'd!
First off, Isn't this story a year+ old? Sheesh.
Second off, if you're worried about URE on X number of disks, what about a single capacitor cooking off on the raid controller? No serious data is stored on a single raid controller system, without good backups or another raid'd system on completely unique hardware. Yes, if you put a lot of disk on one controller and have a failure you have a higher risk of *another* failure. That's why important data doesn't depend on *only* RAID, and why lots of places use mirroring, replication, data shuttling, etc. This isn't new. Most folks that can't afford to rebuild from backups or from a mirror'd remote device also couldn't have used 12TB for anything *but* bulk offline file storage because it's slower than christmas VS a 'real' storage array. Using it for the uber HD DVR? Great. Oh no, you lose X-files's last episodes. This isn't banking data we're talking here.
Prioritize your data. I cannot believe that a home user has 12TB of important stuff. Back up your critical records both on site and off [1]. Back up the important stuff on site with whatever is convenient. Let the rest go hang.
[1] Use DVDs in the unlikely event you have that much critical data. Few home users will have a critical need for that stuff beyond the life of the media. Any that do can copy it over every five years, and take the opportunity to delete the obsolete stuff.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
(though it's been running since '04 without any problems, and my HD health monitors show it in good shape)
Oh man.... you didn't just say that out loud did you???
That sinking feeling deep in your gut when you KNOW you screwed up bad summed up with: {head desk} {head desk}
My data backup scheme is to steganographically embed my entire filesystem into nude pictures of Sarah Palin, and then upload them to usenet.
You see? You see? Your stupid minds! Stupid! Stupid!
This is why you scrub your RAID arrays once a week. If you're using software RAID on Linux, for example:
echo check > /sys/block/md0/md/sync_action
The above will scrub array md0 and initiate sector reallocation if needed. You do this while you have redundancy so the bad data can be recovered. Over time, weak sectors get reallocated from the spare bands, and when you do have a failure the probability of a secondary failure is very low over the interval needed for drive replacement.
Most non-crap hardware controllers also provide this function. Read the documentation.
Can You Say Linux? I Knew That You Could.
Isn't it more cost effective to do RAID 1, with a nightly backup to an external. At least in my home, I do not require mission critical hot-swapping capabilities. Then again I only have 3x 1TB hard drives. Also, after RTFAing the author of the article assumes that an unrecoverable read error corrupts your RAID array. It does not, typically your bad sector gets added to the list and mapped out of being used. Speaking of used, article assumes that entire drive is being used, but if the error on the part of the drive not covered with data, this is also a non issue.
I will not be pushed, filed, stamped, indexed, briefed, debriefed or numbered. My life is my own.
LOL, thanks for saying that.
That sinking feeling deep in your gut when you KNOW you screwed up bad summed up with: {head desk} {head desk}
Wait wait, your RAID 6 vs RAID 1 claim, that's bullshit. Of course RAID 6 can survive the loss of any two disks, it requires you to have at least 4 of them! If you have a RAID 1 array of 4 disks then you can lose any 3 disks. RAID 1 is the best thing from a data integrity perspective, because you can lose any disk in your array but one. It's just awfully inefficient space and performance wise, well, except for reading, then it's great.
You just got troll'd!
I should be safe now, right?
You see? You see? Your stupid minds! Stupid! Stupid!
RAID is not a backup solution.
For those administrators who think it is, you should keep your resume on the array.
I am the unwilling control for my Origin.
The vast majority of Egypts writings were stored on perishable papyrus, not carved or painted on stone. Of all that they ever wrote or stored, we have but the tiniest fraction remaining.
If we lost technology today, there would be nothing left but paper in 20 years. In a thousand, there wouldn't even be much paper.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
btrfs should be an option on Linux, for those who care to go that route
I used to use the old punch card system to backup my data. Sure it takes a while but it was totally worth it... Until one day while attempted to move the many boxes fully of carefully sorted cards I fell down the steps and the cards went everywhere. I learned from that mistake and started writing all everything down on paper... Lot's o' 1's and 0's, my hand hurt.. A lot. But there was a fire at my off site :( sot I had to resort to the ultimate old school back up. A chisel and a rock... a really really big rock.
Eating the brains of your enemies does not make you smarter. But it's still fun.
That's why I chisel all my data (ones and zeros) onto stone tablets. In a few years the pile of stones will be taller than Everest. :)
Redundancy... You keep using that word. I do not think it means what you think it means.
RAID 0, psudo-ironically, is not redundant at all. RAID 1, often called mirroring, are the arrays that are redundant.
And in a thousand years some bearded guy will discover couple of those stones, come down the mountain and will base a religion around it. These things are cyclical.
If you source the original term 'RAID', it goes to an ACM article describing Redundant Arrays of Inexpensive Disks. In RAID 0, which is actually a marketing term, there's striping, but no redundancy that can infer the contents of a missing member of the array. From the perspective of availability, it has none. As you cite, RAID 1 is a mirrored pair, usually the same type of drive, and it also is likely the fastest RAID-- and most expensive in terms of available net data after redundancy for availability. There is also no RAID 6...10, as these are marketing terms, too.
---- Teach Peace. It's Cheaper Than War.
"Shit happens. Pointing out what we all already know doesn't do anything helpful."
Actually, it gives posters like you a chance to remind everyone else that shit happens.
I believe there would be many fewer frustrated/bitter IT workers if more people meditated on the fact that shit just happens. In today's marketplace it is usually IT left holding the bag when things go south anyhow... gotta get acclimated to that and roll on.
Anyhow, I doubt there are many IT veterans not familiar with really expensive, really borked backup systems. Smarter people than me have observed that as technology progresses, existing strategies either age or mature. The ones that age become brittle, and the ones that mature become more robust...
Corporate suits usually insure that both aged and mature technologies will be flogged on long past their rational retirement dates.
And look what happened? Netcraft is already half way to confirming the demise of alt.binaries!
Bio questions? Ask me to start a Q&A journal. Computer analogies available for most topics!
RAID 5 will still be orders of magnitude more reliable than just having a single disk.
No sig today...
Not only are there two parity drives, but the operating system can perform automatic scanning of the drives to ensure that all data and parity disks are correct and silently correct any errors that occur on only one disk. It only takes a few days to scan 12 TB, and if this is done often enough the probability of a two failed disks plus a previously undetected unrecoverable error on a third disk is quite a bit lower than the failure rate for RAID5. RAID5 volumes can be automatically scanned, but if corruption is detected there's no way to know which of the disks was actually incorrect, barring an actual message from the hard disk. Silent corruption is a much bigger enemy of RAID5 than RAID6.
I don't know why the article focuses on RAID5; RAID1 or RAID10 will have exactly the same issues at a slightly lower frequency than RAID5, but more frequently than RAID6.
Ultimately, the solution is simply more redundancy, or more reliable hardware. RAID with 3 parity disks is not much slower than RAID6, and dedicated hardware or increasing CPU speed will take care of that faster than drive speeds increase.
RAID???!!! Aaaaaaah! (Drive dies.)
With 4 drives and raid 6, you get 50% storage and any two drives can fail.
With 4 drives and raid 1(x4) (does anything even support 4x1 mirroring?), you get 25% storage and any 3 drives can fail.
With 4 drives and raid 1+0 (a stripe of two 2-drive mirrors), you get 50% storage, but if the wrong two drives fail, it takes out the array.
With 8 drives and raid 6, you get 75% storage and any two drives can fail.
with 8 drives and raid 1(x8?), you get 12.5% storage and any 7 drives can fail.
with 8 drives and raid 1+0, you'd get 50% storage, and still be screwed if the wrong two drives failed.
Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
Wow. I love your FUD. If you're going to lie, at least make it seem truthful.
Lacking in file system utilities (yes, fsck IS necessary even on healthy filesystems, especially on desktops and portables)
Why no fsck? And if you really feel the need to do something:
License-incompatible with anything worth running it on, other than Solaris itself... which is NOT worth running (see #1 above)
What you mean to say is "Some Operating Systems whose merits can be debated are license incompatible with the license of ZFS." FreeBSD can implement ZFS. Why can't Linux? Because of its license, not that of ZFS.
"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman
article from 2006: http://lwn.net/Articles/190222/
However, when adding that much data into one photo, there are a few, slightly noticeable changes.
See my journal for slashdot ID's by year. Mine created in 2005. http://slashdot.org/journal/289875/slashdot-ids-by-year
You DID see my previous reply, right? It has plenty (hundreds) of anecdotal references as to why fsck is absolutely required, especially when dealing with redundant disks and larger filesystems.
Sure, as long as you consider that the CDDL takes rights away that the GPL grants. See here for a MUCH better explanation than I can do justice with.
Lets hope he discovers some porn this time...
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
That doesn't work for me. Try
I have to say, the ZFS folks have convinced me. There are simply too many places where bit rot can creep in these days even when the drive itself is perfect. The fact that the drive is not perfect just puts a big exclamation point on the issue. Add other problems into the fray, such as phantom writes (which have also been demonstrated to occur), and it gets very scary very quickly.
I don't agree with ZFS's race-to-root block updating scheme for filesystem integrity but I do agree with the necessity of not completely trusting the block storage subsystem and of building checks into the filesystem data structures themselves.
Even more specifically, if one is managing very large amounts of data one needs a way to validate that the filesystem contains what it is supposed to contain. It simply isn't possible to do that with storage-system logic. The filesystem itself must contain sufficient information to make validation possible. The filesystem itself must contain CRCs and hierarchical validation mechanisms to have a proper end-to-end check. I plan on making some adjustments to HAMMER to fix some holes in validation checking that I missed in the first round.
-Matt
I vote for silver halide contact prints of data encoded on high-res inkjet transparencies stored in a fireproof box.
You DID see my previous reply, right?
Yes, I did. It quotes an explanation that you can only fix errors in redundant configuration. Considering that the whole basis for this discussion is RAID-5, I think that's a feasible thing. However, metadata is written in multiple places, so if you want a ZFS fsck to correct a corrupted superblock, it's kinda silly since that superblock is written in multiple places anyway. Also, you can tell ZFS to do a manual scrub (as I shown) which has the advantage of running while the array is running so you can cron script it and still keep the array available.
I'm not going to argue license points. The fact is that ZFS is under an open source license and so is Linux. Sun had every right to use their own license.
"Nature doesn't care how smart you are. You can still be wrong." - Richard Feynman
The Egyptians found a way to preserve their message over thousands of years, surely we can come up with something. :)
And they would have saved future generations from vast amounts of confusion and effort, if they'd only been a little more diligent backing up their pyramid construction HOWTO files.
Given that RAID1 is pretty much for two disks only
No, RAID1 will work with any even number of disks. If you have an array with 4 drives in, you've got 2 pairs of mirrors. In the dual-failure case, you have two scenarios:
1) 2 "paired" drives fail. The data on that half is gone, but the other half is still mirrored, so you've only lost half your data.
2) 2 "unpaired" drives fail - you survived, you just don't have mirrors any more.
I fail to see how any form of RAID will prevent data loss when you lose two disks.
Broadly speaking, it's down to having better data distribution. You've got an extra set of parity data to rebuild the array from. Wikipedia is your friend. As the number of drives in the array increases, RAID6 gets even more attractive than RAID1 (From an cost & data integrity POV, at least - but RAID1 will generally give you better performance)
Of course, it's theoretically possible to treat half or a RAID1 pair as a "hot backup" - just pull it out, plug in a fresh drive, and cart it off to use as your off-site. I wouldn't recommend that as your primary backup strategy though...
Not everything that can be measured matters; Not everything that matters can be measured.
You leave RMS out of this!
Can tolerate a complete drive failure + hundreds of unrecoverable reads per drive on the two remaining drives. The larger the disk, the less likely that both remaining drives will fail on the same sector, so larger drives are an advantage, not a disadvantage, compared to data split across drives that has to be "rebuilt" from the parity info ...
I concur but in a less-condescending way: if there are any people that are already on the way towards building a giant RAID, can someone please give this a try to see what actually happens? (That is, fill the drives with test data, make the array rebuild once or twice, and see if bad stuff happens.) The article is based off the URE spec, but maybe real-world URE rate is lower.
But to be fair to the article, note the date on the post: July 18th, 2007. It wasn't "trivially testable" at the time; terabyte drives weren't exactly cheap yet.
Maybe you missed the bit where I said "arrays of equal sizes", thus causing you to spout bullshit?
It smells to me like you're talking about a 4-way mirror; I don't see how a 4-way mirror gives you the same storage ("equal size") as a 4-drive RAID6 configuration. (and, as has been pointed out elsewhere, I don't think anything actually supports 4-way mirrors)
If you've got 4x1TB drives, you can have a 2TB RAID1 config, or a 2TB RAID6 config. The RAID1 config will be faster, sure - but in the face of 2 drive failures, there's a good chance you've lost half your data (and if your file system spans the whole array, that could potentially mean losing all of it). With RAID6, you'll still have it all there.
Not everything that can be measured matters; Not everything that matters can be measured.
if your first step is to use the largest drive available in a performance raid, then your first step is probably a misstep.
it's better to use those raids with the smaller-yet-faster drives, ie the 10k+ rpm raptors or perhaps some yet to be seen faster better (stronger harder?) SSD's and use the latest ultra large HDDs in a simple redundant raid setup for backups.
multi raids ftw
sigs... don't talk to me about sigs....
The theory of more disks = more disk failures sounds totally logical. But in practice it does not work at all. For 5 years I ran 5 servers with various IDE RAID5 and RAID1 solutions (promise, highpoint). There was a total of about 20 IDE disks. I see a disk failure about once every two months. About 3 years ago I added a Dell poweredge 2600 running TWO SCSI U160 disks on a SCSI RAID1. A single disk fails about three times a year on the dell. I found a cheap NetApp F760 NAS. It has three disk shelves of 72gb fiber channel drives for a total of 28 disks (2TB) making up 4 RAID 4 volumes. I've had this for a little over a year running ISCSI for database servers and have yet to see a single disk failure. NetApp uses a technology, WAFL, that is exactly the same as ZFS and was in fact has been in production for more than a decade. But I digress.
My point here is, the number of disk failures in a particular IT system cannot be generalized upon. There is no global rule for disk failures. My guess is there are so many different reasons for failure that it is practically impossible to predict how a system will behave without looking at the system itself, not at the cloud of disk failures. In my case I had a bunch of IDE disks failing at one rate, a bunch of SCSI U160's failing even more frequently, and a whole lot more fiber channel disks that have yet to fail at all!
Also the whole premise of the article is emphasizing on the failure of RAID5 - then says enterprises won't be affected - but what typical home user even uses a RAID 5? If I were going to give my mother a RAID it would be a RAID 1, not a RAID 5. Furthermore the typical user doesn't even know what RAID is! The typical user still thinks his single HDD is safe! We (geeks) have a long way to go in educating the typical user before we get to the RAID5 is unsafe part, which is untrue anyway. A good disk controller will recover the data in your RAID 5, even with a URE of 10^14. As with most generalizations and statistics, this one is clearly false. I'm sure Seagate loves that Samsung's failure rate effects their drive's failure rate somehow! The title would be better phrased "Crappy disks and crappy disk controllers are crappy". Hmmmm, yes, I like that much better. The previous title was boring and pedantic.
Assuming that you'll have another drive fail before it can rebuild is pretty alarmist. Sure, it can happen... I've *seen* it happen. But it's not the norm.
Most people who run RAID5 are running pretty poor hardware implementations. But on a board with Raptors, via multiple (quality) SATA controllers each connected via PCI-E (avoiding bus contention), I've seen RAID5 rebuilds of over 200 MB/sec. That's a pretty far cry from something like 8 drives hooked to a 32-bit, 33MHz PCI SX8, and getting a tenth of that.
Oh, you're not stuck, you're just unable to let go of the onion rings.
alias rm='rm -i'
:)
Isn't ZFS a filesystem? Why would I care about what filesystem I am using when I am trying to protect my data from disk failures?
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
First they say that these new improved lower failure-rate drives along with bigger disks are going to kill RAID5. Then, they try to tell us that if 1 drive failed it's likely you will lose the array due to an unrecoverable error on the remaining array drives. So which is it? Are hard drives becoming more reliable or more error prone? You can't have it both ways. This sounds like complete crap to me. I'll still be using them at home, and probably 5 years from now too.
what is an Everest pile in, oh say Libraries of Congress?
Rule of Slashdot #0: You and people like you are not representative of the larger population. - A.C.
RAID-5 is UPTIME protection, not DATA protection. What I mean by that is if you have a non-redundant configuration (single disk, RAID-0, JBOD) and a disk fails, your shit is down. You can't use your system any more until you get the problem fixed. Then, it is a rebuild situation. So you can be facing a few days of downtime, maybe more. That can be real annoying.
Well RAID-1 and RAID-5 solve that to a large degree. Now a disk failure doesn't cause a system failure. A disk fails, and provided you can get a replacement before another one fails, and it is a good bet you can, and there is no problem. You continue operating, maybe with a bit of a speed reduction but that's all.
That's why I have a RAID-5 at home. Used to run a RAID-0, since I keep good backups. Well, the a disk failed on me. Didn't lose any data, but I was sitting around with no computer to use, and it was Friday evening which meant that any order from an online shop wouldn't be there till Tuesday even if ordered with fast shipping. I elected to go buy disks locally, despite the premium charged by CompUSA, but this time got enough for a RAID-5.
The issue was never data loss, the issue was that my system was out and would remain so for quite awhile. If that is an issue for you, then RAID is a good answer (since drives are probably the sole most likely thing to fail other than fans). However backups are a completely different matter.
This is mitigated quite a bit by hardware RAID controllers, SMART, and data validation.
I've personally never experienced this "RAID decay" you speak of in the last 15 years of working with storage systems. And some of the arrays our customers have running consist of very old disks and controllers.
- It's not the Macs I hate. It's Digg users. -
Indeed.
Personally, I believe the correct answer to ensuring data recoverability is RAID together with real-time replication. You can usually accomplish this with a very acceptable price-point.
RAID is important to prevent down time due to a single disk failure, and replication prevents loss of data due to an array failure.
Personally I think RAID5/6 will be around for a very long time because it works and there's actually people out there that use it correctly (versus SO MANY people on Slashdot, apparently.)
Gosh, there's even one guy a few posts up that's claiming "The biggest problem with RAID is DECAY." Holy crap. Any RAID card made in the last 10 years will periodically scrub the disks and make sure the parity is correct - or else it will mark a disk as bad.
- It's not the Macs I hate. It's Digg users. -
The problem with anecdotal references is exactly that, they are anecdotal. You can trawl through any number of Slashdot discussions to find opinions that support, oppose, ridicule, pontificate, or elevate to religion any point of your choosing.
When it comes to making engineering decisions these anecdotal references have zero value.
Newsflash:
RAID is NOT backups.
It's not a perfect backup. It's not an imperfect backup. It's not a backup.
Apparently it isn't completely useless to point this out, because not everyone seems to know it.
Why does a guy who says there's no point in pointing out that nothing is perfect spend a paragraph explaining why nothing is perfect?
It's pretty shocking to me that your post is considered insightful.
http://lkml.org/lkml/2005/8/20/95
oops, missed funny and hit overrated.
Sorry about that. To bad this will remove some good mods up above.
It was me, I did it, I moved your cheese
Well, Windows does. Taking a snapshot of NTFS, even on a heavily used 1TB+ file server, takes only a few seconds, and under normal operation the file system is still fast.
NTFS is actually a pretty good file system. It's probably because it was originally designed by IBM.
- It's not the Macs I hate. It's Digg users. -
"This time?"
Ah, I see you've never read "Song of Songs"
Raid 1+0 on the machine - We want a safety net to deal with minor failures and keep the machine online, and the drives need to be fast.
Automatic backups by the hosting company, to local and offsite tape, twice a day, I'm told verified at both locations after written to tape against the original snapshot.
Automatic backups synced to our company office, once a day, verified against md5 and sha hashs of the data when the original backup was created after it is at the office.
Copies brought offsite to a safety deposit box when ever I feel like doing, just in case everything else fails.
Are we safe from all harm? No theres a good chance that a nuclear detonation near our office which is near the primary datacenter will get most of our backups instantly, and possibly get the secondary backups provided by the hosting provider. However, as I told the president of our company, if that happens, I'm really not going to give a damn about restoring our data.
And ... the whole thing was screwed because no one noticed some of our reports were acting wonky in time to keep the last good backup (last snapshot of the database before the server or OS screwed up the file itself) from cycling out. This was the time when that random old backup laying in the safety deposit box saved us.
When that event occurred, since we had NEVER gone to the safety deposit box before, no one even thought about it when we were in the initial panic failure mode. It was actually a few days later when discussing the 'data loss' that the guy who actually controls the safety deposit box said 'What about the backups I take offsite?' and finally my obviously slow brain kicked in and said 'duh'.
For the record, I have since turned in my sysadmin card for obvious and become a developer, and I no longer believe in backups, thats what the sysadmin is for! Now I make the problem worse with bad code rather than being responsible for and protecting someone elses data.
Sorry, what was my point again?
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
...except that 100% of them were posted by professional system administrators who are paid to support and maintain systems using these technologies. None of those were from any posts on Slashdot.
Now, whether any of those career sysadmins are also posting on Slashdot under different names from the ones they used on Google Groups, I don't know.
Just an FYI, the default behavior in Debian is to scrub all of your software RAID arrays monthly via a cronjob that comes with the mdadm package (see the checkarray script).
I wouldn't be surprised if other distros do something similar.
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
A Black Swan is an event that is highly improbably, but statistically probable.
Yes, it is possible for a drive in a RAID 5 array to become absolutely inoperable, and for one of the other drives to have a read failure at the same time. This is highly unlikely though, and is not the Black Swan. The math use to calculate the likelihood of these two events occurring at the same time is faulty. The MTBF metric for hard drives is measured in 'soft failures'; this is very different from a 'hard failure'.
The difference between the two types of failures is that a soft failure, while a serious error, is something that the controlling operating system can work around if it detects it. It is extremely unlikely that a hard drive will exhibit a hard failure without having several soft failures first. It is even more unlikely that two drives in the same array will exhibit a hard failure within the length of time it takes to rebuild the array. In my experience, it is more likely that the software controlling the array will run into a bug rebuilding the array. I've seen this with several consumer-grade RAID controllers.
The true Black Swan is when a disk in the array catches fire, or does something equally as destructive to the entire array.
To echo other people's points, RAID increases availability, but only an off-site backup solves the data retention problem.
...is rising to the level of complexity of a simple operating system. They do a lot of very smart things in there to catch problems before they become unrecoverable, and they even report them to you via SMART so you can tell when they're at an increased risk of failure.
The catch is, there are several different ways to optimize firmware. Drives intended to be used in RAID arrays have different firmware from drives intended for desktop or laptop use. If you use desktop drives in a large RAID array, with a rather fault-intolerant parity RAID implementation, YOU ARE AN IDIOT AND DESERVE WHAT YOU GET.
The hard disk vendors aren't idiots. They make nice, fat margins on drives that get used primarily in RAID 5 arrays, and they're not about to let that revenue stream dry up. You'll have to pay a little more for the RAID-optimized drives, but the price gap between bargain SATA and RAID SATA is much smaller than the gap between IDE and SCSI, so it's still worth it to an awful lot of people.
There's no failure quite as dissatisfying as a complete and total solution to the wrong problem.
I think anyone talking about HDDs here wether SCSI, ATA, Fiber, SATA are all talking about 3.5" form factor.
If you're willing to negate performance, storage size and space, are the 2.5" or 1.8" Hard drives less *prone* to failure than the 3.5" ones? Say I want to be reckless, back up to a few DVDs and smaller format HDDs - different batches maybe different manufactureres. Unplug them onece copied.
Personally, I have well under > 10 GB of REALLY essential work, photos, etc. I mention this at least becuase a drive of equal or grater capacity in 1.8" FF would be affordable. The rest I could care less about. So how do I best protect it?
As the article mentions ZFS, would standardizing to one file format - Windows, Mac, Unix, Linux be a good idea with ZFS? I had a Mac and there's all kinds of crap software I had to install to get Windows to recognize the drives. Apple is considering ZFS. Linux its an option(?) and the BSDs (are working on it?).
No method is foolproof, especially when it's bound to be boring as hell, and you've got an inevitable human factor. You get lazy moving the tapes offsite, you put off fixing a dead drive because there are 4 others, you wipe your main partition upgrading your distro and forget that your CRON rsync script uses the handy --delete flag, and BOOM wipes out your backup.
Jesus Christ, you must be one unlucky soul. Do you live your entire life in a worst-case scenario?
The system that I use for data storage is as follows:
This system may not be foolproof (what is?), but it is pretty frickin' safe, and costs me roughly $3 or $4 per month. Not too shabby for what I would consider to be a fairly robust backup system for a home user.
I suppose the biggest challenge is deciding what goes into rsnapshot. If my RAID array suffered a massive failure, I would definitely lose data. But this is mostly video content, and really, if I lose my mythtv shows, it is not exactly as catastrophic as if I lost, say, my quickbooks data.
There are a lot of things that keep me awake at night, but loss of important data is not one of them.
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
Ummm, brb, quickly looks at his backup script
May the source be with you!
>>RAID 0 is only redundancy, and doesn't do anything to protect data in any way beyond what the file system might do.
??
RAID0 is striping. It's not only not redundant, but it lowers your MTBF on your aggregate drive.
>>If someone steals the SAN, we have a backup and an insurance agent. Fortunately, it's not AIG.
Yeah, theft is really my biggest worry. With two drives getting backups (one inside the machine, one external), a thief coming in and stealing the lot is my worst case scenario. I can't automatically backup to the FTP, so I have to manually remember to do it from time to time, and I'm pretty lazy about that kind of stuff.
OK. I'll bite... " it is almost certain it will see an [unrecoverable read error]." What? Like most stats the 12 terabytes failure rate doesn't mean anything. If it were true, then the servers I have running 4 terabyte arrays would be failing all the time.
Whats all this talk about DVD? Why not sure Blu-Ray? whats it got 50 gigs? Little better...240 Disks for the example above where someone said 1600+ dvds. Also Blu-Ray today have 2 layers. In the future i am sure that will increase, or the next optical medium will be even more condensed. That being said i still believe that SSD is the way of the future.
-EL
... or rather, haven't noticed, the fact that nobody has realize this article was written in Mid-2007?
Why has it taken this long to hit slashdot? Is there a time warp here...
I work for a storage company. And have been in the storage industry for abut 8 years now. And here is the problem with this whole story. "...data from the failed drive, it is almost certain it will see an [unrecoverable read error]." Why is this almost certain? I haven't seen this happen...ever...to the extent that the RAID set wouldn't rebuild. That's, like, the job of RAID 5. Yes, it will take longer to rebuild 2TB drives. And people are building faster RAID contollers. You can always do RAID 1 (which isn't dead, but sure is damn expensive). And with the advent of SSDs...and RAID controllers with a shit load of write cache...you can do what TMS does and RAID 1 from your 128GB SSDs (however many you may have in your array) to a set of SATA disks (which can themselves be mirrored). This way you have multiple failure domains. And although you lose performance, you don't lose DATA. There are always smart people out there to solve such problems...but I see that it's quite sexy to just call something dead. I like the term (from Princess Bride) "Mostly Dead".
The article says "RAID 5 Stops working" blah blah blah. That's not the case. The purpose of RAID 5, as has been mentioned several times in the comments, is to give you MORE time to recover from a failed drive.
No matter what size your array is, having more time to recover from any failure is invaluable. Therefore claiming that because bigger drives are available, does NOT invalidate the value that RAID5 has to an organisation.
Having said that, RAID5 is NEVER considered the ultimate means of protecting your data. If you think it is, then think again. You must always have multiple copies of your data, in multiple locations, and preferably on a magnetic media for long term storage if necessary.
I have a collection of servers, all with various amounts and types of data. These servers have RAID arrays, some mirrored drives only, others with Mirrors and RAID5 for more important data.
Each are incrementally backed up hourly to a "backup server" on to another RAID5 array.
A daily backup is also taken, on to another path.
Each night, this backup server's data is written to an LTO4 tape, and the next morning it is taken off-site.
We also keep monthly tapes on-site, and off-site.
At any one point, I can recover data to any server from up to an hour ago, as of last night from either yesterday's daily HDD backup, or last night's tape. Or at the end of every month for as long as we've had this backup strategy.
This is the best backup strategy I could come up with the budget I had, and I don't pretend that it's the best in the world.
But simply relying on RAID 5 and nothing else, you're asking for serious trouble.
That doesn't work for me. Try
hell, if you want to lose data, you've gotta at LEAST use dd. rm is just removing file handles, all your data is fine, you just cant access it. run
(or whatever disk you want to lose) and then see how many data recovery places will turn you away. the level of data recovery available to the public is pretty crappy, there's a guy offering a reasonably big prize to any data recovery company (or anyone at all i guess) who can recover data from a disk he zero'd with dd and hasnt had any takers yet. i wish i could find the link
The only solution is to regularly read everything:
The chance of avoiding double errors in the form of unreadable sectors during rebuild about doubles each time you halve the time between full reads of all sectors on a drive. (True to about weekly full reads.)
This is because a full read will allow each drive in the array to discover sectors that are becoming iffy (soft/recoverable read errors) and then remap them.
See lwn.net for a discussion and links to some good papers.
Terje
"almost all programming can be viewed as an exercise in caching"
I've only seen Raid Ant Baits III. Where can I pick up some Raid 5?
Well played you Magnificent Bastard (TM) :)
Nothing is perfect, but when you use the word perfect in a trademarked name is sounds really good.
#%^@%!#$!!!! The second one works!!
Mod parent up.
I've seen 3 cases of data loss this year in 6 drive RAID5 with 750GB or 1TB drives. One drive failed, and a second drive failed completely during the rebuild or there was a read error on another drive during rebuild. That is not a fluke given that rebuilding can take days if the system is under high load.
Scrubbing costs too much performance if you want it to be useful. We're using RAID6 in newer products.
thegodmovie.com - watch it
I have been saved by RAID5 before, when one of my drives died a horrible death. A few hours later, I was back up and running. I am very glad I did that.
However, I once accidentally did
chown -R user:users /
instead of on *. the RAID didn't help with that at all, and I consider it quite lucky that I was able to recover from it at all.
Uhm, my computer is pretty stable if I keep it at room temperature, low humidity, and unplugged.
And we'd only need one of 'my computer' to survive to read almost every hard disk currently manufactured. Only thing mine doesn't have is a software emulated raid controller and SCSI backplane. So I may not be able to recover but is easy.
Tapes are overrated. As long as you are careful not to drop them, HDDs are pretty decent.
;).
AU$1000 buys you about five 1TB SATA drives.
So AU$3K can buy you: 5 x daily 1TB drives, 5 weekly, and 5 month.
AU$1K is enough to buy a "build it yourself from decent parts" server - decent power supply, 4 x SATA, 2 x 1Gbps NICs, a core 2 duo (so you can do gzip to two "backup media" drives at the same time ), 2x 1GB RAM (not important but hey it's cheap) and a UPS+power filter, and special removable caddies for those HDDs (make sure they won't overheat the HDDs).
With tape drives, if there is new higher capacity tape technology, you will need to buy a new very _expensive_ tape drive to take advantage of it.
With HDDs you get a whole drive mechanism along with your "media".
If in 5 years time if there are cheap 10TB hard drives, you just buy them and use them.
Whereas if in 5 years time there are cheap 10TB LTOx tapes, you will need an expensive LTOx drive.
For the past decade or so that has been the trend.
I bet the SATA interface will be around for a long time, so in 5 years you'd still be able to read _most_ of your backups. Even if the bearings seize up due to age and lack of use, the data is likely to still be on the platter.
Now if you require _hundreds_ of tapes worth of storage, then using tapes as your "media" may become more viable than using HDDs.
But I get the impression you're not facing that scenario, so HDDs should be fine.
There are special caddies for HDDs, so that you can plug and unplug them.
So you could buy a Backup Server with a gigabit NIC or two, install the caddies, insert the drives, and then do backups.
Seems doable to me. After all AU$4000 is a lot of money in Malaysia where I am (we're the cheap labour, "low brains" country).
So, how much are they paying you in Australia to solve problems like this?
If a sector fails during an array rebuild in RAID-5 (after a complete drive failure), you lose one stripe's worth of data (ex., 64 kB x N-1, where N is the number of drives), you don't lose the entire array. Following the article author's logic, if you have a read error in a single drive, then all the data on the drive is lost.
It's amazing that ZDNet would pay someone this clueless to write an article about this subject, it's amazing they published it without any verification, it's amazing that the article is still online (and essentially uncorrected) after almost one year, and it's even more amazing that someone decided to post this on Slashdot.
NTFS is actually a pretty good file system. It's probably because it was originally designed by IBM.
NTFS was designed and built in-house at Microsoft specifically for Windows NT.
I think that's a little harsh. Most peoples experiences of RAID5 are based around rubbish RAID controllers, which to be honest are possibly in the majority.
RAID5 is a compromise, same as any other RAID level. With a good battery backed controller (as opposed to a bad battery backed controller, or one without) it's reasonably fast and reasonably safe. For some definition of reasonably. And the real winner, is it doesn't waste huge amounts of disk, so you can convince people to use it.
jh
Uhm, my computer is pretty stable if I keep it at room temperature, low humidity, and unplugged.
Pompeii and Herculaneum probably had an environment that were pretty conducive to working electronics in their heyday too.
And we'd only need one of 'my computer' to survive to read almost every hard disk currently manufactured.
You also need to have an idea about how it works and how to make it run, otherwise it's possible to damage it beyond repair in the process of learning about it. It'd suck if they applied 500 VAC at 120 cycles and blew up the power supply, or all the capacitors in the machine had dried out and were non-functional, or it had suffered some other kind of irreparable physical damage. Long-term preservation of information isn't something trivial to achieve, and I think the parent poster is correct in pointing out that it's something we should really start thinking about.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
And in a thousand years some bearded guy will discover couple of those stones, come down the mountain and will base a religion around it.
:-)
I don't think RMS will be around then, and the cult of the GPL won't either.
Please stand clear of the doors, por favor mantenganse alejado de las puertas
Goodness, even the summary says "didn't back up? bummer!". Yes, we all know RAID only hedges against hardware failure. The point of this whole exercise is that RAID 5 doesn't even adequately help with hardware failures once data per drive grows large enough.
URE stands for uncorrectable read error, so corrections via ECC should already be factored into that spec.
Why are you describing ZFS as the only option, are you working for Sun?
Real-time remove replication and distributed storage are real alternative to RAID 5 or 6.
No need to use Solaris. There's a ton of very efficient tools to do that on Linux, like the excellent Zumastor project.
{{.sig}}
It's a metaphor. Other programs can do the deleting for you (remember the itunes installer that deleted the contents of your HDD?). Or you can instead just do something super smart like "I'll write a script to rename all these .jpgs to .jpegs" and accidentally write a script that renames them all to the same thing, thus deleting most of them.
If you think your data is safe because you're using RAID, you're setting yourself up to fall.
http://lkml.org/lkml/2005/8/20/95
Thanks for the real-world data! Out of curiosity, what was the claimed URE rate on those drives?
The reason I ask is: It looks like your observed rate is better than 1 in 10^15 with 96% confidence, and probably nearer to 10^16. Getting an extra order of magnitude (i.e. from enterprise drives, which are 10^15) would be pretty impressive. But it would be quite astonishing if they claimed only 10^14 and actually gave 10^16 instead.
Whoops, technically it seems the "unrecoverable" expansion is more popular than "uncorrectable". But in either case, ECC should already be factored in.
I'm glad the 'not being a moron' thing worked out for you. But, what would you suggest to those in the audience that cannot claim the same. :-)
OS X?
Speaking of redundancy.
Why are you repeating the GP's post to itself?
You cant modify your own home directory?
"Welcome to our world. We are the wasted youth. And we are the future too." Yes, I know these are stupid lyrics.
That's why it is called RAID Level Zero: Zero Redundancy.
Firstly, the core determinants of HDD failures are:
The studies by CMU and Google are not broken down at the application level, i.e. - what purpose were the HDDs serving. For example an HDD serving as an archive will perform differently from an HDD doing constant defragmentation, for the sake of example, or other read/write intensive functions as compared to archiving.
Such a mashing is therefore "unfair". But ok, lets take the numbers produced by CMU and Google. Their rates of failure does seem to threaten RAID 5's (and other RAIDs) reliability with increasing disk sizes. This issue is immediately resolved by the RAID controller - but yes it means an extra performance penalty for the RAID implementation.
As such, RAID 5 will not die. Its the RAID controllers that need to be more intelligent, at the expense of performance.
I actually did this once - just a simple case of tab-whoring and doing things too fast for my own good (it was rm -rf . in my case, in my home directory).
It was a good experience in that ever since that point I've actually been running daily incremental and weekly full backups of everything important to somewhere a normal user has no read/write access. And by 'important', I mean documents etc., not the 500GB of media. It's all about risk vs. cost, and I just don't have anywhere to backup that much data to, when most of it can be reacquired. In fact, I just back up a list of my media instead to make that process easier.
There's a lot of 'ideal solution' stuff being thrown around here, but the fact is that it's not practical for 99.9% of people. Most home user data is not worth the £10000 an ideal backup solution would cost.
I swear we should be allowed to give mod points to sigs... "-1, Offtopic"
...but nobody *forces* you to wait with the replacement of a harddrive until it breaks. You may do it earlier and regularly.
Ohne Worte
Yeah the oldies were more reliable. But... I did some archaeology on an old 386 desktop computer and the HDD still had the price tag on it. $630 for a 120MB drive!!! Nowadays we expect to get a whole home computer at that price and with all the latest. Obviously the production has become more efficient in terms of price but we make computers more and more as disposable and so we get what we pay for.
Stupidity is its own reward.
Oops,selected wrong moderation option. This replay is to wipe that moderation.
Redundancy... You keep using that word. I do not think it means what you think it means.
RAID 0, psudo-ironically, is not redundant at all. RAID 1, often called mirroring, are the arrays that are redundant.
The R iin raid 0 stands for "Risky"
I'll also tell you it's pretty serious when you see a Fedex plane with a cargo fire in the evening news and later find out that your shipment was on it. And oh... it's not covered because it was considered an 'Act of God'.
There is a very nice program called rdiff-backup, useful for cheap disk-to-disk backups. It incorporates incremental changes at your current backup, making it equal to the more recent version and keeping incremental deltas that you can apply to get the old versions (the reverse of incremental backups). Of course it isn't as reliable as proper disk-to-tape backups.
Now, about your situation, I bet all those terabytes don't have the same importance for the company. Are you sure that you can't provide extra protection for some of the data?
Rethinking email
Even if it was feasible to buy all these hard drives or a tape drive, the amount of time it would take to properly do all these back-ups on a useful time scale seems to be beyond the reach of the typical user. Even power users do other things in their lives than worry about their computers. I can't see somebody with enough free time to make CD or DVD or tape backups every so often. And if you are copying your whole 1+ TB drive then it would take forever. It may just be that because I'm a college student I have less time than most people with normal jobs, but I see my dad come home late from work almost every day, and then he's just too tired to want to do anything else. So maybe this whole discussion just becomes irrelevant because not too many people realistically have the time to be able to do all this backing up, and would rather just take the risk of running a RAID setup.
Too expensive? Really?
I pay about 13 USD/month for 90 gb of images at S3. I'm a hobby photographer, and those 90 gbs are primarily RAWs along with a few PSDs etc. In case you're wondering why I have 90 gbs of images..
I don't bother protecting my music and my movies etc. I've got lots of legal DVDs and CDs.. And my mailbox is IMAP - so no problem there. My contact list is synced to my phone both at home and at work. So, for 13 USD/month I've insured myself against losing 5 years of my life.
Stop the brainwash
If you've got 4x1TB drives, you can have a 2TB RAID1 config
No you don't, you blithering twit. That's just not RAID 1 you're talking about, that's something else, like RAID 0+1/1+0. You cannot have a 2 TB RAID 1 array with 1 TB disks, only a 1 TB array. Do some reading and shut the fuck up.
Oh, and while we're at it, there's a space between numbers and their units (it's "1 TB" not "1TB") and it's "RAID 1" not "RAID1".
You just got troll'd!
The article is a year old, and basicly just rehashes part of this paper from 2004...
Seriously...look it up. If you've got only 3 drives it's free, but go ahead and pay for the normal version. Mine works amazingly well; I have about 2.5GB of data including my DVD and CD collections. It runs great, is expandable, and fails more gracefully than RAID5. If one drive fails, you rebuild; if two drive fail you'll only lose one drive worth of data. Bad, but not as bad as losing the whole array. I haven't kept up with the software (it works, therefore I don't mess with it), but there were plans a few months ago to implement a hot-spare option in case of drive failure.
You'll need a separate PC box, but for just a few drives they can be had fairly cheaply.
Is it just my observation, or are there way too many stupid people in the world?
I really hate that this particular bit of FUD keeps floating around. Seriously - look at the ZFS design, it has so much integrity and self checking built into it that the only two cases for errors are:
1. Data corruption on disk due to a corrupted write or bit rot. Depending on the case it may be recoverable from another location in the pool or not. If the fault is in the uberblock you are screwed, but same as any other filesystem where its root data is stuffed. In the not repairable case it simply gives you an IO error and flags an FMA event that this file/object is screwed. In any filesystem that is a blow away and restore event. Same as any other filesytem you might be able to rm and save some data, but generally its gone and recover time.
2. A ZFS software failure of some sort - bad metadata written to disk with valid checksums for example. In this case the actual fix is to the ZFS module and application of a patch, and the short term fix once the bug is discovered is probably a repair tool from Sun, unless the fix involves ZFS being updated so it is able to self correct this fault.
Which brings us to fsck. Everything that fsck does is based on previously encountered behaviour with a known pathology and possible method for repair. Typically this is addressing known forms of corruption caused by the filesystem itself and on disk inconsistancy (writes lost at poweroff etc). Sure XFS had no tool initially, but once the problems were found one was written. UFS has 20 years to experience to put into fsck.
Many of these cases are allready handled by the inherent design of ZFS - self checking data structures, checksums throughout, copy-on-write disk updates.
So, unless you can find an inherent flaw in the design of ZFS, or a bug in the code you have no case to need a fsck utility more advanced than zfs scrub. It may be the case that one day there will be a zfs repair command - but until such a need arises why be so paranoid?
Most ZFS "oh fuck" events I have heard of have either been hardware issues (expensive RAID arrays doing silent data corruption) or some extreme cases of testing abuse. In my professional experience with it over the last 2 years I have only see one issue caused by bad non-ECC memory on my home PC.
"If everybody is thinking alike, somebody isn't thinking" - Gen. George S. Patton
On Slashdot, you don't need to make excuses for the size of your porn archive. We understand.
So you have a very large overhead on your disk use - you get one disk worth of storage, using two disks for it.
However you also get a lot of resilience - you can afford to lose one drive, with no overhead/data loss.
And you get good read performance - as you surmise, if it can be read from two places, then you read from both drives at once, for a faster overall transfer. 5 copies of the drive, means ... well, not _quite_ 5x transfer rate, but very nearly. (You'll probably hit problems eleswhere in your system, as the combined drive transfer rate exceeds system bus throughput)
It doesn't help for write performance though, because you still have to write that file to every drive.
The article is talking about RAID 5 though. That doesn't actually write data in multiple locations in teh same sense. It's a bit faster, because a file might have 100k on one disk, 100k on the next, and 100k on the next and so forth - meaning you can read the whole file a bit faster, but it's not a linear speed increase. It can also service multiple file requests faster, as a randomly distributed data request will be using all the drives at ones.
In the post the writer states that SATA drives
have a URE rate of approx. 12 Terabytes. I would think that the URE rate applies to a single drive. Meaning that on any given drive you can expect for every 12 TB you will get an error. The key here is *for any given drive*. Therefore isn't it erroneous to apply that error rate across multiple drives? Every drive resets the chances back to 12 TB limit. Since drives are still limited to about 1-2 TB in size I would think this issue is years off. When drives are 12 TB then there will be issues.
Can someone tell me if I am wrong?
When I last priced it out for the amount of data I have it was about 50 USD a month, and being in Canada that cost has been higher and higher recently. Plus, with my current setup of Bacula and DVD-RWs there's no monthly cost whatsoever, and I get much faster recovery times. Recently when my wife's laptop drive crashed she was back up and running a couple hours after I bought a replacement disk. With S3 I'd have to wait while my DSL connection downloaded over 40 gigabytes. Add in two more computers being backed up and that's a very large cost in time when recovering (or even backing up).
Again though I'm talking about an ideal solution for me.
Yes, that's what time machine is for. Sadly, my mac is the best backed up machine here. I have an external seagate drive hooked up with time machine and average around a month of backup points. I also burn things on DVD twice a year I can't live without like my iTunes collection. I really wish blu-ray would pick up on Macs for backup purposes. I could backup my iTunes with 3 50GB BD discs. 135GB of data to backup on 8GB DVDs?
Tapes are cost prohibitive and optical hasn't kept up with hard drive capacity. I remember when I could backup my whole computer on 2 CDs. Now, even with BD I'd need 5 discs.
Optical discs have their own problems, but I like to have backups on at least two different types of media. Since tapes are expensive and I've had terrible luck with them professionally, I'd like to stick to optical when possible.
MidnightBSD: The BSD for Everyone
Disk mirroring (aka, "RAID 10") is easy to set up on most systems any more. Disks get cheaper every year. The original reason for RAID was to minimize hardware costs while providing some redundancy for failure-recovery, back when a 1 gig disk used to cost upwards of $10,000. As hardware costs have declined, the financial incentive to cut these corners have also declined. With disks well below $1000, there really isn't much reason not to keep online mirrors and offline full image backups.
At home, now, I keep critical data on mirrored disks, and rotate several other full image backup disks offline. All the disks are identical make/model, and all are bootable. The offline disks are stored in a fireproof safe. I periodically send a disk to my brother just in case of fire or flood. When my critical data grows to fill one of those, I will get a half dozen new disks, each 3 or 4 times bigger than the old ones.
If you have multiple ters of personal data, you might want to consider if all of it really needs to be backed up. The quicken file is important. The sixty-fifth five-hour video of the sleeping baby might not be as important.
You mean "scrubbing"; I don't know what Promise's patent covers, but basic scrubbing and self-diagnostic monitoring has been built in to commercial-grade arrays since forever.
*sigh* Arithmetic really isn't your strong point, is it?
Drive 0 mirrors Drive 1 = 1 TB
Drive 2 mirrors Drive 3 = 1 TB
1+1=2, so a total of 2TB. Really, look it up
But since you've descended to (well, more "started with" rather than actually "descended to") ad hominem attacks, it's perfectly clear that there's no point in trying to have a reasonable discourse with you.
Not everything that can be measured matters; Not everything that matters can be measured.
That's nothing. He was just telling me about his best girl back in Iowa, and they're gonna get married just as soon as he gets back home. He's also only three days from retirement.
I'd say more, but he's got to change into his red shirt for an away team mission consisting exclusively of him, McCoy, Spock, and Kirk.
Did he say whether or not he was using an ACME brand RAID controller?
There are some people that if they don't know, you can't tell 'em.
The point of that article was how to present poor statistical analysis (being someone who hated statistics at uni, I can see my own half-arsed attempts in the article).
Or perhaps its fairer to say "poor statistical analysis accompanies by real-world lack of knowledge".
Now, if you work with SAN's and storage, you probably already see the faults that I really cant be bothered pointing out... we can just sit here and laugh together at yet another ill-conceived zdnet article shall we?
A successful RAID scrub depends on perfect error reporting. ZFS does not.
you had me at #!
...than RAID can, by design. That's the OP's point here. Disk failures are far from the only failure mode; and many failures are neither detected nor reported.
you had me at #!
So RMS = Moses 2.0?
"Our opponent is an alien starship packed with atomic bombs. We have a protractor."
The server is still up, if only as fallback, so I can look this up. My original numbers were only rough guesses.
One is set of 4 Seagate ST3400832A (400GB, i.e. 1.6TB total), with the disks showing 31321h power-up time (that is 3.6 years. The second one is 8 Seagate ST3500641AS (500BG, 4TB total), with disk uptimes of 20548h, i.e. 2.3 years. I did reduce the scrubbing interval to 1 every 30 days about a year ago, so lets factor that in.
Incidentially, there is an error in my original numbers: 2 years at once every 15 days is 50 complete reads, not 100, i.e. 200TB read. Sorry. Better numbers:
1st Array:
2.6 yrs @ 15 day interval, 1.6TB = 63 scans, 1.6TB = 101 TB read
1 yr @ 30 day interval, 1.6TB = 12 scans, 1.6TB = 19 TB read
2nd Array:
1.3 yrs @ 15 day interval, 4TB = 18 scans, 4 TB = 72 TB read
1 yr @ 30 day interval, 4TB = 12 Scans, 4 TB = 48 TB read
That is 240 TB read in surface scans alone, without a single uncorrectable. There is one disk with 4 reallocated sectors and one with 5, the rest is at zero.
As to the uncorrectable rate, Seagate says 1 in 1^14 reads is uncorrectable, i.e. a 512 Byte sector read, if I interpret this corrctly. That would mean one unreadable sector every 5.12 * 10^16 Bytes, i.e. one sector missing every 51 EB. That would fit the observation. Seems there is a major combinatoric goof in the original article. It is still true that one in 10^14 bits read independently (!) would be unrecoverable, but the independence is not there, as you either get a complete sector or nothing.
To spin this further, if it really is 1 in 1^14 sectors, an 8 disk RAID5 array with 1TB disks has about 1.36*10^10 sectors and hence a chance of one unrecoverable sector of roughly 1 in 730 during RAID5 rebuild. At to the 3% failure rate per year number, at 10MB/s rebuild write speed (conservative), the rebuild takes 28 hours, i.e. adding another royghly 1 in 10000 chance of a disk dying during rebuild.
Interessting...
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Wow, reading the Wikipedia article I kindly pointed you to in between two insults isn't your strong point is it? What you're talking about is NOT RAID 1 (sounds more like RAID 10 depending on the relationship between your pairs of RAID 1 couples). You can criticise nested RAID levels all you want, but don't pick one and call it RAID 1 when it's not.
Imbecile.
You just got troll'd!
Because root-mean-square is the ONLY way to get the REAL voltage.
Let me get this straight: you embed your file system into copies of portions of your file system? Is that redundant, recursive, or meta?
NTFS shares it's lineage with HPFS, which was part of OS/2 - originally a joint IBM and Microsoft venture.
- It's not the Macs I hate. It's Digg users. -
And our god said unto us, let us ejaculate onto the face of our woman, with our bestfriend, and his neighbor, and his neighbor's nephew, and the mailman.......
Skiffy is Spiffy, but Ort is tort.
RAID 5 is designed for 3 to 4 disk arrays. RAID 6 should be used up to 6 or 7 disks. Anything bigger, and you really need something like RAID 1 over RAID 5 or a clustering file system. AndrewFS, Lustre, or anything similar can make the chances of losing every copy of your data extremely unlikely.
You should still have data somewhere off-site just in case. A Lustre system at your primary location and a smaller Lustre system for compressed backups at a secondary location should be plenty of redundancy.
First, ZFS is trash. It's not even complete. There are over 300 fixes in the current patch cluster from Sun. Not to mention there are no robust tools to fix it once it breaks. Did I mention software raid rocks? Secondly, hardware raid isn't going away. Finally, the article doesn't take into account increased throughput of new drives that will be developed.
Actually, NTFS is based on the VMS file system. The architect of NTFS came over to MS from DEC along with Dave Cutler, who was the architect/project mgr of NT.
The one that almost bit us in the ass was a setup where we had a staging environment that had a few hundred gigs of files that would using rsync to push them out to a handful of webservers across the globe.
The disk array that was on the staging machine failed to come up one day, so rsync dutifully synced up the empty /mnt/staging mount point to all the webservers.
In some backup situations (certainly my personal stuff) it's usually acceptable to tell rsync to never delete stuff on the other end. Sure you can still corrupt one file and propagate the corrupt version, but you are less likely to wholesale blow away your backup.
4 years is too long. You need to start rotating in some new drives...Even the very best drives don't offer warranty replacement past 5 years.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
I'm no RAID expert, but surely the 10^-14 bit error rate the guy is talking about is the rate of previously undiscovered errors? In which case, those errors won't have been found by a bad-sector sweep, nifty though that feature is for advance warning of other problems. His point is that one error in 10^14 bits, though it doesn't sound a lot, is actually one error every time you read 12TB of previously-working data. Which is what happens during RAID5 disk rebuild.
Peter
Drobo!
How many times does this have to be said.
RAID is not a backup.
Yeah... How many times does it have to be said? I mean, seriously, it seems everybody else who commented here said the same thing...
Bow-ties are cool.
I love how anytime somebody gives an example of a solution to a problem, an anonymous jerk on the Internet has to accuse them of working for the companies interests. That's a WONDERFUL way of trying to discredit me.
I have purchased and setup a lot of RAID systems in my time and Promise was just ONE EXAMPLE that came to mind. Why? I just bought a Promise 4-bay NAS. That was in the marketing literature I had read before purchasing the unit.
Promise is not the only one to have technology and methods like that. I would bet that Adaptec, 3Ware, etc. all have similar technology.
So *fucking* excuse me. Next time I will take the time to find ALL the technologies from EVERY vendor and present them to /. That way jerks like you will have less opportunity to claim that I secretly work for Vendor A serving products from Manufacturer B.
But... just for shits and giggles... why not try to respond to the technology I mentioned? All you have done is to parrot the article. The "problem described in the article" is less likely to happen with the technology that I mentioned. Plain and Simple. Respond to that. I'll wait.
I hate distros that do that by default. I'm not going to confirm every single file I want to delete, so when I delete a directory, instead of just using rm -r, I'm just going to use rm -rf instead. Now, not only have I completely negated any benefit of using rm -i, but I've also lost the confirmation for files I might want to be warned about (I don't encounter this often, but I think it warns you about deleting files you don't have write access to, but can still delete because you have write access to the parent directory and the sticky bit is off).
$x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
$x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
It's still no replacement for offsite backups but Lime Technology (http://lime-technology.com/) has a JBOD+parity solution that uses a parity drive to protect your data but even in a 2 disk failure more you only lose the data from those disks (or if the second was the parity drive you just lose the data from the one disk). And in the case of a single drive failure it'll limp along using the parity drive until you replace the bad disk and rebuild. The 3-drive version is free but if you want to go bigger they charge a nominal cost (sub $100)
It works as a great solution for a media server where you may not necessarily care about all the data but you'd prefer to not lose ALL of it at once should multiple drives fail.
They still spin don't they? Oh drats, there are now disks (SD) that don't spin since they are not disks but they otherwise look like disks... same form factors... sigh.. progress.
Which RAID controllers do that, exactly? That's a bit like saying that SATA controllers will treat a drive with a single bad sector as being dead. I've never seen a controller do such a thing, and I doubt anyone making those controllers would stay in business for long.
Some controllers will consider a drive as "missing" if it doesn't respond for more than 30 seconds or so, and some SATA drives had a bug (more of a design flaw when used in RAID) that made them spend up to 2 minutes trying to recover a bad sector before responding. The result was the controller assumed the drive had died and said the rebuild had failed. In other words, the problem was the delay, not the bad sector. Anyway, you could still restart the rebuild, but this could be painfully slow if you had a lot of bad sectors.
This is not true for modern controllers, SCSI drives or "RAID edition" SATA drives, that never spend more than a couple of seconds trying to recover bad sectors (they simply give up and let the RAID controller handle it).
I'm not sure if the SATA spec has been expanded to include a "I'm busy trying to remap a sector" drive state, which would obviously be the ideal solution, but "TLER" and longer wait times by modern controllers have made the problem essentially disappear, and I think the problem never existed with SCSI / SAS drives, which is what most people would use for "enterprise" RAID-5 arrays.
Hmm, how do you rotate in drives in a RAID0 configuration? I know how to do it in a RAID1 or RAID5 setup, but I don't see how to do it in RAID0 without doing something like cloning the drives.
You've got a bunch of 4 year old drives in Raid 0? Jesus. I'd be afraid to run a defragger or reboot. The average service life of a hard drive these days is considered to be 3-5 years, so it would be a good idea to make a backup of anything you care about.
Yea, you need to duplicate them. Might be easier to just buy one modern drive that is bigger than all the older drives, and copy all the data new drive.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
That's smilar to what I do for my home data (I back up to a colo server though), but I think you might be missing one possibly important thing. You should guard against corrupted files in your backup, which could propagate through your snapshots by running rsync with the -c flag (compares files by a hash of their contents instead of just size and timestamp).
I leave my MythTV shows in the considerably less secure hands of a Linux MD RAID 5 array. I'm surprised you have only 3GB to backup, My digital photo's alone are 29GB and I'm not a big photographer. The size of each of my snapshots is 162 GB.
X.
And there will still be some geek from Slashdot yelling from atop of the mountain that if you had any skills you'd have a backup set of stones located in a climate-controlled underground bunker, and that, by the way, backing up data is important!
You should guard against corrupted files in your backup, which could propagate through your snapshots by running rsync with the -c flag (compares files by a hash of their contents instead of just size and timestamp).
My backup server is an old P3 machine. I wonder if it could even hash all of my files before the next day's snapshot got kicked off? :)
I'm surprised you have only 3GB to backup, My digital photo's alone are 29GB and I'm not a big photographer. The size of each of my snapshots is 162 GB.
I know some people who store the RAW file for every photo that they take. I delete what I don't want. So far, I haven't regretted my decision, but maybe some day I will, and maybe then I'll spring for lightroom or something.
It's 3GB compressed, by the way. I have no idea where I'd find 162GB of data that I want to back up. High def (ahem) home movies, perhaps?
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
Home movies as well, although I wouldn't call them hi-def. I also backup my music collection. I may have lost some of the original CD's ;)
I don't do the -c thing either, but I keep thinking I should. It's probably going to take a real long time indeed. Perhaps it's worth keeping an MD5 database in both locations and checking them regularly. Wouldn't the tripwire tool be good for that?
Personally, I believe the correct answer to ensuring data recoverability is RAID together with real-time replication.
Like, say, RAID-1? :)
Honestly, given the cost/GB ratios these days, the space advantages of RAID-5 seem pretty silly compared to the reliability and performance issues. Why not just go with something like RAID-10 and be done with it?
> With 12 TB of capacity in the remaining RAID 5
> stripe and an URE rate of 10^14, you are highly
> likely to encounter a URE. Almost certain, if
> the drive vendors are right.
So one sector (let's say 4kb) in 12 trillion bits is bad, so you're saying your RAID controller is so pathetic that it cas not continue?!?!?!
WTF?!?!?!?
> Oh, you didn't back it up to tape?
TAPE?!?!?
Try backing up to eSATA drives.
Basically, this moron has no clue of how to actually use a PC.
Andy
But that was from God via Solomon, not via Moses.
...the future crusty old bastards are already drinking the Kool-Aid.
So a lot of people are saying ZFS is a great solution for a number of the issues brought up in the article.
ZFS isn't available on Linux due to incompatibilities of the CDDL, and Linux's GPL license.
CDDL is (in my opinion) more "free" than GPL (which forces redistribution of code; but let's avoid that debate right now.)
Okay, so if ZFS can't be bundled in a Linux distribution and redistributed while maintaining a GPL license, fair enough. But what is stopping anyone from doing the port, and providing ZFS as a freely distributed, do-what-you-want-with package, that installs and runs fine on Linux. If a user chooses to take this freely licensed ZFS and compile/link/install it on their Linux system, that is none of Linux/GPL's business.
Yes, redistribution is thwarted by the GPL, but why would install-it-yourself be problematic? I'd settle for that. Why isn't this available? Why hasn't anyone finished a port that can be used in this manner?
Not trolling, honestly curious about it...
Love many, trust a few, do harm to none.
Well, because it's still too expensive for a lot of businesses to go full RAID 10 on their main storage system.
The disks you buy at NewEgg are cheap, but the disks you buy for your SAN are not as cheap. They might be the same disks, but that's just the way it is. And, the big costs come in the form of cost per slot, not necessarily the disk that plugs into it.
RAID-5 doesn't suffer from any real performance issues. Not for the last 10 years, anyway. Read speed is as fast as a stripe set, and write performance hits are easily mitigated by on-board cache. I kinda thought this was common knowledge..
Replication can be done to a much cheaper unit or DAS, and/or can be sent to an off-site location for better recoverability in the case of a real disaster.
- It's not the Macs I hate. It's Digg users. -
Maybe: he used parens, so it counts maybe as mumbling, maybe as just thinking it.
s/IBM/DEC programmers employed at M$/g
I know tobacco is bad for you, so I smoke weed with crack.
Yeah, as I said, I have nightly backups to another internal drive and to an external 1TB USB drive (which mirrors the whole RAID0 drive). Mainly, I use RAID0 since it makes a very noticeable difference in boot times and load times in games. It might die, but I shouldn't lose any data.
Actually, HPFS and NTFS are completely different. However, even if they weren't it wouldn't matter, because HPFS was also created by Microsoft.