Slashdot Mirror
RAID's Days May Be Numbered
storagedude sends in an article claiming that RAID is nearing the end of the line because of soaring rebuild times and the growing risk of data loss. "The concept of parity-based RAID (levels 3, 5 and 6) is now pretty old in technological terms, and the technology's limitations will become pretty clear in the not-too-distant future — and are probably obvious to some users already. In my opinion, RAID-6 is a reliability Band Aid for RAID-5, and going from one parity drive to two is simply delaying the inevitable. The bottom line is this: Disk density has increased far more than performance and hard error rates haven't changed much, creating much greater RAID rebuild times and a much higher risk of data loss. In short, it's a scenario that will eventually require a solution, if not a whole new way of storing and protecting data."
Don't consider an entire drive is dead if you get a piddly one-sector error.
Just mark it read only and keep chugging.
Or just regenerate and write the one sector from the parity data since all modern hard disks reallocate bad sectors on write.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
What's hard to remove is nigger grease. If you have ever gone to a swimming pool only to find that a bunch of blacks are already swimming in it then you know what nigger grease is all about. It is easiest to see right after they exit the water. It's a thin film on the surface of the water deposited by the vast quantities of oil that their overactive glands produce. Seriously we can be carbon neutral tomorrow if we found a way to make biodiesel out of it as the supply is abundant. Anyway, if you swim in a pool like that you will feel yourself coated with the nigger grease. It is not pleasant. Is there a filter somewhere that can remove even KFC-fortified nigger grease?
Honestly, there really aren't that many unsolved problems in computing if you are sufficiently aware enough to include mainframes and mainframe operating disciplines in your consideration. The basic way the mainframe community solved this particular problem long ago was to, first, take a holistic view about mitigating data loss. Double concurrent spindle failures are just one possible risk element. What about, for example, an entire data center exploding in a spectacular fireball? (Or whatever.) IBM, for example, came up with several different flavors of GDPS and continues to refine them, and they include multiple approaches to data storage tiering across geographies, depending on what you're trying to achieve. Data loss, whether physical or otherwise (such as security breaches), is not a particular problem with this class of technology and associated IT discipline, nor does there seem to be any signs of a growing problem in this particular technology class.
The author says it himself in the article:
"And running software RAID-5 or RAID-6 equivalent does not address the underlying issues with the drive. Yes, you could mirror to get out of the disk reliability penalty box, but that does not address the cost issue."
but he hasn't adressed the fact that today you get 100 times as much diskspace for the same cost as you did 10 years ago when cost was a factor. In real life cost isn't a factor when it comes to datastorage, simply because it's really low in real life projects, as compared to the other costs in a project requiring storage. So if you want the reliability you go get a mirror. Drivespace is dirt cheap.
As for the rebuildtimes, fine, go buy FASTER drives. I dont see the problem. HP and many other vendors have long been trying to sell combined raid soltions (like the EVA) where you mix high storage with high performance drives (like SSD vs. SATA).
The only real argument for the validity of this article is the personal use of drives/storage. And name 3 people you know who run raid-5 on their personal PCs, and I'll show you 3 guys who can't afford an SSD drive.
--- To err is human... Am I more human than most ?
(Certain) RAID (levels) address the issue of potential dataloss due to hardware malfunction. How does moving to an Object-Based Storage Device address this issue better? Actually, I don't see how RAID and OSD are mutually exclusive.
Now that's a stupid article.
It basically says, you can't read a harddisk more than X times before you get an error on some sector, so RAID is dead. That's a logical nonsequitur. RAID is a generic technology that also applies to flash memory cards, USB sticks, anything you can store data on basically. The base technique says "given this reliability, you can up the reliability if you add some redundancy". There's no link to harddisks other than that that's what they're used for right now.
Probably the next meta solution after RAID 6 will be something like ZFS, where the filesystem that works not just on the fs-specific layer, but on the LVM layer so it can log CRCs of files and immediately be able to tell if a file got corrupted (and perhaps fix it with some ECC records.) One can see a filesystem not just writing a RAID layer, but taking recovery data and storing that away as filesystem metadata.
Of course, there is always doing redundant arrays of RAID clusters, say three groups, two data, one parity, or mirroring RAID 5 volumes. You have the usual tradeoffs: The more fancy the RAID scheme, the more disks you need, and the more computing you have to do for every bit thrown at and read off the array.
Long term solution? A move to something other than magnetic storage. This could be optical, it could be SSD if some advance allows very large density increases, or something unknown. The technology would have to have a rate of failure magnitudes better than magnetic, as well as a cost on par with magnetic for it to completely work. Holographic storage has languished for a while, perhaps as the technology improves for that, we may see drives using 3D blocks of that replacing the old fashioned spindles.
Ask, ask, ask
Disclaimer: I work for a storage vendor.
> FTA: The real fix must be based on new technology such as OSD, where the disk knows what is stored on it and only has to read and write the objects being managed, not the whole device
OSD doesn't change anything. The disk has failed. How has OSD helped?
> FTA: or something like declustered RAID
Just skimming that document it seems to claim: only reconstruct data, not white space, and use a parity scheme that limits damage. Enterprise arrays that have native filesystem virtualisation (WAFL for example) already do this. RAID 6 arrays do this.
Lets recap. Physical devices including SSDs will fail. You need to be able to recover from failure. The failure could be as bad as the entire physical device failing, or as bad as a single sector being unreadable. In the former case a RAID reconstruct will recover the data but you'll hit RAID recovery errors due to the raw amount of data that needs to be recovered. Enterprise arrays mitigate the risk of recovery errors by using RAID 6. They could even recover the data from a DR mirrored system as part of the recovery scheme.
And when RAID 6 has a high enough risk that it's worth expanding the scheme everyone will start switching from double parity schemes to triple parity schemes since their much less expensive in terms of spindle count than RAID 6+1.
One assumption is, at some point in the future, reconstructions will be a continual occurring background task just like any other background task that enterprise arrays handle. As long as there is enough resiliency and performance isn't impacted then it doesn't matter if a disk is being rebuilt.
Hardware RAID is dead - software for redundant storage is just getting started. I am looking forward to making use of btrfs so I can have some consistency and confidence to how I deal with any ultimately disposable storage component.
The ZFS folks have been doing it fine for some time now.
Hardware RAID controllers have no place in modern storage arrays - except those forced to run Windows
For enterprise level storage systems, this is also a non-issue because of thin provisioning.
"I love my job, but I hate talking to people like you" (Freddie Mercury)
I admit I'm not an expert, but I was under the impression that RAID was mainly about ensuring you a large number of spindles and some redundancy so you can serve data quickly even if a couple of drives fail while the servers are under pressure. Surely you would not rely on a RAID to avoid data loss since you should be keeping external backups anyway?
the rebuiling times are really astronomical. I don't know how my arrays do it, but it routinely costs me 3+ hours to rebuild
that, and the various scanning / fixing / searching tasks.. endless, if you work with the larger drives, even if sata attached
'course it's nice to have large HD's on desktop pc's, but when you have to fix bosses' PC and it takes 8 (!!) hours to clone, scan and repair.. while he can't work.. that's no good.
my 2cts
The article assumes that when within a RAID5 array a drive encounters a single sector failure (the most common failure scenario), an entire disk has to go offline, be replaced and rebuilt.
That is utter nonsense, of course. All that's needed is to rebuild a single affected stripe of the array to a spare disk. (You do have spares in your RAID setups, right?)
As soon as the single stripe is rebuilt, the whole array is again in a fully redundant state again - although the redundancy is spread across the drive with a bad sector and the spare.
Even better, modern drives have internal sector remapping tables and when a bad sector occurs, all the array has to do is to read the other disks, calculate the sector, and WRITE it back to the FAILED drive.
The drive will remap the sector, replace it with a good one, and tada, we have a well working array again. In fact, this is exactly what Linux's MD RAID5 driver does, so it's not just a theory.
Catastrophic whole-drive failures (head crash, etc) do happen, too. And there the article would have a point - you need to rebuild the whole array. But then - these are by a couple orders of magnitude less frequent than simple data errors. So no reason to worry again.
*sigh*
If you want smaller drives to speed up rebuild times then, erm, buy smaller drives? You can get ~70Gb 10Krpm and 15Krpm drives fairly readily - much smaller than the 500-to-2000-Gb monsters and faster too. You can still buy ~80Gb PATA drives too, I've seen them when shopping for larger models, though you only save a couple of peanuts compared to the cost of 250+Gb units.
If you can't afford those but still don't want 500+Gb drives because they take too long to rebuild if the array is compromised and needs a rebuild, and management won't let you buy bog standard 160Gb (or smaller) drives as they only cost 20% less than 750Gb units without the speed benefits of the high cost 15Krpm ones, how about using software RAID and only using the first part of the drive? Easily done with Linux's software RAID (partition the drives with a single 100Gb (for example) partition, and RAID that instead of the full drive) and I'm sure just as easy with other OSs. You'll get speed bonuses too: you'll be using the fastest part of the drive in terms of bulk transfer speed (most spinning drives are arranged such that the earlier tracks have higher data density) and you'll have lower latency on average as the heads will never need to move the full diameter of the platter. And you've got the rest of the drive space to expand onto if needed later. Or maybe you could hide your porn stash there.
I've managed to get this going, using the excellent FreeNAS - although proceed with caution, as only the beta build supports it, and I've already had serious (all data lost) crashes twice.
However the principle is sound, and I'm sure this will become standard before long - the only trouble being that HP, Dell and the like can't simply offer upgrades for existing RAID cards - due to the nature of ZFS, it needs a 'proper' CPU and a gig or two or RAM. Even so, it does protect against many of the problems now besetting RAID (which was never meant to handle modern, gargantuan disk sizes).
What about fountain codes? The coding there is capable of recovering from a greater variety of faults.
http://www.youtube.com/watch?v=96dWOEa4Djs
This is something the ZFS creators have been talking about for some time, and been actively trying to solve.
ZFS now has triple parity, as well as actively checksumming every disk block.
You can tell how powerful someone is by the magnitude of the crime they can commit and be able to get away with.
But really none of that should be necessary for the general case. Storing data in different physical locations is a good but entirely unrelated issue- the main problem of disk reliability is still very much in need of a solution. That's pretty much the point of the article: You can come up with various solutions which move the problem around, give multiple fallbacks for when something goes wrong.. but there's still the problem of things going wrong in the first place. I shouldn't need to use 12 separate disks spread across the globe just for basic reliability / redundancy
Read that before on slashdot. Why RAID 5 Stops Working In 2009
Actually I like the parity declustering idea that was linked to in that article, seems to me if implemented correctly it could mitigate a large part of the issue. I have personally encountered the hard error on RAID5 rebuild issue, twice, so there definitely is a problem to be addressed...and yes, I do now only implement RAID6 as a result.
For those who haven't RTFATFALT (RTFA the f*** article links to), parity declustering, as I understand it, is where you have, say, an 8 drive array, but where each block is written to only a subset of those drives, say 4. Now, obviously you loose 25% of your storage capacity (1/4), but consider a rebuild for a failed disk. In this instance only 50% of your blocks are likely to be on your failed drive, so immediately you cut your rebuild time in half, halving your data reads, and therefore your chance of encountering a hard error. Larger numbers of disks in the array, or spanning your data over fewer drives, cuts this further.
Now, consider the flexibility you could build into an implmentation of this scheme. Simply by allowing the number of drives a block spans to be configurable on a per block basis, you could then allow any filesystem that is on that array to say, on a per file basis, how many disks to span over. You could then allow apps and sysadmins to say that a given file needs to have the maximum write performance, so diskSpan=2, which gives you effectively RAID10 for that file (each block is written to 2 drives, but with multiple blocks in the file is likely to be written to a different pair of drives, not quite RAID10, but close). Where you didn't want a file to consume 2x its size on the storage system, you could allow a higher diskSpan number. You could also allow configurable parity on a per block basis, so particularly important files can survive multiple disk failures, temp files could have no parity. There would need to be a rule however that parity+diskSpan is less than or equal to the number of devices in the array.
Obviously there is an issue here where the total capacity of the array is not knowable, files with diskSpan numbers lower than the default for the array will reduce the capacity, numbers higher will increase it. This alone might require new filesystems, but you could implement todays filesystems on this array as long as you disallowed the per-block diskSpan feature.
This even helps for expanding the array, as there is now no need to re-read all of the data in the array (with the resulting chance of encountering a hard error, adding huge load to the system causing a drive to fail, etc). The extra capacity is simply available. Over time you probably want a redistribution routine to move data from the existing array members to the new members to spread the load and capacity.
How about you implement a performance optimiser too, that looks for the most frequently accessed blocks and ensures they are evenly spread over the disks. If you take into account the performance of the individual disks themselves, you could allow for effectively a hierarchical filesystem, so that one array contains, say, SSD, SAS and SATA drives, and the optimiser ensures that data is allocated to individual drives based on the frequency of access of that data and the performance of the drive. Obviously the applications or sysadmin could indicate to the array which files were more performance sensitive, so influencing the eventual location of the data as it is written.
Stealing a rhinoceros should not be attempted lightly.
I shouldn't need to use 12 separate disks spread across the globe just for basic reliability / redundancy
You're trying to weasel out of paying IBM protection money !
May contain traces of nut.
Made from the freshest electrons.
Will scalable distributed storage systems like Hadoop and Google File System take over from RAID?
As others have mentioned, this is something that is discussed on the ZFS mailing lists frequently.
For more info there, check out the digest for zfs-discuss@opensolaris.org
and, in particular, check out Richard Elling's blog
(Disclaimer: I work for Sun, but not in the ZFS group)
The fundamental problem here isn't the RAID concept, is that the throughput and access times of spinning rust haven't changed much in 30 years. Fundamentally, today's hard drive is no more than 100 times as fast (both in throughput and latency) than a 1980s one, while it holds well over 1 million times more.
ZFS (and other advanced filesystems) will now do partial reconstruction of a failed drive (that is, they don't have to bit copy the entire drive, only the parts which are used), which helps. But there are still problems. ZFS's pathological case results in rebuild times of 2-3 WEEKS for a 1TB drive in a RAID-Z (similar to RAID-5). It's all due to the horribly small throughput, maximum IOPs, and latency of the hard drive.
SSDs, on the other hand, are no where near the problem. They've got considerably more throughput than a hard drive, and, more importantly, THOUSANDS of times better IOPS. Frankly, more than any other reason, I expect the significant IOPS of the SSD to signal the death knell of HDs in the next decade. By 2020, expect HDs to be gone from everything, even in places where HDs still have better GB/$. The rebuild rates and maintenance of HDs simply can't compete with flash.
Note: IOPS = I/O Per Second, or the number of read/write operations (irregardless of size) which a disk can service. HDs top out around 350, consumer SSDs do under 10,000, and high-end SSDs can do up to 100,000.
-Erik
There are always four sides to every story: your side, their side, the truth, and what really happened.
Article should be titled "Parity - based RAID days are numbered". There's nothing wrong with RAID 1,0, 10
The real problem with "classic" RAID is that 1 single error means a total rebuild of the array.
my other sig is a 500 page novel
The cloud. Just cloud it, baby. Nothing bad ever happens in the cloud; they're so white and fluffy after all.
Um, don't schemes like raid 1+0 solve the parity rebuild problem? Even in the worst case of full disk loss, only one disk needs to be rebuilt and even for a large disk that doesn't take very long. Am I missing something?
In theory, there's no difference between theory and practice; in practice there is.
The RAID concept can be extended to multiple PCs forming a storage grid. One open-source implementation is Tahoe LAFS.
Actually, storing data in a multiple data center / high availability environment is a completely related issue. The summary above talks of "entirely different paradigms." Cloud storage would be multiple data center based, which is entirely different from keeping the only copy on your local drives. In this concept, your machine would have enough OS to boot, and enough hard drive space to download the current version of whatever software you are leasing. Your personal info would always be maintained in the data centers, and only mirrored locally. Have a home failure? Drop in a new part or even a new PC, (possibly with an entirely different operating system, such as Chrome,) connect to the service, and you're 100% back.
It's no longer a novel concept for the home market. Consider Google Docs. It's not even being sold as "safer than RAID", it's being touted as "get it from anywhere" or "share with your friends". Safer than RAID is just a bonus.
So are we ready to move all our personal information to clouds? I certainly am not, but Google Docs are wildly popular and a lot of people are. I long ago learned that I can't look to myself to judge what the mainstream attitudes are in many things.
John
RAID 4 is where you have one dedicated parity drive. RAID 5 solves this by spreading the parity information for each drive to all the other drives in the array. RAID 6 adds a second parity block for increased reliability, but as a result of the increased write for that extra parity block, it slows down write speeds.
The real key to making RAID 4, 5, or 6 work is that you really need 4-6 drives in the array to take advantage of the design. I wouldn't say that it will fall out of favor though, because having solid protection from a single drive going bad really is critical for many businesses. Backups are all well and good for if your system crashes, but for most businesses, uptimes are more critical yet. So, backups for data so corruption problems can be rolled back, and RAID 5,6,10 for stability and to avoid having the entire system die if one drive goes bad. What takes more time, doing a data restore from a backup for when an individual application has problems, or having to restore the entire system from a backup, with the potential that the backup itself was corrupted?
With that said, web farms and other applications can get away with just using a cluster approach instead of a single well designed machine(or set of machines) have become popular, but there are many situations which make a system with one or more RAID arrays a better choice. The focus on RAID 0 and 1 for SMALL systems and residential setups has simply kept many people from realizing how useful a 4-drive RAID 5 setup would be.
Then again, most people go to a backup when they screw up their system, not because of a hard drive failure. With techs upgrading hardware before they run into a hard drive failure, the need for RAID 1, 4, 5, and 6 has dropped.
I will say this, since a RAID 5 array can rebuild on the fly(since it keeps working even if one drive fails), the rebuild time itself does not significantly impact system availability. Gone are the days when a rebuild has to be done while the system is down.
Sorry if this is a bit offtopic, guys.
I noticed the crystals tag on the story, which reminded me of the old Star Trek episodes where someone would open a case of storage crystals, select one, and then access some tremendously huge amount of data on a local terminal using it.
The thought that popped into my head the other day was this - how do they always seem to know what crystal to select? There would often be 20 - 25 in a case, and they were all unlabelled!
I had an amusing image of some kind 100 petabyte crystal technology marred by the user's sticking labels on the sides of them with "movie collection" scrawled in biro!
*cough* http://www.b-virtual.org/display/WWWBVIRT/Distributed+Storage+System *cough*
Seems like an adequate solution to the problem, at least according to the talk I heard last week. BTW, I am not associated with B-virtual.
I am the Shield Anvil. And I am not yet done.
So, in summary, current RAID designs have problems with large drives. This basically means that you'll encounter issues. The simplest way of saying this is that it fails. A quick recap: RAID has problems.
I use RAID6 for several high-volume machines at work. Having double parity plus a hot spare means rebuild time is no worry.
But if you are not a fan you can always throw something together with ZFS's RAIDZ or RAIDZ2 which is also distributed parity but the ZFS filesystem checksums and keeps multiple (distributed) copies of every block to detect and fix data corruption before it becomes a bigger problem.
People using ZFS have been able to detect silent data corruption from a faulty power supply that other solutions would never have found just because of the checksumming process.
No raid based solution equals having a backup... period.
If your worried about down time, build in redundancy as was mentioned several times above...
We run raid5 and raid 10 on various systems for their data, and backup to multiple destinations (tape and hard drive)... we can afford some downtime if 0things fail...
Not sure I seal a real problem...
Is he saying that you can never read a whole hard disk because it will fail before you get to the end?
That's what it seems like he's saying but my hard disks usually last for years of continuous so I'm not sure it's true.
No sig today...
Actually, storing data in a multiple data center / high availability environment is a completely related issue. The summary above talks of "entirely different paradigms." Cloud storage would be multiple data center based, which is entirely different from keeping the only copy on your local drives. In this concept, your machine would have enough OS to boot, and enough hard drive space to download the current version of whatever software you are leasing. Your personal info would always be maintained in the data centers, and only mirrored locally. Have a home failure? Drop in a new part or even a new PC, (possibly with an entirely different operating system, such as Chrome,) connect to the service, and you're 100% back.
Unfortunately, that's just moving the issue to the cloud, which (since it is storing great gobs of data) is likely to be using the highest capacity drives available in some sort of RAID configuration.
Nothing for 6-digit uids?
Long time listener, first time caller...
How is this news? Mirrored raids cut possible storage space in half for something a simple backup would take care of in most instances. The data access increase provided by striped raid is negligible at best, and if you lose that raid, you lose everything.
The only possible advantage to a raid is *if* you have a mirrored raid and *if* you're working with a major business server that you can't afford to be down for any length of time. Of course, you still have to bring it down to swap the drives and rebuild the array. So what's the point?
IIRC, Google doesn't use RAID for their data. They "just" ensure there is always a specified number of distributed copies available. If one copy becomes unavailable, make another copy.
It seems that time is of the essence when doing a rebuild, why not consume disks faster, but decrease the time it takes to do a rebuild. So in a raid setup with a hot spare, come up with an algorithm to basically write to the spare a distributed set of the data that is being written to the raid. The algorithm part, I guess, can be better explained as a daily/weekly backup. So If I'm writing an incremented number every day to the same spot on a disk, maybe the hot spare only has the data written to it on fridays the next time it is updated. My guess is that data is usually changed infrequently, and more or less you only add data. So the hot spare should have pretty close to 1/2 the correct data on it upon a disk failure. Then you only have to rebuild the other 1/2 or so. So my guess would be the hot spare would have a fraction of the writes (closer to 1/1 than 1/2) of a normal disk, and none of the reads until it's ready to go into service. 1/2 the rebuild time or so. Maybe it's worth it?
Nice database you got there. Shame if anything happened to it.
Don't discourage the boy. Weaseling out of things is important to learn. It's what separates us from the animals
--AlexC
Just because I dont agree with climate change doesnt make me a troll
And then like AOL, Google goes out of business (shocker I know) and all your data is lost forever. The cloud is good for a lot of stuff, but for data storage it should be part of the solution, not 100% of it.
http://hardware.slashdot.org/article.pl?sid=08/10/21/2126252
Not from weasels, though...
...because someone thought that since they had RAID, they didn't need to back up the data...
What gets me, especially in the Linux world, is the difficulty or sometimes the impossibility of monitoring the arrays for their state. We've had several controllers that we've only found out about bad disks on physical inspection. This limits the controllers we use and thus might be using a lesser-performing controller only because we can monitor it...
rm
Sci-Fi Storm
Here's what I want, folks:
A 5.25 inch device with 5 double-sided platters running at 5400 RPM. Basically the same size as a desktop CD/DVD drive, ala Quantum Bigfoot.
I want 8 sides of the platters dedicated to data, and the other two sides dedicated to parity (or one parity and the other servo), essentially a self-contained RAID on a single disk.
I want all data heads to write and read simultaneously, in Parallel. The idea is to have 64 byte sectors on each platter which are recombined into a 512-byte result. 8 heads writing and reading in paralell means HUGE throughput for sequential operations.
It's RAID 5 or 6 on a single disk, although without spindle redundancy.
And I also want a high-performance option: 2 sets of read/write heads 180 degrees apart, which effectively would cut seek times in half, making the drive perform more like a 10k RPM drive. With current densities, that's 12 TB in the volume of a DVD drive. It solves speed, sector error recovery and capacity issues. The only thing missing is a data bus that can handle the throughput.
I thought windows home server resolved this issue a while ago. Their non-raid multi disk approach is well documented regarding it's function as well as philosophy. http://www.google.com/search?q=windows+home+server+doesn't+use+raid
I was suspicious of this at first but reading up on it, it sounds pretty good.
Flappinbooger isn't my real name
You are absolutely correct in that the mainframe world has dealt with all of the modern recovery issues. But think of the actual USE of storage these days. What used to be a colossal database is now just a bunch of a bunch of home videos from my camcorder. Not only has the cost of storage dropped to nearly nothing, the threshold for using it has dropped even lower. I'm perfectly willing to commit a few megabytes every time I push the button on my digital camera. I remember college, where my mainframe disk quota was a mere 256K.
Today's challenge is to get mainframe-class recovery without bringing back to mainframe-style prices. Some of this is controlled by the way we USE data storage. And then there is all the "savings" we get from server consolidation. Everything we do to consolidate just makes storage management a bigger headache. The trick is to evolve not just the low-level, "invisible" management of storage, but the high level applications as well. If I don't truly NEED to have 10TB on a single mount point, perhaps I should have multiple volumes, distribute my storage, and find a way to be happy with twenty 500GB volumes instead. The easiest way to avoid the recovery time of a 10TB RAID set is to not build one.
I was in mainframe IT long before RAID was commonplace. We commonly faced limits of 450MB on indexed files, because that's as much as you could get from a hard drive back in the early 1980's. Modern Oracle DBAs must be scratching their heads at all of the tablespace management options that seem so redundant when you have RAID storage. This was the pre-RAID method of storage management, in which database container files could be of any size, mounted anywhere, and utilized in all sorts of creative ways to circumvent the hardware limitations of storage in those days. Today, it represents little more than an opportunity to inadvertently bring out the worst of both worlds by setting up these two storage methodologies in conflict with each other.
Having worked with plenty of enterprise grade raid (EMC symetrix, clarion, and Dell SAN devices) I can say that capacity and rebuild times are not a problem for high-end arrays.
What will bring the problem to the masses are these stupid consumer NAS boxes. It is very easy to build a 4 or 8 TB array for home use using relatively cheap hardware. Unfortunately, no home user/abuser, that I know, has the skill set to manage or protect such a large array of data.
My most recent experience with a Western Digital sharespace was awful. Here is a box with a Gigabit NIC, and 4 - 2TB hard drives in a RAID 5 array that has transfer rates around 9MB/sec at best. Combine that pitiful performance with a rebuild/reformat time of over two days - and you know where this is going.
Average joes are going to put their entire lives on these things and never back them up due to the time and space cost. When a failure does occur - it will take days to perform a rebuild of the array - vastly increasing the likelyhood of another failure and permanent data loss.
Crappy RAID's days are numbered - good RAID implementations will be with us as long as hard drives have ANY failure rate at all.
-ted
When a RAID Array becomes too large to be efficient, it's time to deploy some other storage management solutions. The idea is to keep RAID arrays small, manageable and reliable. As mentioned elsewhere in comments, storage today is inexpensive compared to the other costs of large projects.
From the commercial side I saw ibm's GDPS mentioned, but there is also the General Parallel File System from IBM which allows you to stripe your data over multiple arrays over multiple networked systems as well as to have multiple mirrors. The storage does not have to be RAID storage either.
Also, there is the XAM space for storage management in the Content Addressable Storage space. The current implementations are mostly commercial though Sun's (Oracle's now) Honey Comb project provides a version of the XAM interface.
There are also several open source storage management solutions that provide for redundancy and stability - Twisted Storage is one. Tahoe and XTreemFS are two others.
Pick the right technology for the job, there are solutions out there and I'm sure new ones on the way.
there is a storage platform called XIV, it distributes the data in what they call RAID-X Go check it out at xiv.com also EMC is using Enterprise Flash in their arrays in a Raid-5 config, and i think in 18-24 months we will see EFD (enterprise flash) be priced very closely with a 15K FC drive.
reference links
TwistedStorage
XTreemFS
Tahoe
XAM
GPFS
The chart he's using goes from SCSI, to fiberchannel, to SAS... to SATA. When you go from professional/server interfaces to hobby/desktop ones, of course the rebuild time skyrockets. If you did this article a few years ago and slid ATA in as the last data point instead of fiberchannel, you'd be seeing the knee showing up then instead of now. How about looking at 2010 and doing the calculations with 6 Gb SAS interconnect and 3 Gb drives, instead of 1.5 Gb SATA and 1 Gb drives?
Build multiple smaller arrays of drives and with the right file system, you then pool the storage, eg ZFS, LVM, btrfs, and many others. In theory as well (and practicality), you could do raid of raids of raids ... etc. (eg raid 5 array consiting of raid 5 arrays)
You also can have something like hadoop, or the various other solutions coming, which allows for clustering across your infrastructure.
Remember google does not have one single big massive array of disks running, to serve all of us peons.
..and the sooner people learn that, the better. Studying ZFS' design reveals a great deal about what RAID simply cannot achieve.
you had me at #!
Consider Google Docs.
If you have so much data that you're likely to encounter an error when rebuilding your RAID array, I don't think Google Docs is going to cut it.
Give me Classic Slashdot or give me death!
I just don't see evidence of this in real life.. I have a couple companies as customers who together are running six SANs from three different vendors. There must be about 100 raid arrays between them. We replace a drive about twice a year between all of them.. There is definitely no flaming ball of fire anywhere. We put in a new drive, it rebuilds over a couple hours and we are good for another six months. Plus I don't know about the other vendors but IBM is scaling upwards, too.. you get a few SANs under an SVC and you can migrate data from SAN to SAN at will.
Laws are rules for the court, but merely a bottom bar to hit for life. Think beyond laws in your actions always.
I don't see the situation really evolving for Raid.
A long time ago, big servers used to have around 50 disks totalling 1 GB of storage, MTBF of individual disks of 200,000 hours, some form of Raid, a bandwidth of 10Mb/s
Today's servers have also about 50 disks, totalling 40TB of storage, MTBF of individual disks ranging from 200,000 to 1.6 million hours, Raid 6, a bandwidth of 40Gb/s
So I fail to understand why Raid would become obsolete.
We are just adding extra layers of security by mirroring the data in different locations. But the Raid technology is not dead. servers, even those in the cloud still need Raid.
Not from weasels, though...
Eagles may soar, but weasels don't get sucked into jet engines.
Modern drives perform bad block replacement internally. By the time you actually see drive errors at the OS level, the drive has already experienced massive failures.
Filling the drive with helium should help; the speed of sound in helium is 3x higher than in air, and it offers less resistance.
(Hydrogen would be even better, but it has a tendency to interact with metals in unfortunate ways.)
There a number of Enterprise storage array vendors that have very good solutions. Last year my company looked at all the major storage vendors and we chose 3PAR for a number of reason including raid rebuild issues. They had two things going for them. 1) They only rebuild the blocks that are written to. 2) Since they wide stripe across all the drives in the array, they can rebuild from more drives. We have had 2 drive failures since the we purchased the array. Both times the applications did not notice any performance slow down during the failure and rebuild times. This was different from our other arrays we currently have.
While we selected 3PAR, there are a number of storage vendors that do something similar that would fit most companies need for quick rebuilds.
And that "new solution" is a few minutes older than RAID5 itself: RAID1. C'mon, people, RAID1 is the bees knees. Between RAID1, RAID0 (and various combinations of the two), the dirt-fucking-cheap huge-fucking-capacity disks you can get now, and high bus bandwidths, life is great. Everything I used to solve with RAID5, I now solve with RAID1, and it just doesn't have problems. Worried about another disk dying when you're rebuilding a new one? Just spring for the extra $100 (oh, the horrors) for a couple more disks. There's no rule that says you can't have 4 or 5 disk redundancy, and at today's prices, you can afford it. RAID is such a non-problem now, you just have to use the non-parity levels.
Ok, but how else could we improve things? More ECC and redundancy at the sector level. e.g. I'm willing to give up a lot more capacity for even more ECC. If I have 512 bytes of data, I'm ok with using 1024 bytes per disk: let both fail before you call it a disk failure. And those of you who cry about all this (and on top of multi-disk RAID1!) being so space inefficient, I want you to go to newegg and look at dollars per gi^H^Hterabyte. (Is it still terabytes today, or have they moved up to the next prefix yet? (joking, but when I go back and read this in 10 years, it won't sound like a joke anymore))
As copyright owner of this comment, I authorize everyone to defeat any technological measure which limits access to it.
Are you kidding me? Google Docs is not safer than raid. It has failed and been down for hours. At best it is as good as raid.
Yes, this is as cool as it sounds! Along with the implementation of the quantum computer (which have been speculated to crack pretty much every encryption algorithm known, because it's capable of sending every possible answer at the same time), the implementation of the Quantum hard drive has been explored. Different orientations of spin give 1's and 0's as well as both 1&2 spin (superposition factor). These properties could be used to create very dense and stable hard drives. Someday our HDD's will be the tape drives of today!
What is it(data) worth?
Define these failure types: minor, normal, major, critical, catastrophic.
What are the chances of ____ failure?
What is an acceptable downtown for _____ failure?
These are the questions one needs to ask (and others) before saying "Raid" or "Tape" or "Enterprise Solution" or .... whatever.
If we don't have answers to these questions, then the discussion is simply an exercise in mental masturbation. Simply put, we have to be able to effectively mitigate against the types of disasters by appropriate counter measures weighing against the cost of replacing the data and the likelihood of that disaster.
Losing data is critical to everyone. How much effort is needed to protect it, varies widely.
RAID is simply one level of protection. It isn't a backup, it isn't redundant data. It is just a means to mitigate against a certain kind of failure, nothing more, nothing less.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
RAID actually IS a valid solution to combat the problem of failing drives, even when the technology in those drives is a moving target. Ariel densities have gone up, meaning tolerances have to be tighter, which makes it harder to build a reliable drive than it was some time ago, and that trend will only continue to increase. RAID 5 specifically might not be the best solution for today's needs, but RAID 6 is a step in the right direction. And to call RAID 6 a bandaid is hardly fair. There is no storage methodology that you could put in place now that could be guaranteed valid 10 years from now. Besides, 1+0 is gaining popularity and there are extensions beyond that. RAID 1+0 in particular is much quicker in rebuild time than raid 5/6. You give up half your disk space to redundancy, but you take comparatively little performance hit during rebuilds and it's very fast in the day-to-day. Granted, in worst case scenario you are no more redundant than you would be with RAID 5 (i.e., you lost both disks in the same mirror before it could be rebuilt on a hot spare), but in the best case scenario you could lose up to half your disks and keep chugging along.
Is the parity drive a bottle neck to RAID performance? I wonder what the impact would be of using SSDs for the parity drives, and regular drives for the data drives? Assuming you either don't need the same size drive for the parity drives as you do for the data drives, or you simply don't need any more data than an SSD holds, and you can afford 1 or 2 SSDs but not enough for the entire array to be SSDs. :)
Faster to just copy it to a usb key. You have multiple copies of your data, and no longer have to worry about network latency, or even if there IS a network available.
Ermmm, how about simply moving to a smaller form factor drive. Most servers have moved to 2.5" bays already and I see no reason to doubt SAN vendors will start offering SFF shelves as well. Another approach is to just throw enough cheap disk at the problem that long background rebuilds aren't a concern. Multiple RAID sets off redundant storage stacks have better DR characteristics regardless.
That isn't to say alternate approaches to data integrity aren't called for as well. It's clear that future filesystems are going to include some level of end-to-end checksumming and offer software approaches to data replication. Likewise, there are plenty of approaches to data replication that don't follow traditional filesystem conventions; consider Google FS or CouchDB.
A meme that has been going around is that the size of the disk relative to the hard error rate is getting to the point where reading the entire disk once or twice guarantees getting a hard error. My experience with disks, though, is that they do not "wear out" by how many blocks you read from them. They do tend to wear out with age, vibration, heat, start/stop, etc. Is the hard error rate really based on number of bits read (and, it's also not a guarantee you will get a hard error - it's the minimum) or is it an estimate based on expected usage vs age? Or is it not related to the media at all but instead a measure of how many bits you can transfer over the link without an error?
This meme is really getting silly. Long rebuild times are a problem but we're still talking about days for devices that have typical life spans of years.
redundant array of redundant arrays of inexpensive discs...
RARAID
Donald 'Duck' Dunn: We had a band powerful enough to turn goat piss into gasoline.
Surely the HDD manufacturers can add extra error-correction data to the hard disks.With 2Tb to play with I wouldn't mind losing 5% (or even 10%) to error-correction.
No sig today...
The sealed drives we use now showed up in the '80s. Before that the platters were not part of the drive, they were in a plastic cover to keep the dust off. On the mainframes the cover held a stack of platter; on the minis there was just one or two 5mb platters inside. We would place the whole stack with cover into the drive, then rotate the handle to pull the cover out, leaving the spindle of platters in the drive. Then just close the dorr and push the button to spin it up.
In either those old open ones or the "new" sealed ones, the head flies on a cushion of air, but the distance from head to platter is microspic; a piece of dust is big in comparison. In the old open drives, if the head hit even a tiny piece of dirt, it could "crash" into the platter and gouge out a rip. If you haven't heard it, it was actually fairly loud and startling.
The big problem with RAID on bigger disks is that you're splitting data across N+1 disks, so if one disk fails catastrophically and needs to be rebuilt, you need to have good data on the remaining N disks, and as the disks become larger, the chances that all that data is good are no longer near-100%. So you're likely to get some bad data even with RAID, though you're not going to have as much bad data as you'd have without RAID, and you're still avoiding the problem that if you disk fails, you lose everything; you're just not guaranteed to lose nothing.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
You heard it here, remember me well.
deleting the extra space after periods so i can stay relevant, yeah.
Fry: What happened?
Dr. Zoidberg: All six thousand hulls have been breached.
Fry: Oh, the fools! Why didn't they build it with six thousand and one hulls? When will they learn?
There are three reasons you run raid, other than because it's cookl
SSDs are about speed, but don't have high capacity unless you've got an infinite budget. That's not the problem here. The problems are that
SSDs don't really help you on time, because you can't afford to run your whole shop on SSDs if it's big enough that RAID's not reliable. The recovery problem is helped somewhat by disk block error-correcting codes: you can detect if a block has corruption and rewrite/remap it when you're first storing it, so the ratio of bad blocks on disk isn't as bad as you'd expect from raw error rates, and you may need to continually recheck your raid block and rebuild bad blocks proactively rather than finding them after failures.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
It's about RAID controller speed, and that just isn't scaling as fast as disk capacity or business requirements for disk space.
To build or rebuild a RAID5 / RAID6 array is very expensive - in order to calculate parity on one drive you have have to read every other drive and XOR all of those bits together. Since in a RAID5 / RAID6 array there is no dedicated parity disk (parity is distributed across all drives) replacing one drive requires that every bit on every other drive is at either read or written during the rebuild process. That means that the rebuild time is O(n) on the number of disks in the array and also O(n) on the size of the disks in the array. BTW, the performance of a RAID5/6 array during a rebuild is terrible - the controller is flat out already.
To give you an example, a well known company I worked for a few years back was archiving event data onto a cluster of storage machines. The cluster itself was a specialized spatial database optimized for parallel searches, so that TBs of data could be searched in a second or two. IIRC, in 2004 each storage machine within that cluster contained 2TB RAID5 arrays split across 6 disks and a rebuild took most of a day. By 2007 each storage machine contained about 10TB split across 6 disks and a rebuild took most of a week.
Now consider time between failures. Carnegie Mellon did a study of 100000 drives during which MTBFs claimed by manufacturers was an average of 15 times more than that attained during the study. So that expensive SCSI 1.2 million hour MTBF really has a MTBF of about 100000 hours.
In a RAID array the MBTF of any single disk is the MBTF of a single disk divided by the number of disks. So that 100000 hours MBTF divided by 6 drives equates to 700 days between disk failures. 2 years sounds reasonable enough. But, iIn my former employer's cluster though there were 40 machines (and growing by about 4 each year). So the MBTF for a single disk in the entire cluster was about 17 days. On average, every 17 days someone had to babysit an array rebuild that took most of a week.
It gets worse though. The same Carnegie Mellon study showed a strong link between failure and disk age, ie your oldest drives are most likely to fail. That means that the chances of a second drive failing in the array before the rebuild is complete cannot be ignored, and indeed we saw this happen on 2 occasions. Fortunately backups existed and the data was recovered. By the time I left we were using raid 15 ( read performance is maintained during a rebuild of one of the constituent raid5 arrays ), and had an active data migration policy so that older machines were taken out of the cluster before all of their disks started failing.
I hate to think what problems Google must have with this.
I have seen rebuilding problems TWICE where a 2nd drive died or had sector errors during rebuild. I've also avoided problem but had drives in the RAID all dying or having errors within the same year of the 1st drive's problem. Its not comfortable to spend 6 months swapping out most the drives when you can only afford to lose 1 of them.
The solution is preemption. You replace drives every 3 years reguardless. see google's study on drive life. 5 years seems to be the furthest one should go and 3 years is safer. Its also true that a significant number of drives show trouble in the 1st month (under heavy use) and I was about to start a burn in policy before I left that job. We ordered 30-40 drives at a time.
Buying the disks at the same time is somewhat unavoidable because if you buy lots you get discounts; PO approval etc. I think the risks are low of multiple drive deaths. I would however replace a whole batch immediately if just 1 died from the lot (my new policy.) Key to this is tracking the drives by AGE and batch. Oh, I USED to swear by seagate.
Hardware raid may cost but just try getting back online when the raid controller DIES. Buy two. I prefer software RAID, especially on servers; screw pci drivers! I never was a fan of parity RAID; use RAID 10. You can lose up to HALF of all disks in RAID 10; sure, a specific half not random half... its all a gamble anyhow.
Ultimately the #1 question is always: "What is my data worth to me?"
Going in the future, I've been WISHING somebody will standardize drive clustering; hopefully with dumb controller boards on the disks and a standardized controller board. Sure its a hardware controller-(and centralized caching?)- but it doesn't have to be RAID it can be a block level manager without the limitations of RAID. If standardized, then the main problems of resolving controller failure and vendor lock would be gone; plus you could would have an upgrade path. Today, this can not be done because you get tied to a vendor, a series, or even just 1 model of controller board.
The problem will continue to grow to the point where people will want management not dumb RAIDs; then we might see a move towards a convention and maybe standardization.
Democracy Now! - uncensored, anti-establishment news
With disk space getting cheaper and cheaper, "N+1" plus transaction logging may be the way to go.
If a drive fails, mark that half of the mirror offline then when the drive is replaced, replay the log on that half.
Granted, this won't work if your throughput is so fast the log can never be replayed, but in most cases you'll have the drive replaced within 24 hours, if not within one.
You'll still have the rare case where both mirrors have a disk failure at the same time, in which case you'll degrade to RAID 5 or whatever RAID N you are using as a baseline.
In a large enterprise, it may pay to segregate your data between "gotta have it as fast as local storage," "gotta have it within fractions of a second," "gotta have it within seconds," "gotta have it within minutes," "gotta have it within an hour," "gotta have it within a day," and "gotta have it within a week" and buy your drives accordingly. "Gotta have it within minutes" data can be stored on drives that are usually powered off. "Gotta have it within an hour" can be on drives that are not even in the computer room and are fetched by a low-level clerk when needed. "Gotta have it within a day" data would be archived data in an offsite data center that can be retrieved fairly quickly. Everything else, mostly old data kept for legal reasons that probably nobody will ever read, would be in the "gotta have it within a week" bin.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Probably doubling as incandescent lights in the development lab. OK, that may be a bit of an exaggeration but have you ever touched a 15K RPM drive that's been running for a while? They get damned hot. I mean "burning your finger" hot. You don't use those in enclosures that aren't designed to provide a lot of air flow to prevent the drives from cooking themselves to death. (And more air flow inevitably means louder enclosures so most users will balk at deploying them in a non-data center environment.) Now imagine the heat dissipation and the cooling needed for a 30K RPM drive.
CUR ALLOC 20195.....5804M
So are we ready to move all our personal information to clouds? I certainly am not, but Google Docs are wildly popular and a lot of people are. I long ago learned that I can't look to myself to judge what the mainstream attitudes are in many things.
Mainstream... fads?
That's pretty silly...
Yes - running a home raid-5 array with 4 drives (3 for the array, 1 spare) - it's days are limited. The risk profile goes up because data density has grown faster than rebuild times. Fair enough.
That's why raid methods continue to evolve to keep up with the times, and enterprise solutions continue to become more sophisticated.
But we'll still be using "Redundant arrays of independent disks" for quite a while I imagine.
Raid may be coming to end, but its nothing to do with the speeds of drives. Its all about size. Its all about the fact that you are GUARANTEED to have a sector fail on read after x number of reads. Now because x hasnt changed much over the decades while drives have gotten vastly larger, we're approaching x but we've still got a while to go yet and in the enterprise (where raid actually matters) its not something that'll really be a problem for at least another 5-10 years. This is not a problem that raid 6 or zfs solve.
But, the problem is this, if a drive fails in a raid 5 array and the array runs off to rebuild it then when you get near x you'll fail during the rebuild and that will keep occuring until the array gives up. Raid 6 still has the same issue, it'll still read all the drives to rebuild the broken disk, same with ZFS. Unfortunately, there has been alot of industry spin suggesting raid 6 and ZFS solve this issue which is patently false.
Raid 6 was designed because of how arrays were being deployed, when large scale disk boxes were being deployed (SAN etc) we got to a point where if you had enough disks you might as well have an engineer on site because the failure rate of drives was such that when you got to a certain number of drives (we're talking hundred of thousands here, not that hard in a large data center packed with SAN arrays) you could guarentee at least one drive would fail EVERY SINGLE DAY. So, with raid sets getting larger (i.e. 10-15 drives in a single raid 5 array and they were getting much bigger than that) it became important to save yourself from a dual-drive failure (because the likelyhood of it happening went up exponentially as the set got larger), having two parity drives solved that issue - hence raid 6.
ZFS was designed for performance. Raid 5 (and 6) SUCKS for performance when your not using some form of off-load device (hardware raid card or SAN array head) and so ZFS was designed to (in part) give people the ability to do that kind of thing on the server cpu without killing the performance of having 6-10 striped drives. The problem is that if you throw some data at a drive, computers are very good at it. They can just say "write this chunk of data here please" and the rest just happens. With raid (if its done in software), what you have to do is calculate the parity of ever single byte of data on the array - this hurts alot and destroys the whole transfer mechanism.
The point is though, we're approaching x and the only real solution is to fix x. Though, with SSD's coming along quite nicely, the problem will likely resolve itself and we'll be using raid 6 SSD's. SSD's are quite capable of solving this little drama because x can be quite variable unlike their "spinning-rusty-metal" counterparts. They can also be scaled alot easier for physical size (something we havent seen yet). Its very hard to produce a tonne of different sizes when it comes to disks - i.e. we have three formats, 1.8", 2.5" and 3.25" and alot of that has to do with producing some "spinning-rusty-metal" in an efficient way at different sizes. i.e. thats just not practical for motorized components. SSD's on the other hand are a completely different boat. Because they are just chips on a little board they can scale very well in terms of physical size. Right now it hasnt happened, they are doomed to follow the pre-set sizes of their cousins because of the fact we're forced to use both.
This wont always be the case, eventually not only will SATA, FC, infiniband (an already mostly dead tech any way) and SAS be a thing of the past (They were designed for spinning rusty metal and with people like seagate threatening to sue ssd makers for using sata and the like, ssd boys will probably make their own interface). The traditional interfaces for HD's are not fantastic (as it turns out) for ssd's, you could in fact make a much better interface if your talking directly to ssd's. This also hasn't happened, but one day it will, it'll hit the consumer market and that will be the death of "spinning-rus
Your story claims RAIDs then you define it more precisely at RAID 3,5, or 6 but the talk about it again referring to all RAID types. Get a clue! Nothing wrong with RAID 10 or RAID 1 when it comes to reliability and rebuild times! On top of all this you claim RAID is old technology and state how it will be replaced but there is not replacement so how can this be? Stupid author!
Come on guys, just store everything in RAM.
If the point of RAID is access times then clearly RAM beats the hell out of all the competitors here.
If what you needs is uptime then the system is always on, thus RAM.
If you're worried about running out of space in your server rooms ram is alot smaller than HDDs.
If you want redundency just set mirrored tmpfses.
If you're worried about failure RAM failure rate is WAY lower than a HDD and has ECC built in, plus, hotplugging RAM is cake.
So, WHY, I ask, are we still using hard drives? What could possibly go wrong?
There we go. Somebody is talking about the article.
I'm not a storage expert, but something about this article seems half-baked. For example, looking at the rebuild table, from 2002 to 2005, rebuild time stayed roughly the same, while drive size went from 146gb to 300gb and bandwidth went from 89 to 119 MB/sec. Is there a problem with that?
Then, from 2005 to 2009, rebuild time dropped, even though disk size went from 300gb to 450gb, because the number of drives on the channel doubled from three to six. Again, is there a problem?
Now looking at his second table, I guess the author's point is that since rebuild times have increased from 5.28 hours in 1994, to 6.6 hours in 2009, and drives have gotten 100x more reliable (10e14 vs 10e16), the ponderously slow, 2-order of magnitude growth in drive reliability won't keep up with the dramatic 25% increase in rebuild times.
Is this the point he's trying to make?
ZFS seems to solve many issues with a higher-end block manager approach in the filesystem itself; it is like a smart software raid.
However, ZFS is not wide spread and outside of SUN not something I can trust for serious use at this time. Its also a server bound solution as well; some higher end hardware would be nice that had ZFS features like the high end solutions already out there but doesn't cost a fortune with vendor lock.
I have (for fun) created Raid 10 with triple redundancy but i never used it in production (due to cost.) Again with a RAID 10 you are using at least 4 disks. You COULD lose 2 disks and be ok as long as it is not the wrong pair of disks. It goes up the larger the stripe goes. Yes, the whole thing is gone if you happen to lose the wrong 2 drives. The other RAIDs can't lose 2 disks except raid 6 which isn't even available in many situations.
Democracy Now! - uncensored, anti-establishment news