Slashdot Mirror
RAID Vs. JBOD Vs. Standard HDDs
Ravengbc writes "I am in the process of planning and buying some hardware to build a media center/media server. While there are still quite a few things on it that I haven't decided on, such as motherboard/processor, and windows XP vs. Linux, right now my debate is about storage. I'm wanting to have as much storage as possible, but redundancy seems to be important too." Read on for this reader's questions about the tradeoffs among straight HDDs, RAID 5, and JBOD.
At first I was thinking about just putting in a bunch HDDs. Then I started thinking about doing a RAID array, looking at RAID 5. However, some of the stuff I was initially told about RAID 5, I am now learning is not true. Some of the limitations I'm learning about: RAID 5 drives are limited to the size of the smallest drive in the array. And the way things are looking, even if I gradually replace all of the drives with larger ones, the array will still read the original size. For example, say I have 3x500gb drives in RAID 5 and over time replace all of them with 1TB drives. Instead of reading one big 3tb drive, it will still read 1.5tb. Is this true? I also considered using JBOD simply because I can use different size HDDs and have them all appear to be one large one, but there is no redundancy with this, which has me leaning away from it. If y'all were building a system for this purpose, how many drives and what size drives would you use and would you do some form of RAID, or what?
At first I was thinking about just putting in a bunch HDDs. Then I started thinking about doing a RAID array, looking at RAID 5. However, some of the stuff I was initially told about RAID 5, I am now learning is not true. Some of the limitations I'm learning about: RAID 5 drives are limited to the size of the smallest drive in the array. And the way things are looking, even if I gradually replace all of the drives with larger ones, the array will still read the original size. For example, say I have 3x500gb drives in RAID 5 and over time replace all of them with 1TB drives. Instead of reading one big 3tb drive, it will still read 1.5tb. Is this true? I also considered using JBOD simply because I can use different size HDDs and have them all appear to be one large one, but there is no redundancy with this, which has me leaning away from it. If y'all were building a system for this purpose, how many drives and what size drives would you use and would you do some form of RAID, or what?
Wikipedia has a very informative article regarding RAID and the various levels, in fact here it is. http://en.wikipedia.org/wiki/RAID
Chicken fried butter sticks? Do
With Linux you can create a RAID5 md device, say /dev/md0, then run LVM on top of that (pvcreate /dev/md0 ; vgcreate MyVgName /dev/md0) and use that to carve out your storage. The key here is to create partitions on each drive, eg filling up the entire disk, and create your raid5 with those.
/dev/md1. So now you have /dev/md0, pointing to the first 500GB of each disk, and /dev/md1, pointing to the 2nd 500GB of each disk.
/dev/md1 and graft it onto your LVM volume group. (pvcreate /dev/md1 ; vgextend MyVgName /dev/md1). Now your LVM VG just doubled in size, and you can use all that new space. Whatever you do though, do NOT create any "striped" logical volumes (the "-i2" option to lvcreate; LVM's Poor Man's RAID0, basically) because you will suffer terrible performance, since you'll be striping across different volumes on the same physical spindles (a big no-no for any striped configuration). But if you use the extra space by creating new filesystems or growing existing ones, you shouldn't see any trouble.
If you buy 1TB drives further down the road, here's what you do- With each disk, create a partition identical in size to the partitions on the smaller disks, then allocate the rest of the space to a second partition.
Join the first partition of the disk to the existing RAID set. Let it rebuild. Swap the next drive, etc. etc. Then once you've done this switcharoo to all the drives, create another raid set using the 2nd partition on your new disks--call it
Take that
Just be sure that any replacement drives you have to buy... you must partition them out similarly. I'd recommend pulling back on the partition sizes a bit, maybe 5%, to account for any size differences between the drives you bought right now and some replacement drives you may purchase later on which might be slightly lower in capacity (different drive manufacturers often have differing exact capacities).
the real at&t mix
Go RAID5. RAID5 = Hardware failure resilience + maximum storage.
Go Linux. The Linux MD driver allows you to control how you RAID- over disks or partitions. there are advantages. We will discuss.
First, don't get suckered into a hardware RAID card. They are *NOT* really a hardware card- they rely on a software driver to do calculations on your CPU for RAID5 ops. Software RAID is JUST AS FAST. Unless you blow the big bucks for a card with a real dedicated ASIC to do the work, you're fooling yourself.
Now, you want to go Linux. By using the md driver, you can stripe over PARTITIONS, and not the whole disk. By doing this, you can get MAXIMUM storage capacity out of your disks, even in upgrades.
Say you have 3 500GB disks. You create a 1TB array, with 1 disk as parity. On each of these disks is a single partition, each the size of the drive. Now, you want to upgrade? SURE! Add 3 more disks. Create three partitions of EQUAL size to the original, and tack it on to the first array. Then, with the additional space, you can create a WHOLE NEW array, and now you have two seperate RAID5's, each redundant, each fully using your space.
Another advantage with MD is flexibility. In my setup, I use 5x 250 drives right now. On each is a 245GB partition, and a 5GB partition. I use RAID1 over the 5's, and RAID5 over the rest. Why? Because each drive is now independently bootable! Plus, I can run the array off two disks, upgrade the file system on the other 3, and if there's a problem, I can always revert to the original file system. So much flexibility, it's not even funny.
I recommend using plain old SATA, in conjunction with SATA drives, and just stick with the MD device. For increased performance, watch your motherboard selection. You could grab a server oriented board, with dedicated PCI buses for slots, and split the drives over the cards. Or, you can get a multiproc rig going, and assign processor affinity to the IRQ's- one card calls proc 1 for interrupts, the other card calls proc 0. If you have multiple buses, then performance is maximized.
The last benefit? Portability. If your hardware suffers a failure, then your software RAID can move to any other system. Using ANY hardware RAID setup will require you to use the EXACT same card no matter what to recover data. Even the firmware will have to stay stable or else your data can be kissed goodbye.
Windows? Forget about it.
Good luck!
I concur. You would be crazy not to have redundancy--without out it one disk failure will pull down a good chunk of your data.
As for growing the array. From what I understand (and I have not tested this) you can grow the size of the array if you replace all the disks (one at a time with a resync obviously). Also, as of linux 2.6.17, you can add a disk to the RAID and grow it that way.
I would caution against making your array very large (either in disks or in space). Consider the case of a 3 disk RAID array where each disk has a probability of failing in any given second of 10^-10 (you would do this analysis using the reconstruction time of your array as the time window). The probability for any two drives not failing is (1-10^-10)^2. The total number of 2 drive pairs in a 3 disk RAID is 3, thus the probability of the array not failing in any given second is (1-10^-10)^6=0.99999999940. Over a period of five years, the overall probability of no two drives failing is (1-10^-10)^(6*157680000)=0.909729. If you increase the array size to 10 disks, the overall probability of two drives failing is 0.241927 (the number of 2 drive combinations is 45 so you replace the 3 with 45).
echo check > /sys/block/md1/md/sync_action
3ware made some pretty good cards.
Eh?
LVM and RAID are orthogonal solutions, and don't do the same thing. LVM will let you make a single larger partition out of a number of real partitions, and before anyone says that's the same as RAID0, I should point out that RAID0 is not a real RAID level (as it has no redundancy). The circumstances for failure for LVM and RAID0 (JBOD too) are basically the same - if one part fails, you will quite possibly lose the whole lot.
As for hardware RAID, I would not necessarily recommend that either, as it moves the single point of failure without resolving the problem. Replacing a broken controller with something compatible some years down the road can prove impossible, especially with onboard controllers. There's also the fact that a number of RAID controller cards are buggy and others do most os the work in software drivers anyway! Performance is also no longer a reason to use a pure hardware RAID solution, especially now that multi-core machines are available cheaply.
Hot-swap is still someting that requires a good hardware solution, but that's about it. Good (and well supported) RAID products cost good money too, and for most of us it's just not worth doing - better to use software RAID, buy more RAM, and pocket the rest.
-- Steve
I think the comment about CPU performance was more about the fact that with faster CPUs, the speed benefit of a hardware raid solution isn't as useful. I checked the raid6 personality on my 1Ghz celeron, and it reports that it can calculate RAID parity at a throughput of 985MB/s, using the SSE parity calculation routines. That's more than you do any useful file serving with (it has to go out on the network, and that'll saturate gigabit ethernet). The I/O performance advantage of a hardware controller doesn't seem too useful here. I'm also not sure why software raid can't benefit from the multi-channel read performance available with raid 1, 5, etc.... Why can't the software issue a read command to two drives simultaneously? The comment about a buggy raid controller obviously wasn't talking about a new controller, but one that has eventually failed. I imagine they generally become obeselete before they fail, but even so, a failed controller really sucks if you need to figure out how to get the raid stack running on another controller. I had a quick look into it at one stage, and it broke my brain trying to work out what controllers worked with what....