Slashdot Mirror
What Kind Of Software RAID Are You Running?
ErikZ asks: "Lately, I'm having issues with my RAID. Specifically, closed source drivers for my RAID card that only support Red Hat 9. So I've decided to Ebay the card, and try to figure out how to turn 4 SATA drives into a software driven RAID 5 setup. Yes, I know I'll lose all the data, and I'm not worried about it. Finding a 4 port (or more) SATA controller card, that's well supported under Linux, has been difficult. Everyone wants to slap on their own RAID chip and charge you another 100$ for the pleasure. Where can a guy get a highly recommended, well supported, 4 port SATA card for Linux? The Rocket 1540 cards have vanished off the face of the earth. There are a few motherboards out there that have 4+ SATA connectors on them, but they also add RAID and some other cutting edge features that aren't well supported under Linux.
So, I thought I'd try another route and ask Slashdot: What are you using for your Linux software RAID needs? What do you suggest?"
Why not just let linux handle the raiding of the drives? No special hardware needed ourside of the drive controllers you already need to hook the drives up.
For more information check: man md
Also RAID 5 is distributed parity raid, no data loss if only one drive goes. it takes two failures to lose data on a raid 5 array.
The best option for real hardware SATA (or IDE) RAID in Linux is 3ware, bar none. Their drivers have been in the official Linux kernel since the early 2.4 days, and they just work. Highly recommended.
Why real hardware RAID? Say, for example, your boot drive goes out in a software RAID configuration. Your system is suddenly unable to boot, requiring manual intervention for a rebuild. With hardware RAID, the BIOS built-in to the card handles things smoothly and your system can boot without a problem.
I took some notes when I was setting up my home 4x250GB RAID 5 server. I found there to be three categories of RAID solutions. Might be helpful for you in deciding. Copied them below and added a few extra comments.
t ml#ss4.1
It really depends on what you are using the server for. I ended up going for the pure Software RAID option. Its for home and I'm a cheap. If you're not cheap or it is for a work server, I'd stick with the pure Hardware solutions.
________
Hardware RAID:
The expensive Adaptec, 3ware, etc SCSI cards found in most servers.
Pro - Offloads XOR calculations from the CPU to internal processor.
Pro - No manual intervention required in case of a raid failure.
Con - Expensive.
Con - Third Party and/or closed source drivers often required.
Semi-Hardware RAID:
These are typically the SATA RAID controllers that are built into motherboards but includes cheapo add-on cards. Generally if less than 150 bucks, not full hardware RAID. I believe all of the RocketRAID cards fall in this section.
Pro - No manual intervention in case of a disk failure.
Pro - Cheap.
Con - Minimal or No CPU offloading.
Con - Third Party and/or closed source drivers often required.
Software RAID:
Use Linux and plain old SATA/PATA controllers to handle all of your RAID needs.
Pro - Very cheap.
Pro - No worry about driver incompatiblity or closed source drivers.
Con - No CPU offloading. You essentially trade CPU power for Disk speed/redundancy... and its a significant trade.
Con - Manual intervetion required in case of disk failure.
Con - PATA Only. Must be one drive per channel! NO SLAVES! Apparently data loss can occur on both drives in the chain if one goes bad. http://www.tldp.org/HOWTO/Software-RAID-HOWTO-4.h
Performance is also hurt in a Master/Slave combo.
________
The general consensus among linux kernel engineers and software RAID users is:
1. As long as the onboard SATA chip is well supported on your linux kernel, use the onboard chip.
2. Don't worry about the "hardware RAID" built into the motherboard. You don't have to use it. In fact, most people bypass it.
3. Use the non-BIOS SATA driver for your motherboard. Some motherboards have two different chips. Mine (an Epox 8RDA+Pro nForce Ultra2/400) uses both the common Silicon Image SIL3114 which supports 4 SATA drives and an additional 2 SATA drives provided by the onboard nForce 2 Ultra Gigabit MCP chipset. Quite nice for RAID and I still have normal PATA IDE drives 0 - 3.
4. Quite often the SATA RAID hardware only supports RAID 0,1 and 10 (or 01 depending). If you're looking for RAID 5 then you'll have to buy a more expensive outboard solution. The problem with outboard solutions are that they will eat into your PCI bandwidth. If you will be using PCI-X then you will probably also be paying significantly more for your outboard solution. Most people have a ton of CPU lying around, so handing off the I/O doesn't really buy you that much.
5. When it comes down to it you might as well just use software RAID because you have more control over it. You can use the onboard SATA controllers which allow you to take advantage of the increased on-motherboard bandwidth as well as having a significantly less expensive solution.
6. Another advantage to using Linux software RAID is that you don't have to learn a new RAID management system everytime you upgrade your machine and controller. You can also connect to your machine remotely and manage your raid system through a firewall. Sometimes you can do that with your hardware RAID system and sometimes you need to manage it from the BIOS itself.
7. Once you get comfortable with software RAID you can experiment with mixing and matching various I/O systems underneath it. One of the things I'd like to play with would be using software RAID with Firewire 800 external drives in a pseudo-SCSI arrangement.
8. The LVM2 system doesn't need software RAID, but it works very nicely with it none-the-less and gives you snapshot support etc.
9. Personally, I'm going for RAID 10 (striped mirroring) because drives have gotten very inexpensive and I don't mind burning a few more to get higher I/O rates. Remember, if you go with a mixture of RAID 0 and 1 then you want a striping over mirroring -- that way if you have a single drive failure the array keeps going.
Have fun and don't use RAID instead of backups. Backups save the stuff that you deleted intentionally but need to recover.
hardware RAID has these advantages:
/
... your raid sets? I don't know, you'll have to investigate this yourself.
1) offloads operations to the controller, so eats less CPU/IO bandwidth.
2) can have battery backed cache
3) often looks like "just a scsi controller" to linux and the boot loaders, so booting from f.e. a RAID5 set is often easier.
software RAID has these advantages
1) is cheaper
2) CPU time lost makes hardly any difference
3) has well-tested and supported tools to manage your raid setup. (imagine if you could only set up your raid sets by rebooting and entering the raid bios)
4) disk-layout is non-proprietary (controller died? don't have the same brand lying around? manufacturer left the market? no problem!) - so all-around more flexibility.
Look here for properly supported sata disk controllers:
http://linux.yyz.us/sata/
Some of these cards come with BIOS smarts that provides you with software raid which offer you the advantage of point 3) of hardware raid, ie: bios and boot loader support for your raid.
however, this does mean that the on-disk layout has to be recognized in linux, so linux can make sense of it and set up the raid sets properly. In linux 2.4 there were some drivers that did that themselves, however for linux 2.6 there's now a little userspace program that recognizes a whole bunch of on-disk layouts, and sets them up using the device-mapper facility (part of LVM2).
The advantage of this is that you can use the same well-tested and -supported linux drivers mentioned on http://linux.yyz.us/sata/ , but still use the (bios) facilities provided by the hardware. Another advantage is that this program will probably be used by all ATARAID ("mostly-software-raid") devices on linux, so it is, or will be well-tested and -supported in itself.
You can find this program, called DMRAID here:
http://people.redhat.com/~heinzm/sw/dmraid
So if you decide to go the SW-RAID way, think and decide if you want the advantage of dmraid. I haven't tried this myself yet, and the only aspect I'm unsure of is the management aspect of it (like with HW-RAID drivers) - DMRAID doesn't use MDADM, so how can you properly monitor, hot-add,
MDADM itself isn't going away any time soon either, if I understand correctly. (And even if it does, it's probably very likely that they'll make DMRAID understand the MDADM on-disk layout to provide an upgrade path.)
If however you decide to go the HW-RAID way, make sure you get a reliable and reputable manufacturer - with open source drivers (!), preferably with a known on-disk layout, and be prepared to spend money. I've heard a lot about 3ware, but I have no direct experience with them myself, so I can't vouch for them.
The ol' software raid 1 boot trick depends highly on the behavior of your BIOS under a failed drive condition. This is not the same thing as you get when you unplug a drive. Some BIOS may boot fine; some may not boot from the 2nd hard drive if the first is still attached and failing. It may also depend on how your drive has failed. If the drive electronics are failed and shorting the wrong pins on your IDE controller, then you may not get past the drive detection code in the BIOS at all.
This is simply one advantage to using a real hardware raid card like the 3ware. There are plenty of other reasons too: Does your chipset/hardware support hot swapping? If you use SATA, does it support command queueing? Do your drives? How much cache does it have? Does it have cache? Can it tolerate all types of hardware failure? Does it have *ahem* 16 ports with individual controllers for each drive? It's not like the BIOS/IDE chipset makers write out in their specs how their hardware performs under drive failure conditions so you have the overhead of testing each configuration to make sure it works proeprly before you have to rely on it. It's not so much a performance difference between hardware and software raid (until RAID-5 anyway) but an issue with how the hardware will respond when something goes wrong, which is one of the primary reasons for using anything above RAID-0 anyway.
Yes, running a 3ware card costs more. There are times when that $400 costs a lot less than the time spent configuring and testing an alternative software-only implementation. There are times when it doesn't and spending another $400 doesn't make a lot of sense. I have run both setups. I have machines deployed with both IDE software-only RAID arrays, IDE 3ware arrays, SCSI software RAID5's, SCSI Adaptec RAID's etc.. it's all application specific. There's no reason to call somebody a goon for recommending 3ware hardware. It's really good hardware; maybe you should try it sometime.