Slashdot Mirror
What Kind Of Software RAID Are You Running?
ErikZ asks: "Lately, I'm having issues with my RAID. Specifically, closed source drivers for my RAID card that only support Red Hat 9. So I've decided to Ebay the card, and try to figure out how to turn 4 SATA drives into a software driven RAID 5 setup. Yes, I know I'll lose all the data, and I'm not worried about it. Finding a 4 port (or more) SATA controller card, that's well supported under Linux, has been difficult. Everyone wants to slap on their own RAID chip and charge you another 100$ for the pleasure. Where can a guy get a highly recommended, well supported, 4 port SATA card for Linux? The Rocket 1540 cards have vanished off the face of the earth. There are a few motherboards out there that have 4+ SATA connectors on them, but they also add RAID and some other cutting edge features that aren't well supported under Linux.
So, I thought I'd try another route and ask Slashdot: What are you using for your Linux software RAID needs? What do you suggest?"
Why not just let linux handle the raiding of the drives? No special hardware needed ourside of the drive controllers you already need to hook the drives up.
For more information check: man md
Also RAID 5 is distributed parity raid, no data loss if only one drive goes. it takes two failures to lose data on a raid 5 array.
I like this one: MegaRAID SATA 150-4. Admittedly, I've only used it under OS X Server, as it's apparently what Apple uses in their OEM; but they do have linux drivers and I can only assume that they work as well, if not better. Straightforward setup on the CLI, and not too expensive.
Personally, for $300 I wouldn't screw around with a software raid unless this is your own personal box and the drives only have MP3s.
--
$tar -xvf
The best option for real hardware SATA (or IDE) RAID in Linux is 3ware, bar none. Their drivers have been in the official Linux kernel since the early 2.4 days, and they just work. Highly recommended.
Why real hardware RAID? Say, for example, your boot drive goes out in a software RAID configuration. Your system is suddenly unable to boot, requiring manual intervention for a rebuild. With hardware RAID, the BIOS built-in to the card handles things smoothly and your system can boot without a problem.
Of all the PC RAID card manufacturers, 3Ware has the best reputation. However, you cannot boot from one drive in a 2-drive mirror. If for some reason you don't have a working 3Ware card, you cannot get your data. It is lost.
If you use 3Ware cards, keep one or two spare cards.
Why not a rocketraid 1640? They support 4 SATA drives and support (so they claim) Linux. I've run a Highpoint card under FreeBSD with no problems whatsoever...well, the management software won't work, but, hey, I can check things with the command line....
-- Fugacity: Confusing chemists since 1908
I took some notes when I was setting up my home 4x250GB RAID 5 server. I found there to be three categories of RAID solutions. Might be helpful for you in deciding. Copied them below and added a few extra comments.
t ml#ss4.1
It really depends on what you are using the server for. I ended up going for the pure Software RAID option. Its for home and I'm a cheap. If you're not cheap or it is for a work server, I'd stick with the pure Hardware solutions.
________
Hardware RAID:
The expensive Adaptec, 3ware, etc SCSI cards found in most servers.
Pro - Offloads XOR calculations from the CPU to internal processor.
Pro - No manual intervention required in case of a raid failure.
Con - Expensive.
Con - Third Party and/or closed source drivers often required.
Semi-Hardware RAID:
These are typically the SATA RAID controllers that are built into motherboards but includes cheapo add-on cards. Generally if less than 150 bucks, not full hardware RAID. I believe all of the RocketRAID cards fall in this section.
Pro - No manual intervention in case of a disk failure.
Pro - Cheap.
Con - Minimal or No CPU offloading.
Con - Third Party and/or closed source drivers often required.
Software RAID:
Use Linux and plain old SATA/PATA controllers to handle all of your RAID needs.
Pro - Very cheap.
Pro - No worry about driver incompatiblity or closed source drivers.
Con - No CPU offloading. You essentially trade CPU power for Disk speed/redundancy... and its a significant trade.
Con - Manual intervetion required in case of disk failure.
Con - PATA Only. Must be one drive per channel! NO SLAVES! Apparently data loss can occur on both drives in the chain if one goes bad. http://www.tldp.org/HOWTO/Software-RAID-HOWTO-4.h
Performance is also hurt in a Master/Slave combo.
________
The general consensus among linux kernel engineers and software RAID users is:
1. As long as the onboard SATA chip is well supported on your linux kernel, use the onboard chip.
2. Don't worry about the "hardware RAID" built into the motherboard. You don't have to use it. In fact, most people bypass it.
3. Use the non-BIOS SATA driver for your motherboard. Some motherboards have two different chips. Mine (an Epox 8RDA+Pro nForce Ultra2/400) uses both the common Silicon Image SIL3114 which supports 4 SATA drives and an additional 2 SATA drives provided by the onboard nForce 2 Ultra Gigabit MCP chipset. Quite nice for RAID and I still have normal PATA IDE drives 0 - 3.
4. Quite often the SATA RAID hardware only supports RAID 0,1 and 10 (or 01 depending). If you're looking for RAID 5 then you'll have to buy a more expensive outboard solution. The problem with outboard solutions are that they will eat into your PCI bandwidth. If you will be using PCI-X then you will probably also be paying significantly more for your outboard solution. Most people have a ton of CPU lying around, so handing off the I/O doesn't really buy you that much.
5. When it comes down to it you might as well just use software RAID because you have more control over it. You can use the onboard SATA controllers which allow you to take advantage of the increased on-motherboard bandwidth as well as having a significantly less expensive solution.
6. Another advantage to using Linux software RAID is that you don't have to learn a new RAID management system everytime you upgrade your machine and controller. You can also connect to your machine remotely and manage your raid system through a firewall. Sometimes you can do that with your hardware RAID system and sometimes you need to manage it from the BIOS itself.
7. Once you get comfortable with software RAID you can experiment with mixing and matching various I/O systems underneath it. One of the things I'd like to play with would be using software RAID with Firewire 800 external drives in a pseudo-SCSI arrangement.
8. The LVM2 system doesn't need software RAID, but it works very nicely with it none-the-less and gives you snapshot support etc.
9. Personally, I'm going for RAID 10 (striped mirroring) because drives have gotten very inexpensive and I don't mind burning a few more to get higher I/O rates. Remember, if you go with a mixture of RAID 0 and 1 then you want a striping over mirroring -- that way if you have a single drive failure the array keeps going.
Have fun and don't use RAID instead of backups. Backups save the stuff that you deleted intentionally but need to recover.
Software RAID: ...
t ml#ss4.1
Use Linux and plain old SATA/PATA controllers to handle all of your RAID needs.
Con - PATA Only. Must be one drive per channel! NO SLAVES! Apparently data loss can occur on both drives in the chain if one goes bad. http://www.tldp.org/HOWTO/Software-RAID-HOWTO-4.h
Performance is also hurt in a Master/Slave combo.
Um...why PATA only? I've done software raid using loopback devices. I can't see why SATA drives would be any less likely to work.
tasks(723) drafts(105) languages(484) examples(29106)
Nothing fancy... just a RAID-1 set, using Linux md.
Since a desktop/workstation machine does mostly reads anyway, I am getting the benefit of striped reads. I don't really care that my writes incur a slight penalty.
Granted, hardware RAID would use less CPU time... but hardware RAID is tied to a particular card. What happens if you move your disks to a new machine? You have to move the RAID card. If you go with an integrated RAID solution on your motherboard, that's tough.
With Linux md RAID, that is not a problem. Just plug the drives into your new machine and go.
Be seeing you.
scott
Sorry that was just a confusing line. In my original post change "PATA Only" to "This Con only applies to PATA drives". Then it should make more sense.
3ware is definitely the way to go. We use them exclusively where I work for SATA RAID, and Adaptec for SCSI. Works in Linux, FreeBSD, and Windows, has REAL hardware RAID (unlike the Promise cards), and gives you great management tools, both in firmware and software. The only possible issue you might have is that they are certainly not the cheapest solution. Then again, how much is it worth to you not to be called about an array failure at 4 AM?
Also, since you're using SATA, why only 4 drives? Even with the Raptors, that doesn't seem like a high-performance setup. The minimum we have in a more than RAID 1 (simple mirror) setup is 6 drives, and those are 15k U320 drives, for SATA we have at least 8 drives. If you're going to eat the time on rotational latency, at least make it up with more spindles, and don't forget to have at least one hot-standby drive (which would also require you to have a decent card/software RAID setup, cheap cards don't do hot-standby).
--That's the point of being root, you can do anything you want, even if it's stupid.
> Software RAID:
/boot is raid1 across 5 disks, the rest is raid5. Read performance for a 5 disk raid1 would probably be fantastic :)
/boot is still readable.
> Con - Manual intervetion required in case of disk failure.
You can get around this for some failure modes, as long as your boot partition is always raid1. I do this at home,
The success of this depends on the disk either failing so badly that the system can't see it anymore and so boots off another disk, or that the part of the failed disk that holds
dmraid, apparently. Once you've migrated the data off your array, do yourself a favour, and rebuild it using Linux md RAID. :-)
hardware RAID has these advantages:
/
... your raid sets? I don't know, you'll have to investigate this yourself.
1) offloads operations to the controller, so eats less CPU/IO bandwidth.
2) can have battery backed cache
3) often looks like "just a scsi controller" to linux and the boot loaders, so booting from f.e. a RAID5 set is often easier.
software RAID has these advantages
1) is cheaper
2) CPU time lost makes hardly any difference
3) has well-tested and supported tools to manage your raid setup. (imagine if you could only set up your raid sets by rebooting and entering the raid bios)
4) disk-layout is non-proprietary (controller died? don't have the same brand lying around? manufacturer left the market? no problem!) - so all-around more flexibility.
Look here for properly supported sata disk controllers:
http://linux.yyz.us/sata/
Some of these cards come with BIOS smarts that provides you with software raid which offer you the advantage of point 3) of hardware raid, ie: bios and boot loader support for your raid.
however, this does mean that the on-disk layout has to be recognized in linux, so linux can make sense of it and set up the raid sets properly. In linux 2.4 there were some drivers that did that themselves, however for linux 2.6 there's now a little userspace program that recognizes a whole bunch of on-disk layouts, and sets them up using the device-mapper facility (part of LVM2).
The advantage of this is that you can use the same well-tested and -supported linux drivers mentioned on http://linux.yyz.us/sata/ , but still use the (bios) facilities provided by the hardware. Another advantage is that this program will probably be used by all ATARAID ("mostly-software-raid") devices on linux, so it is, or will be well-tested and -supported in itself.
You can find this program, called DMRAID here:
http://people.redhat.com/~heinzm/sw/dmraid
So if you decide to go the SW-RAID way, think and decide if you want the advantage of dmraid. I haven't tried this myself yet, and the only aspect I'm unsure of is the management aspect of it (like with HW-RAID drivers) - DMRAID doesn't use MDADM, so how can you properly monitor, hot-add,
MDADM itself isn't going away any time soon either, if I understand correctly. (And even if it does, it's probably very likely that they'll make DMRAID understand the MDADM on-disk layout to provide an upgrade path.)
If however you decide to go the HW-RAID way, make sure you get a reliable and reputable manufacturer - with open source drivers (!), preferably with a known on-disk layout, and be prepared to spend money. I've heard a lot about 3ware, but I have no direct experience with them myself, so I can't vouch for them.
"With striping, I lose $20 for buying two 80 gig drives instead of one 160 gig drive, and I get twice the speed."
I found on my system that the difference between 2 x 80gig drives and a 160 was only about 33% faster, due to the higher data density and associated transfer rate on the larger platters.
Still, if you're going to stripe you can still challenge the data transfer rates of much more expensive hard discs... I'm going to trade up to a brace of 250's before too long.
Maybe if you're using a 486.
The CPU overhead of software RAID 5 is insignificant on any remotely modern machine. Even "ancient" ca. 500Mhz P3s have checksumming speeds over 1GB/sec