Slashdot Mirror
What Kind Of Software RAID Are You Running?
ErikZ asks: "Lately, I'm having issues with my RAID. Specifically, closed source drivers for my RAID card that only support Red Hat 9. So I've decided to Ebay the card, and try to figure out how to turn 4 SATA drives into a software driven RAID 5 setup. Yes, I know I'll lose all the data, and I'm not worried about it. Finding a 4 port (or more) SATA controller card, that's well supported under Linux, has been difficult. Everyone wants to slap on their own RAID chip and charge you another 100$ for the pleasure. Where can a guy get a highly recommended, well supported, 4 port SATA card for Linux? The Rocket 1540 cards have vanished off the face of the earth. There are a few motherboards out there that have 4+ SATA connectors on them, but they also add RAID and some other cutting edge features that aren't well supported under Linux.
So, I thought I'd try another route and ask Slashdot: What are you using for your Linux software RAID needs? What do you suggest?"
Why not just let linux handle the raiding of the drives? No special hardware needed ourside of the drive controllers you already need to hook the drives up.
For more information check: man md
Also RAID 5 is distributed parity raid, no data loss if only one drive goes. it takes two failures to lose data on a raid 5 array.
It's been running my raid for a couple of years...I think. I cannot honestly remember when I first configured it.
There are some things to be aware of. If you want to mount / as a raid, it can be tricky. The initrd needs to be properly configured, or the drivers must be built into the kernel.
Sometimes, the raids don't shut-down completely. I've never been able to completely solve this problem. Most of the time it's OK, but some machines have trouble. The most common culprit has been NFS.
GRUB & LILO need some special care when you plan to mount root from a raid mirror. It does work and it has all of the appropriate relibility features.
I like this one: MegaRAID SATA 150-4. Admittedly, I've only used it under OS X Server, as it's apparently what Apple uses in their OEM; but they do have linux drivers and I can only assume that they work as well, if not better. Straightforward setup on the CLI, and not too expensive.
Personally, for $300 I wouldn't screw around with a software raid unless this is your own personal box and the drives only have MP3s.
--
$tar -xvf
But can't you just use your raid card as a SATA card, and ignore the raid functionality? Why do you absolutely need it to be non-RAID? I'm sorry, but I'm having real trouble understanding what's the difficulty here...
The best option for real hardware SATA (or IDE) RAID in Linux is 3ware, bar none. Their drivers have been in the official Linux kernel since the early 2.4 days, and they just work. Highly recommended.
Why real hardware RAID? Say, for example, your boot drive goes out in a software RAID configuration. Your system is suddenly unable to boot, requiring manual intervention for a rebuild. With hardware RAID, the BIOS built-in to the card handles things smoothly and your system can boot without a problem.
You know you've seen one too many memes on LiveJournal and other such places when you just read that "What Kind of Software RAID Are You?" and think somebody's posted a link to a quiz.
Warning: Apple/Nintendo fangirl. Likes her electronics cute & cuddly. May be rabid.
Of all the PC RAID card manufacturers, 3Ware has the best reputation. However, you cannot boot from one drive in a 2-drive mirror. If for some reason you don't have a working 3Ware card, you cannot get your data. It is lost.
If you use 3Ware cards, keep one or two spare cards.
Why not a rocketraid 1640? They support 4 SATA drives and support (so they claim) Linux. I've run a Highpoint card under FreeBSD with no problems whatsoever...well, the management software won't work, but, hey, I can check things with the command line....
-- Fugacity: Confusing chemists since 1908
A righteous and just one against those godless peer-to-peer commies, thank you very much.
I took some notes when I was setting up my home 4x250GB RAID 5 server. I found there to be three categories of RAID solutions. Might be helpful for you in deciding. Copied them below and added a few extra comments.
t ml#ss4.1
It really depends on what you are using the server for. I ended up going for the pure Software RAID option. Its for home and I'm a cheap. If you're not cheap or it is for a work server, I'd stick with the pure Hardware solutions.
________
Hardware RAID:
The expensive Adaptec, 3ware, etc SCSI cards found in most servers.
Pro - Offloads XOR calculations from the CPU to internal processor.
Pro - No manual intervention required in case of a raid failure.
Con - Expensive.
Con - Third Party and/or closed source drivers often required.
Semi-Hardware RAID:
These are typically the SATA RAID controllers that are built into motherboards but includes cheapo add-on cards. Generally if less than 150 bucks, not full hardware RAID. I believe all of the RocketRAID cards fall in this section.
Pro - No manual intervention in case of a disk failure.
Pro - Cheap.
Con - Minimal or No CPU offloading.
Con - Third Party and/or closed source drivers often required.
Software RAID:
Use Linux and plain old SATA/PATA controllers to handle all of your RAID needs.
Pro - Very cheap.
Pro - No worry about driver incompatiblity or closed source drivers.
Con - No CPU offloading. You essentially trade CPU power for Disk speed/redundancy... and its a significant trade.
Con - Manual intervetion required in case of disk failure.
Con - PATA Only. Must be one drive per channel! NO SLAVES! Apparently data loss can occur on both drives in the chain if one goes bad. http://www.tldp.org/HOWTO/Software-RAID-HOWTO-4.h
Performance is also hurt in a Master/Slave combo.
________
The general consensus among linux kernel engineers and software RAID users is:
1. As long as the onboard SATA chip is well supported on your linux kernel, use the onboard chip.
2. Don't worry about the "hardware RAID" built into the motherboard. You don't have to use it. In fact, most people bypass it.
3. Use the non-BIOS SATA driver for your motherboard. Some motherboards have two different chips. Mine (an Epox 8RDA+Pro nForce Ultra2/400) uses both the common Silicon Image SIL3114 which supports 4 SATA drives and an additional 2 SATA drives provided by the onboard nForce 2 Ultra Gigabit MCP chipset. Quite nice for RAID and I still have normal PATA IDE drives 0 - 3.
4. Quite often the SATA RAID hardware only supports RAID 0,1 and 10 (or 01 depending). If you're looking for RAID 5 then you'll have to buy a more expensive outboard solution. The problem with outboard solutions are that they will eat into your PCI bandwidth. If you will be using PCI-X then you will probably also be paying significantly more for your outboard solution. Most people have a ton of CPU lying around, so handing off the I/O doesn't really buy you that much.
5. When it comes down to it you might as well just use software RAID because you have more control over it. You can use the onboard SATA controllers which allow you to take advantage of the increased on-motherboard bandwidth as well as having a significantly less expensive solution.
6. Another advantage to using Linux software RAID is that you don't have to learn a new RAID management system everytime you upgrade your machine and controller. You can also connect to your machine remotely and manage your raid system through a firewall. Sometimes you can do that with your hardware RAID system and sometimes you need to manage it from the BIOS itself.
7. Once you get comfortable with software RAID you can experiment with mixing and matching various I/O systems underneath it. One of the things I'd like to play with would be using software RAID with Firewire 800 external drives in a pseudo-SCSI arrangement.
8. The LVM2 system doesn't need software RAID, but it works very nicely with it none-the-less and gives you snapshot support etc.
9. Personally, I'm going for RAID 10 (striped mirroring) because drives have gotten very inexpensive and I don't mind burning a few more to get higher I/O rates. Remember, if you go with a mixture of RAID 0 and 1 then you want a striping over mirroring -- that way if you have a single drive failure the array keeps going.
Have fun and don't use RAID instead of backups. Backups save the stuff that you deleted intentionally but need to recover.
Software RAID: ...
t ml#ss4.1
Use Linux and plain old SATA/PATA controllers to handle all of your RAID needs.
Con - PATA Only. Must be one drive per channel! NO SLAVES! Apparently data loss can occur on both drives in the chain if one goes bad. http://www.tldp.org/HOWTO/Software-RAID-HOWTO-4.h
Performance is also hurt in a Master/Slave combo.
Um...why PATA only? I've done software raid using loopback devices. I can't see why SATA drives would be any less likely to work.
tasks(723) drafts(105) languages(484) examples(29106)
Nothing fancy... just a RAID-1 set, using Linux md.
Since a desktop/workstation machine does mostly reads anyway, I am getting the benefit of striped reads. I don't really care that my writes incur a slight penalty.
Granted, hardware RAID would use less CPU time... but hardware RAID is tied to a particular card. What happens if you move your disks to a new machine? You have to move the RAID card. If you go with an integrated RAID solution on your motherboard, that's tough.
With Linux md RAID, that is not a problem. Just plug the drives into your new machine and go.
Be seeing you.
scott
I've considered setting up software raid on my Linux server, but I haven't found any doc yet about what happens in the event of an unexpected crash or poweroff part way through writing a RAID-5 stripe.
Suppose I have 4+1 disks in a RAID-5 configuration, and during a write to a stripe of the disk only two of the disks are written to before the system crashes. This leaves me with 2 disks with new content, 2 disks with old content, and a useless parity.
I found a page at RedHat that indicates that as of 2001 there was no multistage commit. Has anything changed since then? Do the Linux MD tools address this?
Sorry that was just a confusing line. In my original post change "PATA Only" to "This Con only applies to PATA drives". Then it should make more sense.
3ware is definitely the way to go. We use them exclusively where I work for SATA RAID, and Adaptec for SCSI. Works in Linux, FreeBSD, and Windows, has REAL hardware RAID (unlike the Promise cards), and gives you great management tools, both in firmware and software. The only possible issue you might have is that they are certainly not the cheapest solution. Then again, how much is it worth to you not to be called about an array failure at 4 AM?
Also, since you're using SATA, why only 4 drives? Even with the Raptors, that doesn't seem like a high-performance setup. The minimum we have in a more than RAID 1 (simple mirror) setup is 6 drives, and those are 15k U320 drives, for SATA we have at least 8 drives. If you're going to eat the time on rotational latency, at least make it up with more spindles, and don't forget to have at least one hot-standby drive (which would also require you to have a decent card/software RAID setup, cheap cards don't do hot-standby).
--That's the point of being root, you can do anything you want, even if it's stupid.
> Software RAID:
/boot is raid1 across 5 disks, the rest is raid5. Read performance for a 5 disk raid1 would probably be fantastic :)
/boot is still readable.
> Con - Manual intervetion required in case of disk failure.
You can get around this for some failure modes, as long as your boot partition is always raid1. I do this at home,
The success of this depends on the disk either failing so badly that the system can't see it anymore and so boots off another disk, or that the part of the failed disk that holds
dmraid, apparently. Once you've migrated the data off your array, do yourself a favour, and rebuild it using Linux md RAID. :-)
On my systems, I have a software RAID-1 "boot drive."
.)
If one drive in the pair fails, things keep ticking along smoothly. They're really just identical partitions with identical data on different disks.
LILO merrily writes boot code to the array without episode. Meanwhile, the machine's BIOS is happy to boot from disks other than primary-master, all by itself.
I've booted the system after randomly unplugging devices. It works just fine.
Why do all of you 3ware goons think that the world wants to buy hardware which offers no clear advantage over having no hardware at all? (As if I want to add -more- potential points of failure to my systems . .
Kid-proof tablet..
hardware RAID has these advantages:
/
... your raid sets? I don't know, you'll have to investigate this yourself.
1) offloads operations to the controller, so eats less CPU/IO bandwidth.
2) can have battery backed cache
3) often looks like "just a scsi controller" to linux and the boot loaders, so booting from f.e. a RAID5 set is often easier.
software RAID has these advantages
1) is cheaper
2) CPU time lost makes hardly any difference
3) has well-tested and supported tools to manage your raid setup. (imagine if you could only set up your raid sets by rebooting and entering the raid bios)
4) disk-layout is non-proprietary (controller died? don't have the same brand lying around? manufacturer left the market? no problem!) - so all-around more flexibility.
Look here for properly supported sata disk controllers:
http://linux.yyz.us/sata/
Some of these cards come with BIOS smarts that provides you with software raid which offer you the advantage of point 3) of hardware raid, ie: bios and boot loader support for your raid.
however, this does mean that the on-disk layout has to be recognized in linux, so linux can make sense of it and set up the raid sets properly. In linux 2.4 there were some drivers that did that themselves, however for linux 2.6 there's now a little userspace program that recognizes a whole bunch of on-disk layouts, and sets them up using the device-mapper facility (part of LVM2).
The advantage of this is that you can use the same well-tested and -supported linux drivers mentioned on http://linux.yyz.us/sata/ , but still use the (bios) facilities provided by the hardware. Another advantage is that this program will probably be used by all ATARAID ("mostly-software-raid") devices on linux, so it is, or will be well-tested and -supported in itself.
You can find this program, called DMRAID here:
http://people.redhat.com/~heinzm/sw/dmraid
So if you decide to go the SW-RAID way, think and decide if you want the advantage of dmraid. I haven't tried this myself yet, and the only aspect I'm unsure of is the management aspect of it (like with HW-RAID drivers) - DMRAID doesn't use MDADM, so how can you properly monitor, hot-add,
MDADM itself isn't going away any time soon either, if I understand correctly. (And even if it does, it's probably very likely that they'll make DMRAID understand the MDADM on-disk layout to provide an upgrade path.)
If however you decide to go the HW-RAID way, make sure you get a reliable and reputable manufacturer - with open source drivers (!), preferably with a known on-disk layout, and be prepared to spend money. I've heard a lot about 3ware, but I have no direct experience with them myself, so I can't vouch for them.
Why aren't we discussing the real issue? Should we allow "ebay" to be used as a verb.
Intron: the portion of DNA which expresses nothing useful.
You can't just ask what RAID solution to use; you need to specify your optimization criteria. For example, if you are going for inexpensive, high capacity, and good redundancy but don't care about hotswapping, you can go with a reasonably cheap PATA or SATA hardware RAID. If you are willing to skip high capacity you can get an external SCSI SCA rack, an inexpensive SCSI card, and a bunch of small SCSI SCA drives. If you want high capacity and high performance without too much cost, get a bunch of large PATA drives, a similar number of FireWire or USB 2.0 (or both) enclosures, as many FireWire or USB 2.0 interface cards as necessary, and do it that way (which also gives you hotswap). If you are willing to spend lots and lots of money, get a Network Appliance server.
Personally, I'm not worried about performance beyond being able to play full-motion video. I have a PPC 604 180MHz from 1997 with a SCSI card and a RAID rack. 8x18GB at RAID 5 gives me 118GB or so of redundant storage, and I serve it over NFS to my other machines. Just for kicks, I have it going through a cryptoloop, too (LVM on cryptoloop on Linux RAID5 on SCSI). The initial cost was low (the drives were $15 each, the rack was around $100 on eBay, the trays were given to me, the SCSI card was under $40 on eBay, the 100Mbit ethernet card was about $20, and the computer had been a doorstop until I put Linux on it). The ongoing (electricity and cooling) costs are a little high (they are 10K drives), but that's life. I can play an MPEG or AVI from two machines on the network at once without hiccups, so I'm happy.
If I were going to build a RAID server today, I'd probably buy a Mac Mini, four large PATA drives, and four FireWire enclosures. Assuming 160GB drives, I'd have 320GB of RAID5 storage available over NFS (with a spare drive to swap in) for an investment of under $1200, and I can vary that cost with the size and number of drives. Yes, I'd be daisy-chaining FireWire, which means that each drive has only a portion of the total bandwidth. Then again, my network card will only manage 100MBit, so 3/4 of the FireWire bandwidth will be of minimal use anyway (except for reducing latency due to readahead and such, of course).
Maybe if you're using a 486.
The CPU overhead of software RAID 5 is insignificant on any remotely modern machine. Even "ancient" ca. 500Mhz P3s have checksumming speeds over 1GB/sec
On Usenet I posted a detailed description of how I built a 2.8TB RAID storage array for under $4100.