Slashdot Mirror
Managing RAID on Linux
A person deciding to go with RAID faces a panoply of options and gotchas. Hardware or software? How many controllers? ATA or SCSI (or ataraid)? RAID 1 or RAID 5? Which file system or distribution? Kernel options? Mdadm or raidtools? /swap or /boot on raid? Hybrid? Left or right symmetric? One poster pointed out that putting two ATA drives on the same controller could impact performance. Yikes! Didn't I do that? Upon discovering that O'Reilly had just published its Managing RAID on Linux book, looking at sample chapter , I bought the book and let my blood pressure return to normal.
RAID is one of these subjects that is really not complex; it's just very hard to find all the information in one place. This is precisely the book to solve the problem. Author Derek Vadala, sysadmin and founder of Azurance.com, an open source/security consulting firm, has gathered a lot of information and even personal anecdotes to go through the decision making process when going over to RAID. He goes step-by-step through that process, educating us about hard drives, controllers, and bottlenecks along the way. This exhaustive book may be the first to bring RAID to the masses.
Although parts of the book (RAID types, file system types) may seem already familiar to experienced Linux users, it is helpful nonetheless to have everything in a nifty little book. A section of file systems provided not only a rundown of the merits and drawbacks of each one, but also a guide to their utilities. I learned for example what "file tails" for Reiser are, and why using them causes performance to degrade after reaching 85% capacity. The book compares raidtools with mdadm as well as lovely commands like nohup mdadm -monitor -mail=paranoidsysadmin@home.com (which, if you haven't guessed, causes the system to email you RAID status reports upon boot).
People who use software RAID may skip over the chapter on RAID utilities for the leading RAID controller cards. Still, there was one interesting tidbit: Why, the author asks, do makers of controller cards put all their BIOS utilities on DOS floppies which require us to find a DOS boot disk? Seriously, how many of us carry around DOS boot disks nowadays? The book made me aware for the first time of freedos, an open source solution that solves precisely that problem.
The Software RAID stuff was pretty thorough and clarified a lot of things. The book does an excellent job in helping to identify and eliminate bottlenecks and optimizing hard drive performance (using hdparm and various monitoring commands). The anecdotes and case studies definitely clarified which RAID solution is suited for which task.
I am less impressed by the book's sections on disaster recovery and troubleshooting. Although these subjects are brought up at several places in the software RAID chapter, the book could have discussed several failure scenarios or used a fault tree (such as the famous Fault Tree in Chapter 9 of the Samba book, a marvel for any tech writer to read). The book doesn't even discuss booting with software RAID until the last 10 page of the book and then gives it only a single paragraph (even though the author acknowledges it as "one of the most frequently asked questions on the linux-raid mailing list."). Call me old-fashioned, but isn't the ability to boot into your RAID system ... kinda important? As someone who just spent a significant amount of time troubleshooting RAID booting problems in Gentoo, I for one would have liked more insight into the grub/lilo thing. Also, in the next paragraph in the last chapter on page 228, the author casually mentions that "all /boot and / partitions must be on a RAID-1." Say what? Please pity the poor newbie who religiously follows the instructions in the book but fails to read until the end. I'm not sure what the author meant by this statement, but it required a much more substantial explanation and needed to go into a much earlier chapter.
These complaints don't detract very much from this excellent book, a true O'Reilly classic and a model of clarity and helpfulness. This book provides enough knowledge to avoid the dread and uncertainty that comes with trying to tackle Linux RAID. With a book like this, a sysadmin can sleep a little easier.
Recommended Readings:- Reliable Linux , by Iaian Campbell, Wiley and Sons, Dec 2001, ISBN: 0471070408. Gives excellent information not only about RAID but on general Linux reliability issues.
- Software RAID in the Linux 2.4 Kernel by Daniel Robbins. (Part Two).
- Linux Journal Article on Software RAID by Joe Edwards, Audin Malmin and Ron Shaker. ( Part Two).
- "How to do a gentoo install on software RAID" by Chris Atwood. Gentoo User Forum.
Robert Nagle (aka Idiotprogrammer )is a Texas technical writer, trainer and Linux aficionado. You can purchase Managing RAID on Linux from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
but the easiest way I've found is to go with hardware RAID. It's easier to setup, doesn't put any extra load on the CPU, and only costs a few hundred dollars extra.
Mind you I'm thinking of RAID used in producion instead of someone RAIDing two drives in there home machine.
Is it possible to use Firewire and a service like Rendevous to make an intelligent redundant system? It's a thought at least. My firewire drive I use for my Inspiron works nicely enough. Would firewire be cheaper than RAID for servers, however?
Syr GameTab.com - Game Reviews Database
panoply
n. pl. panoplies
Looks like number one is most appropriate, although I've never referred to my arrays as "splendid".
"all /boot and / partitions must be on a RAID-1."
/boot must be RAID1, but / can most assuredly be RAID 5 (or, I presume, any of the other RAID levels). I have this running on an ol' RedHat 7.0 box:
/dev/md1 / ext2 defaults 1 1
/dev/md0 /boot ext2 defaults 1 2
/dev/md0 /dev/sdb1 /dev/sda1
/dev/md1 /dev/sda6 /dev/sdb6 /dev/sdc5
With raidtools, at least,
Hunk 'o fstab:
Similar hunk 'o raidtab
raiddev
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
#nr-spare-disks 0
device
raid-disk 0
device
raid-disk 1
raiddev
raid-level 5
nr-raid-disks 3
chunk-size 64k
persistent-superblock 1
#nr-spare-disks 0
device
raid-disk 0
device
raid-disk 1
device
raid-disk 2
*Shrug* Wonder what the context of that quote was within the book?
the no
I've stepped away from the software RAID idea on my boxes, due to the availability of cheap hardware RAID, such as Promise's SX4000. It will do hardware RAID 5 for four+ drives and has a SDRAM slot for cache expansion. Coupled with LVM, it ended up being a good solution for me, as I had both the reliability, and good volume management if I wanted to combine arrays.
The problem I've had with the software RAID is reliability and expandability. It is a pain in the ass if you lose a drive in the array, and it is next to impossible to add a drive (other than a stand by drive) to your existing RAID 5 setup.
Aah, opinions...
------------------ D. A. Davenport: http://www.firebin.net
(From the raid howto)
4.7 The Persistent Superblock
Back in ``The Good Old Days'' (TM), the raidtools would read your /etc/raidtab file, and then initialize the array. However, this would require that the filesystem on which /etc/raidtab resided was mounted. This is unfortunate if you want to boot on a RAID.
Also, the old approach led to complications when mounting filesystems on RAID devices. They could not be put in the /etc/fstab file as usual, but would have to be mounted from the init-scripts.
The persistent superblocks solve these problems. When an array is initialized with the persistent-superblock option in the /etc/raidtab file, a special superblock is written in the beginning of all disks participating in the array. This allows the kernel to read the configuration of RAID devices directly from the disks involved, instead of reading from some configuration file that may not be available at all times.
You should however still maintain a consistent /etc/raidtab file, since you may need this file for later reconstruction of the array.
The persistent superblock is mandatory if you want auto-detection of your RAID devices upon system boot. This is described in the Autodetection section.
While I'm certainly a proponent of "dead-tree" documentation, I have to take a moment to disagree with one of the statements made -- I'm sorry, but newsgroups, while perhaps containing out-of-date info, are (if it's a good newsgroup) capable of letting you know the current state-of-affairs. This is substantially -less- true with books. Case-in-point is Samba: it's *DARN* hard to know, from the Amazon description (or wherever) which Samba books describe the current state (2.4 and above) of Samba, whereas the FAQs, newsgroups, etc., are fairly obvious on it. Bottom line? I'll take a good book any day, but when in doubt, I'll go with current info gleaned off the newsgroups and other on-line resources.
Why, the author asks, do makers of controller cards put all their BIOS utilities on DOS floppies which require us to find a DOS boot disk? Seriously, how many of us carry around DOS boot disks nowadays?
Well, given Dell's recent announcements, I suppose fewer and fewer of us will be doing so.
But really, the author's point is so moot that it's embarassing: if it's my job to maintain a RAID array, and the utilities are on DOS floppies, of course I'm going to have access to a DOS boot disk. So what ? Just how hard is it to carry such a thing around, and why is this is a worthy thing to rail about, in a book about RAID ? If the author wastes too much time talking about stuff like this, this book can't be that useful - arggh, I've wasted too much of my own time already.
It's not that hard.
- Power down the computer
- Remove cover
- Blow out all dust and insect husks
- Spray in RAID
- put cover back on for 15 minutes.
- Remove cover again
- blow out insect husks.
Trolling is a art,
RAID 0 is pointless - gosh, I wish all the video editing studios out there knew this. They've been duped into believing 150 megs a second sustained has value. What morons.
Too bad cheap RAID5 cards don't exist. - Hmm, you mean like the Promise SX4000 that costs $150?
I'm Rick James with mod points biatch!
Dude, that's hardware. Turn off the dma on your drives with hdparm.
/dev/hdb
/dev/hdb
hdparm -d 0
You might also have to turn off 32 bit mode:
hdparm -c 0
Of course, this will slow things down.
Be sure everything's jumpered correctly.
Also, of course, I'm not responsible if you fry your data!
-- I am. Therefore, I think!
the poster obviously doesn't know what he's talking about.
a 'rubbish' 500Mhz CPU - 500,000,000 ops / sec
a 5ms access time SCSI HDD - 200 ops / sec.
so what if the CPU on the RAID card is a pathetic 100MHz job, it'll still be able to keep up with the data flow from the HDD, even when that data is being burst through.
How much cache ram have you got on that RAID card is a better indication of performance improvements for your hardware.
I think that a better title for this book would be,
RTFM: RAID - The Fucking Manual.
Life is the leading cause of death in America.
You actually feel good about the Linux drivers that Promise gives you with the SX4000? I bought this card, and I wished I stayed away from it.
I am using it with four 120gb IDE drives with 8mb cache. For starters, if you use anything but the sxcslapp program in Linux to configure the drive, your drives are corrupt. All of 'em. And, your bios will return corrupt information regarding them. This causes DOS not to boot (hard freeze), and Linux to produce keyboard smashings on boot. This is a known firmware problem, and I'll be damned if they have any flashes available, even though the card is four months old. I just checked before writing this review.
Once I figured out that all the work had to be done with sxcslapp in Linux, I started building my RAID5, albeit with caution. Things here went pretty well, except a) performance sucked about as bad as a single drive and b) the closed source drivers rebuild the raid array with no warning if a drive fails and is replaced, even if the file system is mounted. So, this means that if you have a drive that bombs and you replace it, anything you write to the raid array will be wiped out. I could have used some notification.
The Linux drivers are horrible. They are written in 'Engrish', and the documentation might as well have been written by someone who doesn't understand computers. "Select the remove drive from array option to remove a drive from array". This continues for all of the options in their menu-driven app.
I am also forced to use Red Hat 7.3 for this. Great. I now have a cluster of Debian 3 servers I administrate and one Red Hat server.
I would have returned the card if my reseller would have taken my money. It's about equally expensive to buy IDE add-on cards, or maybe a bit less, and the software RAID in Linux seems to be firmly documented. I've used RAID1 in software on servers before, and it works nicely.
3-4 years ago, when we decided to use hardware RAID on our Linux servers, we bought some DPT Smartraid V hardware RAID controllers. Unfortunatly DPT was bought by Adaptec some time after. Adaptec has been really good at getting the driver included in the kernel, but the takeover seemed to delay this proces, so the time in between was a rough ride.
The lesson learned was, never have a production Linux system with (binary) drivers tied to a specific kernel or distro version.
That said, we have been very happy with the controllers, and since at least two disks has died without warning, the expense has easely been worth it. Our systems are used 24/7/365, so every minute of downtime annoys somebody. RAID really makes me sleep better, restoring a server from a slow tapestreamer, at some ungodly hour, while people nervously checks in, asking when we will be up again, is something I really want to avoid too much of.
YMMV, but I think hardware RAID still has an edge over software raid, mostly because I find it simpler to maintain in the long run.
If you are into LVM's, FS tools, and software RAID, go to:
http://evms.sourceforge.net/
and _drool_. Future stuff for now on production servers, but nevertheless.