Slashdot Mirror
Experiences w/ Software RAID 5 Under Linux?
MagnusDredd asks: "I am trying to build a large home drive array on the cheap. I have 8 Maxtor 250G Hard Drives that I got at Fry's Electronics for $120 apiece. I have an old 500Mhz machine that I can re-purpose to sit in the corner and serve files. I plan on running Slackware on the machine, there will be no X11, or much other than SMB, NFS, etc. I have worked with hardware arrays, but have no experience with software RAIDs. Since I am about to trust a bunch of files to this array (not only mine but I'm storing files for friends as well), I am concerned with reliability. How stable is the current RAID 5 support in Linux? How hard is it to rebuild an array? How well does the hot spare work? Will it rebuild using the spare automatically if it detects a drive has failed?"
Been doing this with 5 Maxtor Firewire 250gig drives for a good while, and regular ide drives for years before that. It's always been very stable and has had no problems with drives going bad as long as you replaced them quickly. I moved to firewire though, because it was much easier to see which drive went bad out of the set, and you could hot swap them.
Generally for situations where you really need to make sure the data stays safe, I'd just stick with hardware. If you can spend that much on some harddrives, I don't see why you can't spend the money on hardware.
Though from what I hear, software RAID on Linux works decently.
Take it from me, stick with a hardware raid 5, reliablity is thru the roof, and cards are now around 300-500 for one with 128 mb of ram. Ince you spent 960 dollars on the harddrives, you might as well trust their organization to something of equal quality.
my 2 cents
unless you want to wait forever for fsck
Do not look into LASER with remaining eye!
Is there a good resource for hardware/software RAID support on linux? Tech support is always a challenge and we have a number of 3ware 8way and 12way powered by 250gb drives. We often have lots of mysterious drops on the array that require reboots or even rebuilding the array. Royal pain in the ass.
Laboratree - Scientific collaboration based on OpenSocial.
To me, a cheap 500 Mhz computer (who probably have 64-128 megs of ram, I guess) is gonna choke on 8 250 GB hardrives on Software RAID 5. Some other posters suggested buying an hardware controller, and I agree. If you host stuff for your friends, you can always charge them a little extra to compensate.
Eureka Science News - automatically updated
And are all 15,500 sites about how maxtor drives suck? I guarentee you the aren't. I've been using maxtor drives for as long as I can remember, and not one has failed. I still have a maxtor from five years ago and it's still running fine.
I used to work at Kiva Networking and we used hardware raid 5 on some machines and software raid 1 and raid 5 on others. Maybe it was just me, but the software raid 5 disks always seemed to last longer. Never much problems with it. In fact, we had more problems getting the hardware raid controller to work with Linux or with buggyness than anything.
It must be obvious that you donn't know much about software RAID, because 3ware isn't software raid--they make hardware.
And they make quite good cards too, that are highly supported in linux,freebsd, etc. (I have an 8 port SATA raid card in use atm)
Have you tried the 9500 series? It looks much nicer than their older offerings.
We've run several 7810s, 7850s in the past, totalling quite a few terabytes. All in all it's not too awfully bad, but the cards do seem to have trouble with dropping drives that don't seem to have any real problems (they recertify with the manufacturer's utility often with no errors).
If you go 3ware though, get the hot swap drive cages from 3ware. They are expensive, but it makes it much nicer.
I've had enough abrasive sigs. Kittens are cute and fuzzy.
I have 4 IDE Maxtor 200G drives on two Promise controllers and it's really very stable. I've done this for about the last 4 years for a home network share, with very very good luck.
/dev/md0 576877984 506579124 40995112 93% /data
/proc/mdstat
:)
After setting it all up, I encourage you to attempt to pull a cord from a drive, and boot, and make sure you know how to recover. Nothing can compare to the knowledge of knowing what to do in a serious failure.
My raid5:
knitterb@machine%> cat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdk1[3] hdi1[2] hdg1[1] hde1[0]
586075008 blocks level 5, 32k chunk, algorithm 2 [4/4] [UUUU]
unused devices:
0.000u 0.000s 0:00.00 0.0% 0+0k 0+0io 88pf+0w
knitterb@machineb%>
Good luck!
-bk
VOB.
The higher the technology, the sharper that two-edged sword.
Moderators, mod this up!
The guy could be looking for people's experiences rather in additional to any technical documentation, which is not only smart, but the hallmark of a responsible sysadmin (where knee-jerk comments tend typically aren't).
I have a smaller array, but it's been largely trouble free.
:(
However, when a drive did drop off line, unless things are on their own channel, it seems to knock off the entire IDE channel as well. It ended up taking the better part of a day to get everything back online again, without any data loss.
It even seems like any time there's an IDE hiccup, you can knock your array offline.
It's definitely cheaper than hardware RAID, and I haven't noticed any performance problems, but sometimes the stability of good old SCSI raid is something I miss.
Results 1 - 10 of about 85,800 for slashdot sucks. :-)
You get what you pay for.
Try a dvd collection. 250 dvds at 8 gb per = 2 TB. I've had around 2 TB on a couple of my machines for a long time for that purpose.
Does anyone know if software raid-5 in linux uses a chunk of memory specifically for the write cache? Most hardware raid5 cards have memory for this so that the card can tell the OS everything is done being written and then flush it out of cache at its convinience. If software raid-5 actually relies on the writes to be finished, it would be drastically slower (for writes).
As an ISP building servers for our own use, we've never had problems with Linux RAID-5 but plenty of problems with recalcitrant disc interfaces that won't easily support hot-swapping and/or clean reporting of drive failures. Even 3Ware cards (until relatively recent firmware revisions) could cause system crashes before they cleanly reported a drive as failed with Linux RAID-5 sitting on top (yes, Linux software RAID-5 is much faster than the hardware RAID-5 with their 8000 series cards). So my advice is just to test a failure: try unplugging a drive from a test array while it's running and see what happens. The "plug & play" aspect might not work as advertised but you shouldn't lose any data through it (we've not).
Matthew @ Bytemark Hosting
I've been using RAID 5 with a 3 18 GB SCSI drive setup for about 6 months now, it works very fast and reliably.
The best advice I can give is to make sure each drive has its own channel if you are on standard ATA, you didn't specify SATA or regular ATA. If you're using SATA then all the drives get their own channel by design. If you have more than one IDE device on a channel in a RAID, performance will suffer because IDE can't write to both devices on the channel simultaneously.
pr0n cache
While it's not Linux, I've been using Vinum with FreeBSD for about 3 years with RAID 5 and have never had any problems. My current box is an old VIA 600MHz C3 with FreeBSD 4.8 and a measly 128MB of RAM. As far as benchmarks go, my RAID seems to blow away all of the cheapy hardware cards performacewise as well.
BTW, I switched from Linux to FreeBSD for the server years ago for the stability.
If you got the money, go hardware... I have two systems that are identical with the exception that in one I broke one of the ide connectors of the 3ware controller and had to go software raid instead. I have 6 120GB disks in each, on separate channels... There is no problem with the stability and both setups survived at least 2 disk failures each.
Unfortunately the performance is another issue - I get three times the throughput on the 3ware controller. So the $300 I spent on the controller (used) was well worth it...
Peter.
The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.
Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?
Furthermore, block sizes default to 4k, though you could go to 8k or 32k block size. At any rate, you don't need a gig of RAM to handle this.
Finally, XOR is not that expensive of an operation, and a 500Mhz CPU is going to be able to handle that faster that any but the most expensive controller cards.
So unless you are actually a RAID kernel developer, I don't buy your story.
This is the worst possible type of advice. Do you have any reason for not using them? Maybe you've bought dozens and they've all blown up and burnt your house down, which would be a good reason to not buy 3Ware. Maybe you work for a competitor.
For all I know, you could have a very good reason. But if you tell someone to make sure to to stay away from something, you should provide a reason. Especially if it's something that seems to have a really good reputation.
________________________________________________
suwain_2
Software raid is fine for simple configurations, but if you want to "do it right" - especially considering that you just dropped about a kilobuck on HDDs, go Hardware. A good, reasonably priced true hardware RAID controller that will fit the bill for you is the 3Ware Escalade 7506-8. It has 8 IDE ports, 1 for each drive - you don't want to run two RAID drives in master/slave mode off of a single IDE port; it will play hell with your I/O performance. It's true hardware raid, so you don't have to worry about big CPU overhead and being able to boot with a failed drive (a major disadvantage to software RAID if your boot partition is on a RAID volume, certain RAID-1 configurations excepted). You can buy them for under $450. provantage.com price is $423.48 (I have no relationship with them other than I've noticed that their prices tend to be decent).
Help save the critically endangered Blue Iguana
The cheap raid controllers are almost always software raid and not worth it. If performance is critical some of the higher end SATA and SCSI raid stuff is worth it, but a lot of that sucks too so do benches, take recommendations and don't believe in brand names...
Except for the early 7000 series they are good cards and have decent performance too. I'm very very happy with the 3ware I have even though its one of the quite early designs.
Another piece of advice would be, since you have eight identical drives, to use only seven drives in the RAID array, and keep the eighth one out of the array entirely, either outside the computer in an antistatic bag or as a "hot" spare--installed but idle.
When one of the drives fails--and one of the drives will fail--this will allow you to swap in the replacement drive immediately, before another drive fails. (Remember, if two drives fail in a RAID-5 array, you lose data.) You can then return the defective drive, get a replacement from Maxtor, and when that one arrives FedEx in a few days, that one will be your new "spare."
You can either keep your spare drive unused, outside the computer, or keep this spare "hot"--in the computer, connected and ready to go, but unused by the array or anything else, and have the array fall over to it automatically when a drive fails.
Both ways offer advantages. If you keep the drive out of the computer, since you need to shut down to remove the bad drive, you can install the spare drive at that time. If you were to keep the drive "hot" in the meantime, your extra "new" drive has been spinning for months or years, and exposed needlessly to heat. Which increase its probability of failure, making it essentially as likely to fail as all your other drives that have been running the whole time.
However, keeping the spare "hot" means that the array can be rebuilt sooner, in some cases automatically before you know there is a problem. This can reduce the possibility of data loss. You will have to reboot twice--once to remove the defectie drive to return to Maxtor, and once when the replacement arrives to install it as the new hot spare.
Which of those two choices is a judgement call, but it's absolutely critical to have a spare drive on hand.
If all it does is serve files, it should do fine. The 500Mhz is not going to be a factor at all, in fact, the CPU will be idle most of the time. The real thing to optimize in a file server is the ATA bus speed and hard drive latency.
This is a VERY big issue. We've found that Promise Technology RAID controllers have problems, and the company doesn't give tech. support when the problems are difficult, in our experience.
--
Government data compares Democrat and Republican economics.
> Do yourself a favour and buy some more or less cheap hardware RAID controllers. You won't regret it. Software RAID is nothing more than "showing it's possible".
There is no such thing as a "cheap" hardware RAID 5 controller. Well there is, but they'll still set you back at least $120 and are crap.
There are RAID controllers from highpoint and promise, et al that are card-based, but they are still CPU bound (that is where the XOR really takes place). So they're really nothing more than a controller with a driver that does the calculations in the CPU. These cards are good for booting windows to a software RAID (since that is essentially what they are) but not good for anything else.
Most motherboards especially those with only 2 RAID ports (whether IDE or SATA) are software-based, as well. The nvidia nforce3 250 is one of the few notable exceptions.
But the bottom line here is: Linux Software RAID 5 is a logical approach if simple redundant mass storage is your main concern, and will save you at least $120. Also note that for RAID 0/1 it doesn't really matter if you go hardware or software since they aren't very processor intensive anyway. Pure software RAID 0/1 seems to be easier to set up in Linux (less mucking around with drivers) so it often makes sense to go with it for that reason alone.
The unofficial
Don't hang a pair of drives off each controller. Get a truckload of PCI ATA cards or a card with multiple controllers. Don't slave a drive. (No, I do NOT know what the correct PC term is for this).
Also, give 'mdadm' a whirl - a little nicer to use than the legacy raidtools-1.x (Neil's stuff really rocks!)
Software RAID5 has been working extrememly well for us, but it is NOT a replacement for a real backup strategy.
one better than mcleodeight
Declare at least one hot spare. I would declare two for your setup but YMMV.
/dev/hdh1
nr-spare-disks 1
device
spare-disk 0
The GPL, for those that truely understand.
pr0n
True story.
There are a few things that really help in some cases, but RAM isn't always one of them.
If you've got a lot of data that is read/re-read or written/re-read by clients then RAM really helps, streaming stuff which doesn't get many repeat accesses (eg running a movie editing suite) it might not help at all
For performance its often worth sacrificing a bit of space and going RAID 1. Again depends if you need the space first or performance first.
Obviously don't put two drives of a raid set on the same IDE controller as master/slave or it'll suck. Also if you can find a mainboard with multiple PCI busses that helps.
Finally be aware that if you put more than a couple of add on IDE controllers on the same PCI bus it'll suck - thats one of the big problems with software raid 5 versus hardware which is less of a problem with raid 1 - you are doing a lot of repeated PCI bus copies and that hurts the speed of drives today.
I use raid1 everywhere, disks may be cheap but you have to treat them as unreliable nowdays.
IDE interfaces, and some SATA interfaces, are not designed to be hotplugged. There are special electrical circuits on hot pluggable scsi drives.
Once you yank the plug on an IDE, you *must* power down the system before plugging it back in. Sometimes yanking the plug hangs the kernel.
Power down the system, pull a drive, and then start it up. It should detect the missing drive on bootup. Alternately, follow the instruction on the LINUX RAID HOWTO. You have read that, haven't you?
If you copy (or just write) really big files, then there should be no reason for the raid-5 driver to read all the other sectors in the parity chunk (don't know official terminology for it offhand) since, if done right, you will be writing all of the sectors with brand new information anyway.
When information is power, privacy is freedom.
IOW: Two reads, and two writes. Not six reads and two writes. But yes, large amounts of RAM is a good idea. Of course, if a drive goes south, everything goes out the window and your performance will be shot until you replace the dud drive and everything resyncs.
What will you connect eight drives to? Four PCI ATA controllers? I have eight 200GB drives on my data server using a 3Ware RAID controller, and it has worked wonderfully for 18+ months. I have had a drive fail (due to insufficient cooling), and the system didn't even hiccup. I have a software RAID system at a client's location. Whenever there is a power failure, the system comes back up nicely. However, because of the abnormal shutdown, the software RAID tries to recover one of the disks. This absolutely eats the processor for 16 hours. 98-100% utilization. Fiddling the /proc parameters is no help. I think this is a bug--what could it be doing for 16 hours?
I built a fileserver for work on the cheap, it's been up for about a month now without any problems.
You have two choices with software RAID under linux, raidtools and mdadm.
Raidtools is a bunch of tools for setting up, looking after etc a software RAID under linux and mdadm is essentially the replacement to Raidtools, everything is built into the single mdadm program.
I personally find Raidtools to be easier to use at the command line, however mdadm is the future. Also of note, if you want to set it up easilly without the hassle just use Webmin, its RAID setup is a breeze.
If you are concerned with reliability, don't store everything in one device that can get taken out by fire/electricity/anything. Parity also isn't very reliable, I had two drives go bad in my RAID 5 and I was screwed. Also, Maxtor drives arent reliable, not like SCSI drives.
And as far as IDE drives, Seagate makes 200GB drives with 8mb cache, 5 year warranties, and they were at Frys last week for 50 bucks.. (rebate)
I agree, the parent post is just a troll to see how gullible the moderators are. Apparently he proved his point. :)
I've had enough abrasive sigs. Kittens are cute and fuzzy.
I have a very similar setup, but I'm currently only using three hard drives. I'm running Mandrake 10 on an Athlon XP 2000 with three Western Digital 250GB drives. That gives me 500GB of usable space, which I'll probably upgrade to ~1TB (5 drives) in January. So far I've had no problems. I have a UPS, but I've pulled the plug several times for testing purposes with no ill effects (I tested the setup for several weeks before putting real data on it). I think a big reason this setup works for me is that the server is never under much stress (serving 3-4 household PCs), and it's only used as a file server. I use computers for everything. Work projects, personal projects, audio/video editing, etc. I never delete anything and would be devastated by a data significant data loss. So I use RAID5 with monthly backups (unfortunately backups are still important), and all important data is saved on the storage server. I'd recommend against sticking 8 drives in a box. If you're set on using 8, you might want to consider putting a few in external enclosures. Make sure you keep cool air moving across them to avoid shortening the lifespan of your drives. Currently, I'm only using about 250GB on my 500GB server, with most of the space being consumed by data from my audio/video projects. Unless you're actually going to use 1750GB of space, I suggest using five drives or so in the server while using the remaining three for backups. Also as was mentioned in a previous post, your system may not be beefy enough to handle an 8 drive RAID5 array. Let us know what you end up doing.
As other posters have mentioned, software raid is fine for RAID 0, 1, 0+1. As you get to RAID 3,RAID 5, and RAID 6, however, your processing requirements go up quite a bit.
:)
A SATA RAID 5 card with hardware XOR engine and a DIMM slot for cache might be a cost-effective option for you. (Goes for ~$180 on Pricewatch, or ~$240 on Dealtime)
Oh, and I would have goine with HGST, Western Digital, or Seagate for your drives... but I suppose hardware failure is what RAID 5 is for
We've had two different 3ware hardware RAID cards without any problems in the last 3 years.
I've done software RAID as well using Promise IDE controllers. Fortunately for us we never had a drive fail in the software RAID so I can't comment on how difficult it is to recover from a failure.
Interestingly enough, we ran some fairly intense iozone tests on both the hardware and software RAIDs with very little difference in performance (maybe that's why the parent poster doesn't like the 3ware stuff). But... we also ran these same tests with a fibre-channel SAN disk, again with very little performance difference.
Maybe it was a Bus limitation... I didn't have time to investigate it any further.
You will get responses from people with good and bad experiences, but they are all jaded by their small particular case. After seeing what can happen with dozens of machines (8 drive and 4 drive) running Linux software RAID5, here is some concrete advice.
/dev/hda1, Linux software RAID5 will kick /dev/hda1 out of the array. Now, it's likely that sector 11 might be bad on /dev/hdc. If you hadn't divided up the partitions, you would lose a second disk out of the array during a rebuild.
First, ensure that all of the drives are IDE masters. Don't double up slaves and masters.
Secondly, DON'T create gigantic partitions on each oft he 250's and then RAID them together, you will get bitten, and bitten hard.
Here's the skinny...
1) Ensure that your motherboard/IDE controllers will return SMART status information. Make sure you install the smartmon tools, configure them to run weekly self tests, and ensure you have smartd running so that you get alerted to potentially failing drives ahead of time.
2) Partition your 250GB drives into 40 GB partitions. Then use RAID5 to pull together the partitions across the drives. If you want a giant volume, create a Linear RAID group of all of the RAID5 groups you created and create the filesystem on top of that.
Here's why, this is the juice.
To keep it simple, let's say there are 20 secotrs per drive. When a drive gets an uncorrectable error on a sector, it will be kicked out of the array. By partitioning the drive into 5 or 6 partitions, let's say hd(a,c,e,g,i,k,l)1 are in one of the RAID5 groups, which contain sectors 1-4 (out of the fake 20 we made up earlier)
If sector 2 goes bad on
By partitioning the disks you localize the failures a little, thus creating a more likely recovery scenario.
You wind up with a few RAID5 sets that are more resilient to multiple drive failures.
If you are using a hot spare, your rebuild time will also be less, at least for the RAID5 set that failed.
I hope this makes sense.
My advice to you is to bite the bullet and simply mirror the disks. That way, no matter how badly they fail you'll have some chance of getting some of the data off.
After some initial hicups with the system in Mac OS 10.2, the new 10.3 seems to have a usable software RAID solution. It supports just about any method of connecting the drive to the computer (short of network) and supports most any kind of storage device. It doesn't support RAID5 yet though, just stripe and mirror. I've got four mirrored volumes up on my server, and have had few problems with them. Rebuilding must be done with the volume offline. It's not a perfect solution yet, but it's a nice free alternative for your OS X systems.
I've also tried CharismacRAID, and omg... STAY AWAY from this. After the array crashed for the third time for no apparent reason, I was experimenting with the software to test its reliability when it proceeded to try to LOW LEVEL REFORMAT my BOOT DRIVE. (yes, while I was booted up on it!) It actually managed to zero the partition table and boot blocks before asking the OS to lay down a new partition, at which point the OS thankfully gave it the bird. Kudos to Disk Warrior for being able to salvage the volume, and good riddance to CharismacRAID. (aka "AnubisRAID" fyi)
I work for the Department of Redundancy Department.
The first mistake is Maxtor drives. Worst drives i've seen, and I am talking present time.
Yes you have five, but do you feel like replacing quite a few of them over a few years?
I run software raid1 here at home, on a 2.6 kernel, and I can say that is solid. It appears software raid in general in linux is quite solid.
-- Note: If you don't agree with me, don't bother replying. I won't read it.
This is slashdot. Not a legitimate tech forum.
-- Note: If you don't agree with me, don't bother replying. I won't read it.
Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?
That would depend on the nature of the write. If you're writing the initial data it's unlikely that you'll require reading. However when you go to update the date you may have to perform reads in order to calculate the parity required for the update.
Software RAID 5 is very reliable but does suffer a performance hit. Not because of the XOR computations like many here are suggesting. It's because each logical write needs to be translated into physical reads/writes...which consume time.
The beauty of software RAID, at least software RAID implementations such as Veritas, is that it allows you to spread the RAID across a number of controllers.
Listen to this guy...he knows more than the others who consider the XOR computation the slow link in software RAID 5. It's not.
Maxtor may not be the absolute best built drive ever, or even ev4r!, but they do have a no quibble warranty. Its nice, don't like the way the drive looks at you in the morning, call 'em up and they'll replace it right then and there. Hear a squeak, call, replaced. ATTO showing wierd numbers, call, replace. (I actually did with 2 drives, the read speed was down compared to 2 other matching models, Maxtor didn't question it and replaced the drives.)
Drives from just about any vendor can suck. I know people who had 1000 IBM 75GXP drives (hey, their previous models were ok!) and had to swap a drive a day due to errors (not total failures tho, just early signs of it). Although many people seem to recommend "identical drives" for raid clusters for performance reasons or whatever, mixing vendors or at least production batches does the benefit of a single vendor fuckup resulting in data loss due to all of your drives having similar early deaths.
:-)
As for controllers, lots of big installations use 3ware cards. A bit more expensive than the basic ide/sata adapters, but you get lots of ports in one pci (preferably something better than 32-bit/33MHz) slot, and each drive gets their own ide bus. Now, even if the things have hardware raid5, you might be better off using plain old software raid with them. Depending on the card model it can easily be twice as fast. Embedded CPU's aren't that fast compared to a p4 or a64
> Absolutely incorrect. Promise SX4/SX4000 provides hardware XOR capability on the controller. XOR'ing never has to touch the host processor.
Dude, the SX4 will run you $160. I was talking about the cheap sub-$100 promise (and highpoint) cards.
The unofficial
You said 8 drives? Maybe I am missing something.
You say a 500 MHz machine. That is probably a P3, and hence would not have SATA. You did not say SCSI either. If you are trying to connect 8 drives to a standard PC you will run out of IDE ports. Most PCs come with only 2 ports, i.e. maximum of 4 drives.
So, how are you going to connect them?
2bits.com, Inc: Drupal, WordPress, and LAMP performance tuning.
Is this the definitive article about software RAID under Linux?
Software-RAID HOWTO. In English and HTML: Software-RAID HOWTO.
--
Bush borrows money to kill Iraqis. 140 billion borrowed. With interest, you pay 200 billion. When Saudis attack, invade Iraq?
Check out the project I did over two years ago to provide my home with Network Attached Storage: http://www.mini-itx.com/projects/tuxserver/
This is an awesome little machine that has been serving files to my home network for over two years with nary a reboot. Fast, stable, and zero problems with the software RAID5. I did have one drive go out about a year ago; popped in the replacement and the array automatically rebuilt itself.
One recommendation I'd make is use a journaled file system! I use ReiserFS and it is excellent for use in a software RAID.
If "disco" means "I learn" in Latin, does "discothèque" mean "I learn technology"?
Invest in a UPS for this system. Any uncontrolled shutdown (i.e. power outage) will take quite a long time to fully recover from. It will take a lo-o-o-ong time for the resync to finish on an 8x250GB raidset. Then you really ought to force an fsck as well on the filesystems you've created on that monster.
CUR ALLOC 20195.....5804M
Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?
Unless you write across a whole row in the array, how are you going to compute the new parity without reading in something? This is the "small write problem", and it is why expensive RAID controllers have a non-volite writeback cache.
The current kernel does read in the whole row to recompute the parity for simplicity. Technically, though, you just need to read in the block you are modifying and the parity block, making writes take 4 operations under RAID 5, but unless something has recently changed, Linux doesn't do that. A gig of RAM, however, will allow a degree of volitile write-back cache, to help offset what will otherwise be poor write performance.
"Your logic eludes me. The blocks do not need to be read, as we are in the process of writing. We already have the data, because we are writing, so why would we re-read the data?"
It depends on the nature of the updates. If you're writing long contiguous chunks (e.g. file copy), then you're right, the blocks are already resident. But if you just want to update a single block (database update), it might involve reading from the other drives.
Two pieces of advice: (1) Look into mdadm, it saved my array once when I had to move it from one server to another, (2) look into smartd as a way to monitor the individual disks and detect failures. Okay, well then, _three_ pieces of advice. (3) make sure you look into ext2/3 filesystem parameters like the size of the journal (max it out) and the -R stride= option.
/proc/mdstat regularly).
mdadm will allow a "spare pool" shared between multiple RAID devices and smartd will check the state of the disk controllers at regular intervals. You should put the system _and_ the disks on UPS to avoid losing data in the event of a power failure (the disks need to write their cache to the physical media before it evaporates). Set up something (mdadm or smartd) to email you in the event of a disk failure, or you may be running in degraded mode for quite a while before you discover it (unless you look at
All in all it seems to work fairly well if you spread the disks across multiple channels, if you have enough RAM for page (buffer) cache, and if you get reliable disks. I have a 4-disk SCSI storage box that I have in RAID 5 mode. It has been running for over two years. The server failed and I had to move it, that is when I discovered mdadm -- A LIFE (DATA) SAVER!
Um, no. Since we are writing, we already know what the data is. Just write. No reads.
RAM is helpful on ly in the sense that it can cache data and make things appear faster if the data is already available in cache. It won't really help it read/write data any faster.
If a drive goes out, write performance is inchanged, as the XOR operation must be done no matter what.
Read performance depends - it's just an XOR operation, which is not very difficult. A 500Mhz CPU will still be mostly idle even in degraded mode. Some implementations *could* do the XOR in regular mode too, to check for data errors, in which case, no performance is lost in degraded mode anyway.
Ahh, a comment with good insight. I never thought of it this way, but I'll definitely consider doing this in the future.
I was always stymied about not being able to partition a md software RAID, I didn't think about it from your angle, that it's not what you'd want anyway!
I've had enough abrasive sigs. Kittens are cute and fuzzy.
I did a quick scan of the post and didn't see anything about mixing your software and hardware RAID.
Alot of people will tell you to go with hardware as you have seen in this post, and some say stay with software. Basicly i would go with a mix this provides the best solution you can write across multiple drives and provide fool proof backup giving you the best of both worlds.
Apple is now using this in their XServe RAID http://www.apple.com/xserve/raid/.
If you do go with something along these lines i would definatly upgrade your CPU to at least 1 Ghz and 512 if not a 1GB of ram. Doing this would make a nice fileserver and you could even host a web/ftp server if you wanted access to your files off site.
- MOSKIE
I have run software raid on Linux and FreeBSD just in testing. They seem to work without hiccup and they're stable, although I had no extensive testing. As far as hardware or software, I'd suggest hardware if you need hot-swappability and speed. It's a waste of money if speed is not an option as software raid is just little less secure; I can only see it losing data in the event of the kernel going wrong in a very unusual way or more likely the kernel crashing in the middle of a write. Increasing the ram would definetely take a lot of stress off the kernel and increase speed, but then again, I haven't seen many situations where stored files need to be quickly accessed; I can only think of low compressed, hi-quality movies. As far as swapping drives, if you set up linux properly, the drive will reconstruct correctly (given only 1 dies). If linux dies and you have to redo the OS, the configuration is stored in each disk (partition more properly) that is in the raid so none is lost except time. One of the advantages of software raid is its flexibility, it's partition based instead of disk based. It doesnt seem like much of an advantage (and it isnt). Just make sure you don't make two partitions on the same disk part of the same raid array. My best piece of advice is: make sure every drive gets its own IDE controller. A drive can slow another one down on standard ATA. Also, a drive going wrong could kill the rest of the drives on its channel.
There's already some issues here because these are eight identical drives bought at the same time. They are very likely manufactured by the same machines in the same factory at the same time. This increases the liklihood of multiple disc failures for a variety of reasons. This is why many admins will replace discs on a regular schedule. For example, you buy your eight shiny new discs. Then you run them for a year, and replace one, a few months later you replace another, and so on. Then every two years from then you replace that disc again.
Also, any half-decent RAID implementation will have that hotspare in the machine with its spindle off until it is needed. So it won't have been spinning for months/years at all. Not quite as good as having it in a box as far as wear and tear, but very close.
As someone who built and installed ~400 systems with about 50 TB of storage capacity and ALL on Linux SW RAID5 I can only recommend it. I have bad experiences with HW RAID - when 2 or more disks fails, you can't get your data. Linux SW RAID is easy (OK, not SO easy) to be convinced to recover most of your data except the really bad ones. Also performance is really superb (with P4/Xeon/Opteron CPUs it's much higher then any HW IDE RAID can do).
That I put a RAID5 controller into. It works great. Spend the extra $$$ on a hardware controller. It's well worth it, especially when one of the drives decides to die.......
It's either on the beat or off the beat, it's that easy.
I moderate therefore I rule!
--
It's a poor solution for raid, since if the OS goes, there goes your raid. If you use hardware, at least it'll autodetect.
Um, that's bogus. If your OS goes (probably due to hardware?) then you can simply put the drive in a new computer (same basic master/slave setup) and away it goes. Linux knows how to detect its own RAID arrays!
OTOH, if you have a hardware RAID, good luck getting tech support, especially if they no longer carry that board, or have gone out of business altogether.
At least with Software RAID, your data is not stuck in a proprietary format.
First the setup: it is a Pentium 233 MMX (P1, not P2) with 64 MB of ram. It was first setup with Slack 9.1 a year ago. I did a clean install when Slack 10 came out and all Slack kernels have software RAID enabled by default, so it was recognised automatically. The two drives are quite different: one Maxtor 7200 rpm drive and a WD 5400 rpm drive that have ~ 1 GB difference in size. This computer does not even have a monitor connected to it.
Now the recommendations:
-Read ALL How-To's related to RAID, you can all get them at http://www.linuxdoc.org/ and either print them or have them quickly available on another computer.
-Be sure you have raidtools installed, it's a standard slack package.
-All partitions have to be the exact same size and of the "Linux raid autodetect" type. The software RAID on Linux is partition based, not disk based, which makes it more flexible.
-All drives have to be Master on an IDE channel, no slave drives (or else it will very slow), so 8 drives = 8 IDE channels = 3 additional controllers (or 2 if you have an onboard RAID controller). At this price, it could be interresting to get a 3ware hardware based raid card.
-NEVER, EVER run a hardware RAID array on cheap Promise RAID controllers (but software RAID on regular IDE Promise controllers, OK).
As for drive failure: once, when moving my server, a power cable got unplugged from the master drive on the first IDE port, the server booted without any problem from the second drive which is master on the secondary IDE port. The CDROM drive which is slave on the first IDE port was even detected correctly. I noticed it when I did the lsraid command. I then shutdown the server, plugged the cable back and did the raidhotadd command to rebuild the drive, a little less than an hour later, everything was ok.
It's pretty much what I can think of now...
You're all barking up the wrong tree. A high-speed RAID isn't going to be much good as a file server, the bottleneck is in the networking. Are you going to use gigabit fiber to connect to the server?
don't = do not = two words.
If the smallest block you can write is 512 B (a not very unlikely restriction), an attempt to write 511 B would mean you have to read what's already there if you don't want to garble a byte.
The world will end in 5 minutes. Please log out.
Given the cheap cost of disk these days, RAID 0+1 / RAID 10 is something you really ought to at least consider.
-p.
You may not need much CPU performance for file service.. after all, it's mainly just doing DMA to/from disks. But, I assume it's just your standard PC motherboard, with a single 32bit 33MHz PCI bus.
If you're spending $960 for the disks at Fry's, why not spend another $80 to $250 at that same Fry's and get a current generation motherboard and CPU (they have package deals that are dirt cheap).
For $80, you can get a 5x faster processor, and a much newer chipset with ATA133 and Serial ATA.
For $250, you can get a board with multiple PCI busses, PCI-X and a chipset capable of handling much more throughput than a cheap PC motherboard.
The I/O bandwidth will be your bottleneck with an 8 drive RAID array. The standard 32bit / 33MHz PCI bus only does about 1Gbps. Serving a gigabit ethernet connection will use all your bandwidth by itself.. when you have 8 ATA drives fighting the NIC for bandwidth, you can see a clear problem.
If you're spending that much for the drives, don't hamstring it by skimping on the motherboard. And, in any case, once you have a Linux box installed, you inevitably start using it for many tasks (caching proxy, mail server, ftp server, dns server, www server, etc). So, a beefier system will stand up better.
Hi,
:
The scenario you've mentioned is probably OK to use a software RAID. I use it in a production enviroment without problem with a higher stress that your setup will probably have.
I'd suggest you to consider the following items
a) cooling system - those HD can generate a lot of heat. Buy a full tower case and add those HD coolers to make sure your HDs stay cool
b) Buy the HDs from different brands and stores - RAID5 (either hardware or software) can recover from one drive. If you buy all from the same brand/store chances are that you end up with 2+ drives with the same defective hardware
c) cpu - if you are going to use this number of drives the processor will be a majo bottleneck. Do not forget that RAID5 XOR your data to calculate the parity.
d) partition scheme - use smaller partitions and group them together using LVM. This you help you to recover from a smaller problem without taking a lot of time to reebuild the array
Ok, network could be a bottleneck, if the disk IO gets high enough. I think, though, if this is a 500Mhz system, it is likely to need more attention on the PCI/ATA bus than on the network.
:)
Both network and IO are to be worried about before CPU.
The Promise controllers are SHRAID, which is my own non-standard acronym for Software w/ Hardware-assist RAID or SHitty RAID in less polite company. And the "promise" of true redundancy is a charade (rim-shot, please). Basically, you have all of the disadvantages of software RAID - the need to manually configure bootability of both drives (assuming you're running RAID 1 or RAID 0+1 - if you're running RAID5 or JBOD it's an even bigger pain), plus the need to have specialized drivers on the OS, etc. These controllers (Promise, Highpoint, etc.) should be avoided like the plague for technical reasons alone.
Good, relatively inexpensive IDE and SATA RAID can be had with 3Ware Controllers. 2-drive models start around $140, and they support up to 12 drives on their more expensive controllers. The drives appear as a single physical device to the O/S, whether it's Windoze, Linux, BSD, DOS 3.1, etc.
Help save the critically endangered Blue Iguana
Of course for several disks or Raid 5 your better with off with hardware. But for Raid 1 aka mirroring there is nothing wrong with software and I've been using it for many years at home and at small clients with no problems to speak of.
"It's a poor solution for raid, since if the OS goes, there goes your raid"
That's only a problem if you don't back up your data though. I agree use hardware whenever its possible but in a pinch software works great.
If you wanna get rich, you know that payback is a bitch
The short and sweet of it is that on event of a rebuild, if a single drive serves up a corrupted block, the rebuilt data will be corrupt and with no way of affirming that. Google for ACNC's website, as well as BAARF. The original BAARF site seems to be down.
Disk is cheap, as are controllers.
Here's the BAARF Site Cached on Google. The real site seems to be down. http://64.233.161.104/search?q=cache:8HJz0cFQHgoJ: www.miracleas.dk/BAARF/BAARF2.html+baarf&hl=en&cli ent=firefox-a
Here's an excellent
duscussion of RAID 5 vs RAID 10
by Art Kagel.
I would definitely warn against RAID 5, even hardware RAID 5, unless you're doing 95% or more reads. The performance hit is horrible, even worse for software raid, and why? To save disks? You already have 8! Build a 8-disk RAID10 (or 6 disk with two hotspares) and be done with it. You'll never use all that storage anyway. Performance will be phenomenally better for all sorts of applications that can use the storage. If you're going to go with a card like Promise, be sure Linux includes the driver in the kernel. I had a lot of trouble with a Promise card because the driver was only available on the Promise site for certain versions of Redhat, so I was stuck and couldn't upgrade (or was afraid to). That can be a real limiting factor.
Here's the BAARF Site Cached on Google. The
real site seems to be down.
http://64.233.161.104/search?q=cache:8HJz0cFQHgoJ: www.miracleas.dk/BAARF/BAARF2.html+baarf&hl=en&cli ent=firefox-a
Here's an excellent discussion of RAID 5 vs RAID 10 by Art Kagel:
http://64.233.161.104/search?q=cache:oat67e4ciCMJ: www.miracleas.dk/BAARF/RAID5_versus_RAID10.txt
I've recently made a server for archiving medical images using Redhat 9 with 5 SCSI drives in RAID 5. Works great at a fraction of the price of a vendor solution (ie, $30,000 vs $1,500.) In this case, the vendor was using a Windows NT box yet. Performance suffered big time. The Linux box I use is a 2.4 GHz Celeron with 1GB RAM. The bottleneck is always in the network interface (100 Mbs) so the Celeron is more than adequate. It's been up almost 1 year already with no reboots.
So my advice to you is to install a lot of RAM in this system, whatever the motherboard allows. At least one gigabyte, but preferably two or more.
There's a reason this is posted anonymously. It's absurdly incorrect. Disregard and move on.
http://www.bullnet.com
Sorry about the bad formatting above. I thought I had it straight. First posting to Slashdot, I should have used the preview button, arg...
Ah ok, I see what you are saying... although, chances are that the whole sector has already been read previously and is already in cache anyway.
For example, when the data was read prior to being modified.
The only time this would be a problem is when a whole new file is being written. Even then, the sector may be on the end of a free space inode chain, and so the kernel know no other data is on that sector, and it can do what it wants. Depends on the filesystem, I guess.
To answer your actual question, whether or not the linux kernel's software RAID implementation is safe... "yes". I used it in production for NFS fileservers as far back as the 2.2 series; it performed wonderfully under high load then and has worked just as well when I've used it off and on since, both in production and on test systems. There are lots of suggestions elsewhere in the thread about things to avoid - multiple devices on the same IDE channel is the big gotcha: don't do it, its performance is particularly horrific during array reconstruction, just when you need it to run as fast as it possibly can. Keep those suggestions in mind when you build the system, but you can categorize the RAID implementation itself as more than sufficiently reliable.
Ideology breeds Hypocrisy. Just how much is up to you.
I've been running a software RAID-1 setup for years, on a 2.2 kernel (still haven't bothered to upgrade the kernel). It's been generally stable and fast. The one time I got into trouble was when I tried to compile in journaling filesystem support, which at the time wasn't compatible. As with most things in the kernel, the issues are well known, and as long as you read up on them before you'll do fine.
I do remember it being a bitch to set up though, especially so I could boot off either drive (a requirement I had for quick recovery from primary disk failure). Maybe the process has gotten easier, I haven't read up on it.
This is slightly off-topic because it won't take care of the particular solution being sought, but another interesting way to do RAID-1 is using the controllers from Arco Data Protection. The have some that are physically connected between your IDE or SATA controller and the two drives to be mirrored - they just seamlessly mimic a single IDE device. This makes it possible to RAID-1 any IDE or SATA drive under any operating system or device. I've used them in places like phone systems and voice mail systems that have no provisioning whatsoever for RAID. It can take a little bit of case tweaking, and you have to be sure the power supply can handle it, but it's an interesting solution in certain situations where nothing else can do the job.
Help save the critically endangered Blue Iguana
Use MDADM for building and maintaining the RAID, it is far superior to other tools for this that I have used. It will make life easier.
Second if you have a few spare buck get a 3Ware card and go hardware raid (its much faster and worth it).
-Lord Shadow
I manage a lot of servers remotely. I started out using the hardware RAID support on my server's mobos. But there were issues with that.
First, it was hard getting Linux driver support (I think drivers were available, but it was a matter of downloading them. And I don't beleive they worked on the 2.6 kernel's I used).
Then the RAID setup required BIOS settings. When you only have remote access to a server (and no KVM-o-IP) that means you need to work through a tech at the DC. Not, umm, ideal.
And finally, there was the issue of 'what if I need to move these disks to a different server'. One that doesn't have the same raid controller. Well, it wouldn't work.
Anyway, I ended up using software raid. I've used it now on a few dozen servers. And I'm really happy with it. Performance seems fine, albeit I'm not using it in really IO critical environments like a dedicated database server. In in 99% of cases I'd now use software raid in preference to hardware raid.
What follows are a few tips I'd like to pass along that may be a help with getting a software raid setup...
If you get the chance setup RAID on / and /boot via your OS installer (on a new system). Doing it afterwards is a real pain.
Build RAID support and RAID1,and RAID5 into the kernel (not as modules). You'll need that if you boot from a raid1 boot partition. Note: if you are using RAID5 you'll need RAID1 built in (since I beleive in the event of a failed disk the raid personaility swaps from RAID5 to RAID1).
With a 2.6 kernel build I've been getting "no raid1 module" errors at the make install phase when building with a RAID-ed / or /boot. The 'fix' is to compile the RAID support you need into the kernel (not as modules) then run: /sbin/mkinitrd -f /boot/initrd-2.6.8.1.img 2.6.8.1 --omit-raid-modules (substituting your kernel image name/version).
Every now and then I've had the kernel spit a drive out a raid array. I've found that sometimes the kernel may be being overly cautious. You can often raidhotremove then raidhotadd it back again. And you may never see a problem again. If you do, it probably really is time to replace the disk.
Rebuilding a RAID array goes smoothly. It happens in the background when the Linux machine is in multi user mode. The md code rebuild guarantees a minimum rebuild rate. From memory it takes about an hour or two to do a 200GB RAID1 array.
You can see the RAID rebuild status in /proc/mdstat. I run a very simple script to check the RAID status each day and send out an email if it is broken.
If you are using a RAID-ed /boot, grab the latest lilo since IIRC it has better RAID support than what is in the distros I use.
Hard drive-wise I've been happy with Seagate Barracudas. I've had to replace a few failed Western Digital drives. (Just my recommendation from experience, it could just have been good/bad luck on my part).
One neat trick with Software raid is that your drives don't have to be the same size. You do RAID on partitions. And your raid array sizes itself according to the smallest common denominator in the array.
Tip: always create a bit of spare space on any device you are RAID-ing. e.g. a 4GB swap partition. Then if you have a drive fail and it needs to be replaced, and your replacement varies in size slightly you'll still be able to use it. Not all 40/120/200GB drives are created with equal sizes :).
In summary: Software RAID=good. Decent performance. I've had no real kernel bugs with it. No need for BIOS access. Easy to move drives between servers. Easy to monitor failures. Non-intrusive/minimal downtime when recovering a failed devi
Comment removed based on user account deletion
Overall, I'm very happy with it -- no more rebuilding from scratch every time a boot disk blows!! :) I'd started converting to SCSI boot disks everywhere, but a pair of software RAID2 IDE drives gives me a much better sense of security. My workstation did in fact blow a boot drive a month ago, and rather than being an emergency, I just ambled into Fry's after a week or two and bought a replacement and rebuilt the raid. No muss, no fuss. Feels like living in the third millenium!
I did learn various things the hard way that the HOWTOs don't warn of.
Note that you can't boot off RAID5, only RAID2. The hack they mention of putting /boot on RAID2 and everything else on RAID5 is not worth it with today's drive sizes. Give yourself a 2-16GB RAID2 with a complete bootable system on it, and save yourself mucho grief at very little proportionate cost in disk space.
As of kernel 2.4x, at least, the linux software RAID5 autorecovery is workable but less robust than one might like in the face of serious problems: I had one RAID5 setup totally destroyed because the hardware was flaky leading to constant reboots while RAID5 reconstruction was just underway. After awhile the kernel got confused about the order of the disks (which shouldn't matter, but apparently did) and the whole thing went into a Death Spiral. Lesson: If you're sure the problem is just one flaky disk, feel free to just swap in another and reboot. But if you are in any doubt, play it safe: Switch off RAID autodetect first thing (by fdisk'ing the partition type from FD back to 82. Get the hardware stable, rebuild the RAID by hand, then switch everything back to FD.
RAID5 is a comparative pain in the ass to work with vs RAID2, because under RAID2 any of the partitions can be mounted normally as a non-RAID drive in an emergency, getting you back on the air fast, but not so with RAID5. (You'll want a live Linux CD with a RAID-supporting kernel, likely. Knoppix &tc don't yet ship this way.) So only use RAID5 if the extra space really matters -- the big servers.
BTW: One of the reasons I like software RAID over hardware: If you have hardware RAID and the controller blows and you can't find a matching model, you may be stuck reverse-engineering their RAID scheme to recover your data. No worry about that under software RAID.)
I tested automatic failover to hot spare disks under the kernel, and it worked perfectly for me in a handful of tests. For whatever that's worth.
Do keep an eye on /proc/mdstats readout of your RAID system health. If you're asleep at the wheel and don't notice anything until enough disks fail to bring the whole system down, you haven't gained much. I have a crontab-driven set of Perl scripts which check all sorts of things weekly to minutely and email me if they look wrong: Checking for failed RAID drives is one of the things they do. If you don't have a comprehensive solution like that, the raidtools2 package has an ad-hoc solution specifically to email you on drive failure. USE IT.
FWIW, here's the system I've evolved for partitioning disks in such systems:
I use linux software raid5 on seven 200G disks with LVM on top, and have had good results. It's usually going to be much more flexible than hardware raid. If I run out ide buses, I can use firewire, or scsi, or sata or whatever I want. You can also use different sizes of disks (within some limits). With hardware raid, you're stuck with the number and type of ports on that raid card, and that's it.
/dev/md partitions, then concatenate them together with LVM into one large partition. When I go to add disks, this allows me to pull one md partition out (if I resize the fs down far enough), and expand that md onto another disk, then add it back into the volume. Raidreconf still doesn't sound reliable enough, so I avoid that for resizing
I make lots of smaller raid5
If you do go wtih hardware raid, make *sure* you can do a resize of the raid without losing your data. A lot of cheap raid controllers don't allow this - you have to wipe out all your data in order to add another disk to your raid, which is usually impractical. And you have to assume you're going to expand it.
Also, make sure to turn off write caching on your drives. It's much slower, but write caching is dangerous, especially in raid configurations.
I have build at least two dozen software RAID5 boxes over the past few years. Usually Promise controllers, Maxtor drives. Performance is generally pretty good. Here are bonnie numbers for my 1.2 TB media server (five Maxtor 300 GB drives in Software RAID5). These numbers are a little slower then other systems because it uses an Athlon motherboard. I have found that Intel chipset boards generally give read performance ~100-140 MB/sec.
[root@media root]# more bonnie20.log
Bonnie 1.2: File '/raid/Bonnie.27772', size: 2097152000, volumes: 10
Writing with putc()... done: 14517 kB/s 83.2 %CPU
Rewriting... done: 25060 kB/s 17.1 %CPU
Writing intelligently... done: 41987 kB/s 29.5 %CPU
Reading with getc()... done: 18830 kB/s 96.1 %CPU
Reading intelligently... done: 82754 kB/s 62.2 %CPU
Using an older processor/motherboard is probably not a huge concern. I've used 300 MHz Celerons before. Of course, your performance might not be as high as this, but if you are using this as network attached storage (NFS or SMB), you will likely be limited to 12 MB/sec due to fast ethernet. If you have (and need) gigabit transfer speeds, you should probably use a better motherboard/CPU.
Lastly, remember that you shouldn't skimp on power supplies and an UPS that automatically shuts the system down. The *only* data loss I have ever had on raid5 arrays came because of power-related issues. Heed my warning! 8)
Funny I can see.... but Informative??? Some people find inspiration in the oddest things.
So you stuffed up (didn't create it properly). That says nothing about Linux software RAID. It's just as easy to break a 'hardware' RAID, and what happens when the RAID hardware itself breaks?
Bullshit.
RAM is not an issue at all, you can go with 256M or less and be fine.
You'll likely take a performance penalty from the IDE bus though because when writing to the array your data is sent to n drives instead of one. Caching helps here but if you're paranoid about power outages you'd better turn that off. Other than that linux softraid is perfectly fine and quite a lot people use it in production systems. If you need better performance you'll have to spend on SCSI or one of these hardware-raid IDE cards or a bit of RAM for caching but don't expect wonders from the latter...
More than that. You'll need to buy cache memory for it.
Democrats or Republicans. They are both taking us to the same place and they are not afraid of us anymore.
"I am trying to build a large home drive array on the cheap. I have 8 Maxtor 250G Hard Drives that I got at Fry's Electronics for $120 apiece. I have an old 500Mhz machine that I can re-purpose to sit in the corner and serve files."
/dev/hda-/dev/hdd instead of /dev/hdm-/dev/hdo.
/mnt/data1.
I just setup a very similar system on an Asus A7N8XE Deluxe nForce 2 motherboard. I have 6 200GB PATA IDE hard disks, 2 on the mainboard IDE, and 2 each on Promise Ultra133TX2 IDE cards, 1GB RAM, and a Duron 1800MHz CPU.
The board rocks. Onboard USB2, Firewire, SATA, PATA, Gigabit Ethernet + 100MB Ethernet, and dual banked DDR RAM for extra speed. It runs just fine with a stock kernel and no binary drivers.
Software RAID is fine for personal systems (which mine is). I'd use an Escalade or similar card for a business system. It simply wasn't in the home budget which is prioritized for the wife, 2 year old and 3 month old instead of leet rigs.
Unfortunately I have no idea how hard software RAID is to setup on a Slackware type box. I switched over to Red Hat in the mid 90's when other things were drawing more of my time and interest than maintaining packages. Fedora Core 2 has a point and click RAID creation facility during installation but even as a GUI tool it's real drudge work to setup (unless you just make 1 giant partition)
A few lessons I learned:
*** RUN BADBLOCKS ON YOUR DISKS BEFORE USING ***
I skipped doing this and lost a few hundred gigs of replaceable data that nevertheless i'm not looking forward to replacing. I know backup backup backup. Easier said than done at those volumes. You need to run badblocks in one of the write modes because IDE hard disks will not remap bad sectors unless they're written to first.
Let it run overnight. Before running badblocks one of my drives kept eating my data, and it would not pass a SMART test. I was ready to RMA it before I stumbled on the fix. After running badblocks on all drives once all night long I've been rock steady. I now run SMART tests periodically and use smartd.
Between 2.6.5 (Fedora Core 2 install kernel) and 2.6.8-1 Linux changed the order that the drive devices are enumerated, which caused weird boot problems until I figured out that it was trying to boot drives off my SATA controller that I don't have when it suddenly made the SATA
I chose to use REISERFS. The people at namesys are great, and I don't have to worry about having 964GB of space and no inodes left.
I initially had 4 drives on my Promise card. I was less concerned about speed and more about space since this is a personal server. I had tons of interrupt errors and it turns out the Promise 133TX2 cards can't really handle 4 DMA100 drives, breaking the drives up to multiple cards made a big difference.
Incidentally - I'm running about 3GB of SWAP (I currently have 1GB of RAM - but I saw no performance difference when it had 256MB), with a 4GB root partition, and the rest is under
Heat is another HUGE problem when we have this many drives. In my initial setup I figured just having fans was enough. When my internal case temperature hit 160+ degrees I reconsidered. I now run the case open with a fan blowing across it.
- Dave
http://www.bullnet.com
I've not tried s/w RAID-5 under Linux, as, among other things, a Solaris DSO course I went on recommended against it so heavily -- and not just because Sun wanted to sell us more hardware ;-)
But anyway, I've striped partitions across Linux (LVM2) and Solaris disks (Disk Suite/Vol Manager) several times -- RAID 0 I think that is -- and that seems to be pretty efficient.
Mostly it has to do with the fact that good h/w RAID has one channel and possibly bus dedicated per disk (whether SCSI or IDE) which means there's very low access-contention, but it also helps (a lot) that the dedicated controller handles all the splitting/consolidating of the data across the disks, parity-checks when a disk is replaced, etc, etc, so the OS can get on with what its better at.
Sharing a channel/bus (e.g a single IDE cable or a single PCI Bus) can really affect read/write performance because the data will have to be consolidated in a divided manner across those same channels/buses from all the disks (or all-but-one for RAID-5). The speed of those channels/buses can vary a lot between systems and architectures, though a 500 MHz CPU suggests a less-than-desirable speed for this task...
Parity-checking a replacement disk will also absolutely destroy disk-access speed -- *all* the data on *all* the disks has to be read in to write the correct parity to the new disk -- but a good h/w RAID should still allow the array to be used at the same time without too much pain.
Anyway, since you've got the disks already, it would be worthwhile playing with RAID and finding out for yourself, but if you can can't spare the time (or headaches) I'd definitely think seriously about a h/w RAID solution.
Writing large files to an eight-drive RAID-5 arary will be butt slow unless you have a LOT of RAM.
Not unless Linux has a much worse implementation than MS -- I've had extensive experience with it there, and the performance is quite good.
The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive.
Um, not quite. Some review of RAID levels would be in order. RAID 5 does impose read/modify/write overhead, but there is not a dedicated parity drive -- parity is striped.
My first question to the original poster, however, would be what controllers he plans on using. By the time he connects 8 drives, using N controllers, he's a short distance from what a 9508 from 3ware would have cost.
--- Bill
> well i was talkign about more or less cheap controllers. in the arena of raid controllers 160USD is fucking cheap!
/boot / /tmp and /var /home
/, I did the same thing (pair of RAID 1s) and put a RAID 0 on top of that, for a net of 40gb redundant and fairly speedy storage.
/tmp and /var I made a simple 10gb RAID 0 for each. Not a whole lot of need for redundancy here, I make a point of backing up the important /var stuff.
/home
Not compared to $0.
You see, the typical budget RAID 5 builder just wants to store his collection of MPEG4s, MP3s, and other downloads or perhaps uncompressed hobbyist video. It's not a database, it's not a 150+ employee corporate file server, it's just personal. Performance is not a concern.
And if performance is a concern (say he wants / on these disks) then the cheap way to go is software RAID 0, 1 or 1+0 (aka 10) *COMBINED* with a RAID5.
For instance, I just built myself a new system with four 300gb drives and partitioned each one like so:
50mb -
1gb - swap
20gb
5gb -
For the 50mb, I made a bootable RAID 1 of four drives (grub can boot this, dunno about lilo)
For the 1gb swap, I made a RAID 1 with two drives and a RAID 1 with the other 2. Thus I have a net of two 1gb swap partitions, with redundancy so my system will never crash due to drive-induced paging errors. This is essentially a RAID 0+1, though I let the kernel's swap system handle the RAID 0 aspect by giving them equal priorities.
For the 20gb
For the 5gb
With the four equal-sized partitions that were left, I made the RAID 5 for
Don't you see what a great cost-effective approach this is?!?
Maybe you work for some company with plenty of money lying around for $160 RAID controllers. But I'm in business for myself, and I don't see the sense in spending money where it isn't needed. Besides, the flexibility of software RAIDs (per-partition, not per-drive) would be well worth it to me even if something like the SX4 were cheaper.
The unofficial
If you are storing data that will not change much, it may be simpler to simply make a cron job to copy the data from one drive to another. Set the copy to happen in the middle of the night and you're good. You'll have a copy of the data on a drive that you can put into any machine, and not have the overhead of software raid. I've used software raid, but found that it is slow when you have more than one drive on an IDE channel.
Consider one row in a 7 drive array. It has 6 data blocks and one parity block:
D1 D2 D3 D4 D5 D6 P1
where P1 is the XOR of D1 through D6.
If I write to D1, but leave any of D2-D3 alone, then it is necessary to read SOMETHING in order to calculate the new parity. Yes, I know what I'm writing, but unless I overwrite the whole thing, I must perform extra operations in order to update the parity block correctly. These extra operations degrade performance, and are known as the small write problem. As another AC above said, the update can be done with two reads and two writes; read the old D1 and the old P1, then write the new D1, and write P1 to be (old D1 XOR new D1 XOR old P1). It's a bit of trickery, but it does give the correct parity block. It does, however, take two reads and two writes, to update the one block.
Linux (last I looked) doesn't do this. Instead it takes the simpler approach of reading the blocks in the row that it isn't updating (D2 through D6 in this case), and then computing P1 as the XOR of D1 through D6 again.
The small write problem is a big deal. Although the IOs can happen in parallel, the latency for the write becomes the maximum of the reads plus the latency of the parity write. The larger number of IOs also keep the array busy when it could be doing other things, which degrades the performance of those other operations. And it causes this performance degregation for small updates (those under the stripe size), amoung the most common operations. Even if all of your files are big, and written in a streaming manner, the metadata updates are generally in a different row in the array, and are small, isolated writes. A journaled file system, depending on how it is implemented, can be much worse for generating lots of scattered writes. Here is a paper from CMU that gives one possible solution (one that isn't implemented by Linux). The traditional solution is write caching--you delay the write until either you've updated the other entries in the row, you've read the other entires in the row, or it is otherwise convienient to do the update (i.e. array isn't busy). This is of course dangerous because your data isn't on disk but in RAM. OTOH, witha good UPS, loosing the contents of RAM is a relatively rare event. To sidestep the volitility of RAM entirely, nice HW raid controllers have some amount of non-volitile memory (either NVRAM or battery backed DRAM) for this purpose. Writeback caching can also help perfomance on non-RAID devices, since it allows you to reorder the writes to minimize head seeks and rotational latency. These two, especially the head seeks, are what make disk IO slow.
You obviously don't have much background in storage. Try reading Chen's classic paper on RAID, go and search for a few papers the reference it, and then come back and spout off. Until then, quit giving people bad advice.
Presumably, we are talking about the parity disk here, otherwise there aren't any reads involved at all. In that case, you need to read the old parity in order to compute the new one.
RAID 5... for what? How is it going to be connected to your other computers? Do you have 64bit pci? Total system throughput and network latency will be other issues you'll need to account for. I've used SuSE software raid and OS X software raid... OS X is running like a champ but SuSE's array failed after about a month or so. Not sure if it was a software issue but I'm pretty sure it was. If it's for something like serving mp3s or small files it will probably work just fine but if you're going to be streaming video other than anything above mpeg-2 you should invest in a nice low-end fibre channel solution and save the cpu for something else.
||| I still can't believe Parkay's not butter.
Usually true, but I think the GP is talking about updating the parity block when just one block is updated. Still, all other reads aren't necessary as the parity block is updated (XORred) with the XOR of the old and new data block.
Your case is even better of course, if the block device logic is smart enough to group transactions like these together (safely), which involves (safely) waiting to see if any 'close' transactions will be happening..
Comment removed based on user account deletion
I have used software RAID-5 under linux for a year or so, and it has survived two hot-swapped spares, transferring of CD and DVD images, backups and restores of entire 120GB hard drives, etc. Transfer speeds are as good as a single physical drive, especially since each drive has its own channel (this may suffer a bit if you put both a master and a slave on each channel). i have done three OS re-installs and a few kernel upgrades without losing any data. My file server has only 512MB of ram, and an athlon 1800+, so it's not superpowered or anything, and my array has a terabyte of usable space, so it should be comparable to your situation.
The only PITA part of software raid under linux is the initial array rebuild, which took 3 days (under load) to complete. However, during that time, the raid is still accessible, just not redundant.
This post is full of nonsense (see replies below, some of mine)
Everyone has their own experience with a bad hard drive or a bad batch of drives from a vendor. I know people who swear Seagate is the worst manufacturer, and then some that say WD is awful, and Maxtor, and etc etc etc.
I've used them all, Seagate primarily though (SCSI servers), and have noticed a trend. They all suck the same!
The sooner we can move to cheap solid state storage the better.
Nothing like doing a little cursory research at the more prominent documentation sites for topics on this matter.
In summary:
How stable is the current RAID 5 support in Linux? Quite. It's really the only way to go, and performs about as well as hardware raid.
How hard is it to rebuild an array? It's not. At all.
How well does the hot spare work? Seamlessly. Be sure to use LVM, as it makes things all the more seamless.
Will it rebuild using the spare automatically if it detects a drive has failed? Yes.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
Comment removed based on user account deletion
Apple uses Maxtor in their lower end Macs (and around the year 2000 used them in most of their product line).
I've replaced 5 faulty Maxtor hard drives out of recent Macs owned by friends. Ergo, imho, the drives suck and I don't want anything to do with them.
I was very intrigued by your sudden interest with data integrity and redundance after just buying Maxtor drives...
It strikes me as absolutely bizarre that someone could have such a large collection of porn, unless he's trying to profit from it. What about copyright violations? I doubt that he's produced enough porn to fill up such massive storage.
I've used software raid 5, I've always found it more reliable than hardware. and i've recovered from multiple disk failure.
I am looking at switching to raid 6 on a 2.6 kernel. 2 disk redundancy.
FYI: Promise has seeminly added a crippling feature in their latest non-RAID EIDE controllers to avoid folks making cheap soft RAIDs.
You used to be able to use as many Promise PCI IDE cards as you had PCI slots - I've had up to 5 Promise cards previously, ATA-66 vintage. However, the newer PCI IDE cards will give fatal errors if you have more than 2 in your system - drive timeouts, write errors, etc.
I confirmed this with their tech support. This is something in their BIOS, and cannot be avoided. I'd guess that they purposely did this.
This is not an issue with "generic" IDE controllers, like the one that SIIG makes that has a Silicon Image chip on it. Same price, no hassles.
I have two servers at home using it. One is a file server that has been in use for four years, no issues. The other is somewhat newer, bigger, and has been in use for 6 months. Built one for a friend that is over 1TB, also going strong. They do 50-60 MB/sec easily on reads.
Stick with fairly decent hardware - cheap Intel branded board, good "generic" IDE controllers. Give each drive it's own IDE port. You'll be fine.
For the most part the compaq smartarray line of cards are backwards compatible and you can upgrade or downgrade the cards without changing anything.
I have some limited first hand experience with this but I have only done it a few times so maybe someone else can tell of their experiences.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
"the bottleneck is in the networking"
I'm not sure about that. A gigabit is still 125+ megabytes a second - it might bottleneck your burst rate, but it's unlikely your array is going to saturate it besides that. Especially if multiple people are using them at once.
If you're really concerned about bandwidth, I would buy a second NIC for all your computers and simply make a "storage network". That's something of a hack, but I think it would work pretty decently.
Of course, you're SOL with wireless.
-Erwos
Plausible conjecture should not be misrepresented as proof positive.
the older model 3ware cards work well but have less than stellar performance. you won't get more than 30-35mbyte/sec tops of hw raid5 thruput on a 3ware 8xxx or 7xxx controller.
that said the 3ware controllers are the -best- multiport ide controllers out there, especially for SATA support under linux.
We use them and run linux sw raid5 across all the individual disks on the controller. Excellent performance (it max's out the 32bit PCI bus, easily 100mbyte/sec reads)
I haven't tried their new 9xxx series cards. hopefully those have real speeds for their hw raid5.
Jeez, I've never seen so many plain fools in all my life. Hardware RAID controllers! How quaint.
Here's what you do with those 8 fine drives of yours.
You'll need 9 486's. Get some sort of *nix on each one, preferably several different Linux variants and at least 2 BSD machines (I'd say more, but you know, netcraft confirms and all....) and get them all networked together. Put one drive each in 8 of the machines, format with the filesystem that's most convenient for the system on each box, and get an NFS server going serving that partition.
Then, on the ninth box, mount all the NFS shares and software RAID them.
Trust me. This is exactly what you want to do, and anybody who says different is a dumbass. People who point out what they will invariably say are "obvious shortcomings" of this setup are merely trolls, and not worth your time reading.
Given a choice between free speech and free beer, most people will take the beer.
Vinum on FreeBSD absolutely rocks! You're old 500MHz machine will run FreeBSD beautifully too.
Anybody here remember Walnut Creek's huge ftp archive at "cdrom.com" which back in it's heyday of the late 1990's used to be the biggest, most highest traffic ftp download site on the planet? They used a combination of Vinum software raid and Mylex hardware raid to handle the load. I remember reading a discussion article from them once that until you get a totally ridiculous volume of ftp sessions hammering away at ther arrays, that Vinum was actually a slight bit faster than the hardware array controller.
Sometimes hardware RAID does sucks. We had a HW controller (I think it was 3ware) that sometimes took 30 sec to delete a 100M file. We sent that one back.
Software RAID does have some real advantages, like speed. Put disks on multiple SCSI or IDE controllers with a dual CPU and you can _really_ fly. Also, you can recover from some really screwed up problems with mdadm, if you know what you're doing. Consult the linux raid mailing list: the feedback is great.
That said, hardware raid is much easier.
For a little more (well, maybe more than a little) than the amount of coin a lot of RAM will cost you, go get a 3Ware 8 port RAID card instead. I run one, and it kicks ass. I see the one we got (Escalade 7506-8) on Pricewatch for $366. The RAID is fast, fault tolerant, and has a little web interface to let me know its status. I've currently got the drives configured as 7 in a RAID 5 with the 8th as a hot standby. I am very pleased with it.
Just to insert my 2 cents into this, I have a 4 disc 750 raid 5 SATA array, on
a PIII with 128 megs of ram. LVM on top of the array, and I have never run
into a problem with serving files via NFS or SMB. More ram is always nice, of
course, but again, I have not ran into any problems.
SealBeater
-- Its survival of the fittest...and we got the fucking guns!!!
Fine. Didn't realize you slashdot types were so untrusting. =) At my work we initially went with about 12 systems all with assorted 8 port or 4 port 3ware cards. Out of the 12 systems, only 3 of the RAIDs are currently in working order, and I expect they'll fail shortly. Sure they work great at first, but my idea of a nice RAID 5 setup is that when one drive fails you pop a replacement in and you're good to go. With 3ware cards, a drive fails, you reboot your system, and then you can't even boot up until you pull the 3ware card, or pull all the drives off of the card. I've tried and tried to slowly add the drives back but I always seem to end up with an unbootable system. I'll tell you, I'm 100 percent happier with the ide to scsi RAID boxes we now go with.
Free will is just an illusion
I have machine running with software raid-5 that uses software with database(mysql) connectivity for roughly 100-300 users simultaneously, load isn't that bad, ~30 queries/second, which gives about 0.05-0.30 average system loads (2gig/512ram)
The raid is made from three SATA disks, and each drive has its own controller.
Biggest problem that I have encountered so far is that when I upgraded from 2.4 series kernel to 2.6, the sata devices changed from hdx to sdx and suddenly the arrays vanished.
But that was easy to fix just by telling the software where to look for the drives.
Recovery seems to work well from past experiences on older system. Had ide drive failing, I didn't have spare available right away so I just mounted the array as degraded readonly meanwhile.
After getting replacement, array took awhile to rebuild, and ~8 hours later it was up and running in readwrite mode again.
Haven't tried if hotplug works with serial ata drives but I'm bit suspicious and I'd rather take the system down for a moment to fix it instead.
There are no atheists when recovering from tape backup.
The only time this would be a problem is when a whole new file is being written.
That's the second-most-common operation on file servers (the actual reading of files being the most common).
I don't mean to sound like an ass, but has this guy even RTFM?
3cx.org - A truly bad website.
And keep all those drives cool; use fans.
It seems to me it should be possible to put another power supply in the case and power the extra drives from that. Most modern BIOS versions allow specifying a delay on the hard drive spin-up. Just put the power-on connector in parallel with the main power supply connector. I don't like Antec cases much, or those from any manufacturer, but some full tower Antec cases have room for a second power supply.
But a case with a lot of drives and a motherboard needs a lot of air-moving equipment.
--
15 of the 19 9/11 attackers were Saudis. Many don't like the U.S. Gov. influence on their country.
Office Depot had an 18th anniversary sale, and was selling Maxtor 60GB drives for $18 after rebate. Bought three for my personal test machines, and used my friend's addresses for the rebates.
I often hear bad things about Maxtor drives, but after a whole 40 hours use, they haven't failed once.
Anyone have experience and knowledge to compare these two animals?
We have good luck with drives. The reason appears to be that we install them with their own fans, so each is individually cooled.
--
100 Facts and 1 Opinion -- The Non-Arguable Case Against the Bush Administration
I've had really bad experiences with early Silicon Image based controllers. Those Koutech controllers use Silicon Image chipsets. Is Silicon Image better now?
--
George W. Bush's brother was shown in a lawsuit deposition on 20/20 talking about his prostitutes and using government influence to make money. Family values?
Boot/system - mirror it, forget RAID ! Application data - depends.. Most important, separate the logical and physical access thinking. Indexed data - mirror keys, RAID data! Now - I know, mirroring is (kind of) RAID but not really the same. It really depends on what is the purpose, much less what is the technology. You back up the data ? You use raw disk access or file system ? You use journaled file system or not ? If so - where are the journals ? See - there is much more than just RAID. And so on, and so on.. Have fun.
What type of external enclosure can one use for ATA or SATA drives? All I've seen for sale are SCSI or Firewire enclosures.
Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
I've been running a RAID server with Linux for about 6 years. I ran software RAID-5 for the first 2 years or so and there were several things that came up. There were a few times when the system crashed and the array was out of sync (not all the drives had the same sequence number for the last write). It required some manual troubleshooting and head scratching. There was also the issue of strangeness when the hardware failed. A few times I had a drive go bad in such a way that it was not visible to the BIOS or boot scripts. In those cases the software RAID drivers got confused. They would not allow the operation of the array with a missing disk. I had to install a drive (any drive) even to just use the array in a non-redundant mode. Also, if you're going to do software RAID with IDE drives, absolutely DO NOT use more than one drive per controller channel. It may work for a short time but you WILL have problems.
After a few years of problems like this, I never had a warm feeling about the reliability of the system so I installed a 3ware hardware RAID controller (IDE). After another few years, I upgraded to another one (SATA). I've been most impressed with 3ware's product performance, reliability, and support.. I highly recommend them.
--
JSL
After months of problems with DMA timeouts and lockups caused by using a Highpoint RAID controller and a Promise IDE controller I finally bit the bullet and bought a 3Ware Escalade controller. All the sudden, everything is completely stable.
Do yourself a favor and get a good hardware raid controller and make sure it has good Linux support. Promise sucks. They advertise Linux support on the box - they lie, only with specific 2.4 kernels. 3Ware has good driver support for Linux included with the Linux kernel source code.
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
A great deal of the posts here seem to be implying that the purpose of RAID is to protect your data. This is dangerous. Most data is not lost through hardware failure, but rather through human or program error. If you accidentally do an "rm -rf" on the wrong directory, RAID will still destroy the data. Protect data with backups: backup to tape, backup to another computer, backup to DVD or CD, but never think of RAID as a substitute for backups. My other advice is keep in mind that if you don't watch your logs you may have a drive fail and not know about it for some time. Linux RAID should be paired with log monitoring of some sort, even if it's just a cron job that greps for raid messages.
You can get a relatively slow 5800 for under $50, which should perform better than software RAID. For a little more, you can get a screamingly fast 64 bit PCI card.
These allow Raid 0, 1, 5, 10, and have great Linux support including a daemon to fiddle with the settings and the option of a hot swap drive bay.
I have been using RAIDFrame under OpenBSD for about 2 years now. Never had any issues. From dmesg:
/etc/raid5.conf
/dev/wd0a /dev/wd1a /dev/wd2a /dev/wd3a
/usr/src/sys/arch/$ARCH/conf
../complie/RAID /bsd /bsd.old /bsd
wd1 at pciide1 channel 0 drive 0:
wd1: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd2 at pciide1 channel 0 drive 1:
wd2: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 5
wd2(pciide1:0:1): using PIO mode 4, Ultra-DMA mode 5
pciide1: channel 1 configured to native-PCI mode
wd3 at pciide1 channel 1 drive 0:
wd3: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd4 at pciide1 channel 1 drive 1:
wd4: 16-sector PIO, LBA48, 156334MB, 320173056 sectors
wd3(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 5
wd4(pciide1:1:1): using PIO mode 4, Ultra-DMA mode 5
Kernelized RAIDframe activated
raid5 (root): (RAID Level 5) total number of sectors is 960429504 (468959 MB)
The setup is very simple.
# cat
START array
1 4 0
START disks
START layout
32 1 1 5
START queue
fifo 100
It has been quite stable. The box is a 1Ghz AMD with 256MB of ram.
I tried running the same setup under Linux (Gentoo & Slackware). The software RAID would crack under load and fail a disk. I really would give OpenBSD a try with RAIDFrame. You have to recompile the default kernel with RAID suppot but under BSD it is very simple. CVS the source down and:
cd
cp GENERIC RAID
vi RAID and add:
pseudo-device raid 9 # RAIDframe disk driver (Make the number, 9 in this case one more then the number of disc you are using)
option RAID_AUTOCONFIG
save the file and:
config RAID
cd
make clean; make depend; make
cp
cp bsd
and reboot.
Then just read the main pages for raidctl to see how to set it up (hint: look at my raid5.conf above).
Hope this helps.
I'ved use software raid 5 & 1 on over 6 servers with good success. I've ran several benchmarks on machines using software raid and hardware raid. They both offered compariable performance except in one area. If you constantly fsync such as unbuffered always sync syslog writes. Now this is slow no matter what but it was extremly slow in software raid.
This isn't really much of a problem as you don't really ever want to write to disk unbuffered always calling fsync.
I got me really bad when I turned up the logging in postgresql. While the performance degraded in both software and hardware raid. In the software raid it was really slow. However if you do buffered logging it is equaly fast on both hardware and software raid and almost no performance degradation.
So just remember don't do any unbuffered always fsync writes to disk in software raid. In most cases you'll never do this.
To put in my $.02, we have a dozen servers with > 500 gigabytes each on 7500's and above. They are wonderful controllers, I am going to get one for my home Mythtv box here Real Soon Now.
We have built a dozen RAID5 file servers running Linux and used a 3Ware Escalade hardware controller every time. Every machine has worked flawlessly and continues to work. This includes our off-site backup system. Nothing else has worked as well and, for the $150 or so the card costs, it's well worth using.
No one ever had to evacuate a city because the solar panels broke!
I haven't read all of the comments in detail, but I think one thing that people are often forgetting is that a standard PCI bus has a theoretical maximum bandwidth of 133 MB/s, a level you'll probably never see in real life, especially when there's a fair amount of chatter on the bus from different devices (and you'd get a lot of that with 8 drives plus networking plus who knows what else). Of course, PCI bus layouts vary considerably between simple motherboards and high-end ones.
I don't know if anyone makes PCI-X ATA-133 controllers (non-RAID), so in the final analysis it might be best to get a 3ware card with a 64bit connector and plop it in a long slot. Of course, you need a pretty nice motherboard for that. I guess I haven't gone shopping recently, but they weren't that common the last I checked (and everyone is going to head for PCI-Express shortly anyway).
Of course, it all depends on what you'll use the machine for. If it's just file serving over a 100Mbit network, there's no need to worry that much about speed. It's only a big deal if you're concerned about doing things really fast. I believe good 3ware RAID cards can read data off a big array at 150-200 MB/s (maybe better). My local LUG put a ~1TB array together for an FTP mirror with 12 disks (using 120GB and 160GB drives, if I remember right) about 2 years ago, and testing produced read rates of about 120 MB/s on a regular PCI box (I think.. my memory is a bit flaky on that). Of course, I don't think anything was being done with the data (wasn't going out over the network interface, to my knowledge, just being read in by bonnie++ I suspect).
I was running a P3 600, on an 810 board for a while running with 3 seagate 120gb drives for my brother who lives in a flat under my house. it was definitely not quick, but stayed up. The performance was pretty crap compared to the same hardware running a mirror, but a large part of that was the crappy controller on board.
:/
:-)
:-)
;-)
I upped the box to an athlon 1200 on a sis mainboard and things improved quiet a bit. (both boxes had 256mb sdram) The raid and network performance doubled, and tons of delays disappeared.
After complaints (from the windows user) that it still wasn't utilising the bandwidth well enough, I reluctantly used one of my main PCs (athlon 2400, via kt266, 1Gb) and also tried an extremely cheap Sil680 controller (used just for the disk interface). The individual drive speeds went over 50mb/s. The raid5 was still only running at about 35mb/s. And of course no faster over the network - just a waste of a machine.
After not being able to stand the fan noises of the 2400, I split the drives off and put them in my brothers windows box as a mirror, seeing he's the only one who needed the reduntant space (he has some bizarre psychological need for 'warez' and mp3s) They failed within a day with loads of data recovery fun. He's since warranty swapped two of the for seagate sata's, and reinstalls once a month, and i've stopped caring about the plight of his O/S or warez..
These seagate drives were less than 5 months old, and were getting loads of seek & ecc errors from the start. The were all running with enough cooling (it's summer here 9 months of the year), and the temperatures were monitored constantly. Just crap I guess.
The moral of the story: use decent long lifetime drives from the start, of different batches, don't throw more money on hardware than you have to (a cheap faster ide controller does wonders), and use a decent O/S that is stable and lets you optimized to hell (ie gentoo
Oh yeah, and kill all windows users..
Yeap, you said it, it LOOKS nicer.
But on storagereview.com forums, many are saying how its performance is really sucky.
I have not tried it yet, and will soon be in the market again to move on to the latest PCI-e SATA II controller, but 3ware... I'd pass.
Currently I'm using a LSI Logic card, very good.
LSI Logic are the guys who made Megaraid, so they do know their stuff.
I've also used an Adaptec before, pretty fine card too.
Online backup with Mozy, sounds like Ozzie, but more!
in linux, raid5 is a very solid and fast solution. even with a 500Mhz, it is faster than all but the most expensive hardware cards, as most cards have a 133mhz chip or even less.
:) should give a nice performance boost on many systems. this patch is designed to improve transfering files on networks but is show to work equally well for local devices.
also, software raids are hardware independent. they can be modified easily while booted and without rebooting. if a hot-swapable drive is used, downtime can be eliminted by a hot-swap and a rebuild of a failed drive.
also, i have been in a discussion about the new cachefs patch in rescent mm kernel patches(or maybe nitro?), allowing you to use a cache in ram with any filesystem, so you could mount your raid array through the cachefs with a given amount of RAM for write cache
AND, linux software raid works on a per-partition basis, so you can mix and match drive sizes without wasting space. 8 250GB drives can mate up with 4 300GB drives, and then the wasted 200GB can be made into another array.
you can easily add IDE cards and increase the size of your array.
you can spread your array over a large number of IDE cards for better redundancy, no single card will criple your array, and IDE cards are much cheaper than hardware raid cards.
LINUX can be booted from a software raid! while is has trouble on some hardware raids!(driver issues)
i run a software raid5 over 12 seagate 120GB drives with no problems. i get great transfer speeds accross the (gigabit)network and it's easy to manage drive spindown because the system sees each individual drive while hardware raid solutions typically only allow the system to see the array as a single device.
most hardware arrays are mainly configured at boot time. to build or repair an array, your system will not be working. if you run a linux fileserver/firewall, your firewall doesn't function on hardware raid rebuild, while it does in software.
--
though i would go with a faster processor, you should have very good luck, reliability and performance from an 8 device software raid5. and have a nice 1.7TB array
I am very happy with my linux / 3Ware 4 port raid card combination. It makes it brain dead simple and takes linux out of the loop of things that could trash the raid. I even forgot to install the *drivers* for the raid in the initial install and it all just worked fine... because the box thinks it's one big magical drive. (The drivers were only necessary for monitoring...)
Spend the extra $200 on a 4 port card... put a *big* fan on the drives because that's the #1 killer and you'll be happy.
Pat
It worked okay until one day I had to reboot my file server (moved locations) and I couldn't get the raid to come back up. I lost all my data :(. The bad part is - when it came to forums, irc and generally trying to get help there really wasn't any, a good amount of the documentation out there and the troubleshooting information is for the older tools. I generally believe that when it comes to your data you can only trust tools you can actually support - software raid for all intents and purposes seems highly alpha/beta.
Anyhow I bought a 3ware 7450 Raid controller and haven't looked back - its brutally fast (over 20-30 megs a second in a sequental write), fully supported in linux and it a piece of cake to setup.
Its not bad at recovering either - I had a power failure and the ups failed later on - machine restarted of course when the power came back on and the 3ware controller automatically rewrote all the parity on the disks - everything was fine. While it wrote the parity the system was up and running instantly (raid was in a fail state of course).
How are you going to connect all these hard drives? You mentioned software raid... If you're planning to use the IDE on the motherboard... A motherboard that old might not recognize that large a HD. Got Raid Cards?
smartctl often comes as part of the package "smartsuite." For Debian users there is an apt package available under that name as well.
You can get smartd to execute tests automatically, using the -s option.
In my smartd.conf file, I have :
-s (L/../../7/03|S/../.././05)
on the device lines, which means do a weekly online long test at 3 am Sunday, and a daily online short test at 5 am every day.
mdadm running as a daemon, and watching the md arrays is also a good idea.
The Internet's nature is peer to peer - 20050301_cs_profs.pdf
I have a 160GB Linux software RAID-5 consisting of three 80GB disks, running 24x7 for years now. (when I built the RAID, 80GB was the largest disk capacity you could buy :).
No problems at all. I once had an IDE controller fail - I replaced it (had to reboot of course), and Linux rebuilt the array automagically.
I have not tried using a hot spare.
Warning: a lot of the documentation out there on the web about Linux software RAID is very out of date. If you go this route, DEFINITELY buy the book "Managing RAID on Linux" (O'Reilly). Also be prepared to compile the "raidtools" package, which you need to set up arrays.
I have since added an 8-disk system based on 3Ware's 9000 series SATA RAID controller. I recommend 3Ware for higher-performance systems. (I have 8 250GB disks in a single 1.6TB RAID-5, I get about 180MB/sec read, 90MB/sec write.)
If you run smartmontools, you can configure smartd to not only monitor the SMART status of the disks, but also execute online tests - have a look at the "-s" option of smartd. For my RAID1 array, for each device, I have -s (L/../../7/03|S/../.././05) entries.
mdadm also has a daemon mode which can monitor the arrays, and if there are any failures, send an email to a designated email address.
The Internet's nature is peer to peer - 20050301_cs_profs.pdf
Probably the best move is to have a cron job examine /proc/mdstat and e-mail you if it's troubled.
Sig:Why copyright isn't a fundamental human right
There is a pretty good chance he knows what he is talking about ...
The Internet's nature is peer to peer - 20050301_cs_profs.pdf
Oh, and just to clairfy, the thing won't even post with the 3ware card in. What kinda nonsense is that?
Free will is just an illusion
Cmon jack, you misunderstand me, I'm not saying 3ware is software RAID here, at least I'm sure it's not totally software RAID, although I do suspect a good part of it is done in software just like a winmodem. Anyway, all I'm saying is regardless of what you go with, be it Linux software RAID or some sort of hardware RAID solution, you're best off staying shy of 3ware products (I highly recommend netapp by the way).
Free will is just an illusion
The advantage of HW raid over software is in the speed that HW offers you. Of course, if you are doing RAID 5, you usually aren't concerned with speed as much as redundancy.
This signature is a waste of 42 characters
The woeful inaccuracy of your post is really, really painful. Allow me to rebuke.
First of all, linux software raid has excellent autodetection. You need to set the partition identifier to 0xFD so that the autodetector can identify it. As many have mentioned, software raid has a huge advantage over hardware raid for recovery - you can disconnect the drives from one computer, hook them to another and the autodetect code will figure it out. I know this works because I've done it.
Second, for 8 drives and 2 controllers a card, you'd want four ATA133 adapters. Each adapter has, as you said, 2 controllers. You don't want to sue the slave channel, because that will definately kill performance.
Third, Don't install the OS on the raided partition. Don't keep anything fragile or irreplacable on the OS partition. If you want to backup the configuration, backup the configuration. There's no need to raid your boot drive, and if your boot drive fails you can trivially reinstall.
A cheap batch script is not an effective backup solution. What if files are locked or a file is backed up midway through a transaction? I readily agree that RAID is not a backup solution, but to putting any faith in a "cheap batch script" is profoundly naive.
RAID5 has the advantage that you only lose one drives worth of space to parity information. With eight 250 gig drives on a P3 500, its readily obvious that his goal is to inexpensively store a large amount of data with an effective mitigation against a single drive failure. Software RAID5 is an excellent solution for him.
Lastly, I'd recommend one of the intel gigabit cards, because although the drives will only read 50 or 60 megabytes/s, the whole point is moot if your network connection maxes out at around 10 megabytes a second. The client adapters, like the 1000MT is more than enough, and not that expensive.
fnord.
I'm running raid 5 on i think 2.6.8 with 3 drives. That is, i'm running it on the root partition and it runs alright, although, i have noticed it has goten slugish... maybe a defrag is in order?
When i started out, firefox was loading in 2 seconds and it now appears to be taking around 4 seconds to load. At least i think those mesurements are ok. If you want real speed, i'd think about using raid01 as it seems 4 discs in a raid0 array would be faster than 8 in a raid5? I'm not too sure about that, but raid5 is significantly slower than raid0 apparantly. Also, using those other 4 discs to mirror the raid0 array could be more usful then raid5s parity/crc redundancy.
Giving IE users a taste of their own medicine since 2005 - http://pods.-is-a-geek.net/
Then the nightmare happened. The computer (an old 400 MHZ celeron) stopped accepting keyboard input. No way to get past the boot error. I moved the raid array to another machine, keeping the drives in the same order ( drive 0 on the master on IDE 2, drive 1 and 2 on their own IDE channels on an IDE card). A an install of redhat saw the raid right away and set up the disk accordingly.
Apparently redhat marks each disk with a volume label, so it knew they were part of a raid. I don't know what would have happened if I got them out of order (who knows? Maybe I did), but I certainly feel a lot better about software raid under linux.
Computers are useless. They can only give you answers.
-- Pablo Picasso
Indeed, they are.
In fact, I used to work in a datacenter with some 2500 Proliant servers running mostly the 6400 SA controllers. For about 3 years we had to shut down the datacenter completely about once or twice a year (Mainly for upgrades to the electrical systems) - and in bringing things backup we'd end up needing to replace 4-5 controllers - and they never skipped a beat! Same with upgrades, never had any problems.
Looking for hardware (Currently need: Large Etch-a-Sketch) Have one? See my journal!
Have fun with the RAID array, sounds like fun, but a little advice. RAID gives you much added protection against outages during critical hours. And it does save you from having to restore from backups *sometimes*
However most data loss occurs from HUMAN or SW errors, not HW. RAID won't protect you from either of those. If you want to have an online archive of music and movies and pic's etc... That is very nice, you don't want to loose all the time and effort you put into that.
I *assume* you have a broadband ISP connection. (cable, DSL?) You have a good friend who does too? If so, why not share the cost and the space with a friend, set up rsync between your houses for remote data backup on a regular basis. (There are howto's regarding this.)
Now you can lose a disk or a controller, or the whole server, and all you have to do to recover *everything* is set up a machine with at least as much disk space and setup the rsync to rebuild your copy from your friends house.
If you do this, you can save the worry over all the details and issues of RAID setup and failed disk replacement and such. (I mean do that stuff, but don't fret over it, because you will always have all the data backed up remotely anyway.)
I have used raidweb.com enclosures in the past and they work quite well. They handle all the RAID configuration inside the box and appear as one drive to the host (hence the boxes are totally host independent). The connection between the box and the host is SCSI and I've used off-the-shelf high-end SCSI controllers for this. Their boxes have redundant fans and power supplies. They sound like a jet taking off, but my experience is that they're stable and rock solid. They're rack mountable too.
The only big disadvantage I experienced at the time was the lack of docs on the serial controller, so I only had the audio buzzer signal to go on when a drive failed. I think the box would have sent a signal over the serial link to the host indicating a failure. Then the host could do something interesting with that signal like send e-mail, call a pager, and so on. It would have been nice to have remote signaling, but in this case I didn't need it. The install site always has someone there to handle taking out the bad drive and plugging in the cold spare.
Digital Citizen
Under 2.4, software raid 5 is stable. Monitoring it is trivial (cat /proc/mdstat). Rebuilding the array after adding a drive is trivial (raidhotadd /dev/md0 /dev/hda1). Rebuild happens in the background on the fly and you can watch it in mdstat. You can even set the partition types to raid-autodetect instead of linux (type fd instead of 83 in fdisk) and have the kernel notice and build the raid5 device before trying to mount root.
/proc/interrupts followed by eventual kernel oopses. A single drive channel worked flawlessly but simultaneous access to two drives = problems.
I use 8x250gb Linux raid 5 at work for an online backup server. Its rock solid, a real champ.
Couple gotchas:
1) Only ONE parallel ATA drive per controller. NO SLAVE DRIVES. IDE drives can't disconnect the way SCSI drives can, so having parallel traffic to both a master and a slave drive will KILL your performance.
2) Beware of multiple Promise cards in older (P2 and P3) machines under Linux 2.4. Something has gone haywire in the driver. Works fine on old Pentiums and fine on Athlons and P4s but on all three P2 and P3 motherboards I tried (two different model intel boards and an ASUS P2B) I got rapidly increasing ERR counts in
The CMD Technology Inc PCI0680 controllers (e.g. SIIG, Adaptec) worked great.
Remember: 8 drives, 1 drive per chain = 2 drives per card = 4 cards = 4 PCI slots. Make sure you have enough left for video and ethernet.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
You don't have to read the data from ALL the other drives to do a Raid 5 write, but you do have to read the data that's being overwritten, and you do have to read the parity that corresponds to the data that's being overwritten. Unless, the data you are writing on a specific IO spans most or all of the data drives, then it might be cheaper to read the data that you are not overwriting and create a whole new stripe of data. I AM (and have been for 13 years a RAID developer), but not for LINUX.
why? gigabit copper is just as fast and ~50x cheaper.
I'm running a RAID0 array on 15k SCSI HDs here, and honestly it's _not_ that much faster, except for when I'm recording audio (those WAV files get big FAST). Then again, I could still do it with no RAID.
Overall, unless you're dealing with big files, you won't gain much. (not to mention if you have too many drives in your array and not enough throughput (e.g. only ATA100 with 5+ drives), you'll really only choke your system bus.
this is a fatal flaw in the linux software raid5 design.
real (read:industrial strength) raid5 designs dont mark the whole drive unusable, just the few blocks that had uncorrectable errors.
the linux raid5 driver really needs this feature, and i wouldnt recommend using it until it does.
They make good RAID controllers too for a competetive price. (They OEM the Dell "PERC" cards)
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Mod parent up, this is great advice. I've had superb performance and reliability from software RAID 0 and 5 under Linux 2.4 and 2.6 for the last two years using a fast/wide PCI bus.
Hardware is a trio of Seagate X15-36LP Cheetahs connected to a Tekram U160 card running in a 64bit/66MHz slot of a dual Athlon XP 2000+ Tyan Tiger MPX motherboard [currently $215].
When I moved this controller and drive setup to the standard 32bit/33MHz slot of a dual P3/450MHz machine, disk performance was drastically cut while CPU usage wasn't much affected.
Modern hard drives are pushing 70MB/s transfer rates, so even two of them can saturate the classic PCI bus. PATA, SATA, and SCSI can all support high RAID transfer rates if their controllers are connected to the system by wide-enough / fast-enough interfaces and smart choices are made about [not] sharing channels.
BTW, at least a few of the common Linux filesystems are smart enough to automatically configure themselves efficiently when formatted on top of a RAIDed partition.
It's better to have a combination of both hardware and software RAID. You can take better advantage of a larger server (especially with multiple I/O buses) if you add a couple of independant RAID controllers, then bind them together with software RAID. Raid 50 comes to mind... So you can treat each controller and it's respective drives as individual volumes in the metaraid, spreading out the chance of data loss due to an adapter dying; simultaneously getting a nice performance boost.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
- The idea is that in order to write data to any sector on one of the drives, the sectors from six of the other drives need to be read, all XOR'd together, and then the result written to the remaining drive
That's not RAID 5. That's RAID 3 or 4. 5 is striped data AND parity. What you are describing has a single parity drive. (And yes, it's pretty damned slow.) With the additional caching effects, if you breifly delay writing, both blocks of a parity set may get written at the same time, so zero block reads, 3 block writes. Normally, there will be 1 block read and 2 blocks written.RAID5 is almost always better when offloaded (unless you have like mad SCSI controllers for your direct-attached discs)
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
Probably the best move is to have a cron job examine /proc/mdstat and e-mail you if it's troubled.
Or you can just have mdadmd (pard of the mdadm suite.(comes with my distro (SuSE 9.1))) running, and it'll monitor your raid arrays, and email you when there's a problem.
I lay awake last night wondering where the sun had gone, then it dawned on me.
Completely agree, but I would add an important piece of advice:
/etc/raidtab !!
If you use software raid, BACK UP
It seems like trivial info, but when I lost my os partition with a raid0 set attached, I had a hell of a time trying to figure out what the old raidtab looked like. If you get it wrong (eg in reverse order) when you try to guess, you can end up with directory listings that look fine, but you're corrupting the disk with every write.
Seems crazy, and it may have been an older incarnation of software raid, but I ended up having to re-rip 50 dvds as punishment. Make sure you back it up !!
Servlet v2.4 container in a single 161KB jar file ? Try Winstone
The advice I was trying to give was to not discount Software RAID on account of RAM or CPU - it sounds like plenty for that arrangement. It also sounds like they are not building the very best system, so most if this wont really have any relevance.
I hadn't considered the fact data smaller than the stripe size, but before you pull out your flaming gins look at the parents of my posts to see how far off they were... Software RAID is perfect for the OP's problem, no need to use hardware RAID and Linux is most certainly up to the task.
Since you seem to know so much... isn't it likely, though, for many operations, the data is already in cache? For update operations anyway, the data was probably read recently, which would have had the entire block. In this case, extra reads may not be necessary.
Everything you need to write is already in RAM except the checksum black. So if you have a 7-drive RAID5 array, the RAID subsystem can take 6 blocks of data, compute a parity block from them then write one block to each drive. It's not like it is going to write random sized chunks of data and can not tell what is going to be written without actually writing it to disk.
Even hardware RAID cards typically don't have a lot of RAM. They also don't write to the drives, re-read what they have just written and create a parity bit from it. Neither does software RAID in Linux.
The only time the system should need to read in order to generate a block is when it is rebuilding after a drive has been replaced.
You just spent $960 on the drives (8 x $120). So obviously, you can afford to spend a few dollars. So you probably can spend about $100 more and get a 3Ware raid card (afterall, you saved a lot on the drives themselves.) Since reliability is important, and you got a great deal on the drives (I hope you remember that only 1 mail-in rebate per address when you send in for your refunds), so spend a small fraction more to get something that works - hardware RAID with a real controller.
I would be very interested in doing this (in the near future) but can someone confirm that this will work without any hassles in Linux? I tried doing a setup a long time ago with external drives and ran into all sorts of problems.
The issues were related to the external controller itself and not linux, but would like to find this out before I went down the road again.
..don't panic
Ram isn't the issue.
The bottle-neck in Software RAID is not the RAM, the CPU, the extra math that needs to be done, it's the the PCI bus. In hardware RAID, you send the data to the card, and it distributes it among the attached disks. In a software RAID 1, you need to send the data across the bus once for each disk. In a hardware RAID you need to send the data only once. So far, in a soft RAID 1 with 2 SATA disks, i've not noticed any significant bottleneck.
Can you sacrafice a little bit of speed for some cheap reliablity?
What? Me? Worry?
I have heard from couple of Dell server techs and a Red Hat employee that in their testing performance of software RAID outperformed hardware. Their explanation was that in all but the most expensive controllers, the card's CPU speed was much slower than you might assume... especially compared to a fairly modern PC's CPU.
The Dell guys said that most hardware vendors don't like to mention that because it cuts into sale of the cards.
I've stuck 4 7,200rpm IDE drives in a case... and promptly killed two of them within days. I had to add a rear exhaust fan to the case, a PCI slot blower, and I removed the blanking panels in front of the drives to get more airflow. The drives now stay merely warm to the touch (instead of HOT), and the drives have been fine ever since.
8 of those suckers are going to get toasty without plenty of auxilliary cooling.
DRM 'manages access' in the same way that a prison 'manages freedom'
I've had exactly the opposite experience with 3Ware vs. the competition. 3Ware works flawlessly with drive swapping, putting in spares, bringing out spares; no rebooting necessary. This is under FreeBSD, and with the PATA and SATA versions of the card.
I'm not saying you're wrong, but I've run several servers with different 3Ware cards and have had a great experience.
Anger problem?
The question is, can two cheap power supplies be used in standard case? The question is NOT whether you can pay $2000 for special equipment.
Obviously both power supplies would be attached to the case with the standard screws. That would provide a hundreds amps worth of common ground connection. Obviously the black signal ground wires would need to be connected to each other with a small additional wire.
I am a real electronics design engineer. I can't think of any reason why this would not work. It would probably be necessary to shut off the power to the hard drives separately.
There would be a little common mode noise, however this is something that the signal ICs in the hard drives can handle.
--
24 wars since WW2: Creating fear so rich people can profit.
If the machine is a dedicated file server this is conceivable. But in that case, you're wasting your time worrying about performance on the array anyway since your network will be a super bottleneck anyway. In this situation, the only purpose of raid 5 is for data integrity.
The real place where the performance side of raid 5 matters is for things like application (especially in the web sense) and database servers. That is, where the data consumed by the server is far more significant than the data dispatched to the clients. In cases like this, CPU cycles are generally of high value and any ability to offload work to other devoted hardware is a necessity.
RAID stripes are typically on the order of 64KB-128KB. If you just write one filesystem block (4KB) you have to read the entire rest of the stripe to figure out the parity. But, modern drives are pretty good at large contiguous reads, so it's not that much of a big deal.
You are correct that modern CPUs kick the ass of most (probably all) RAID controllers at performing the XORs. The reason you'd want a hardware RAID system is for saving CPU cycles, cache pollution and memory/bus traffic, not for raw XOR speed. (in software RAID-5 you will see quite a bit of CPU time taken by the RAID computations, vs. virtually nothing with hardware RAID)
And yeah I'd go for as much RAM as possible in a file server.
I dunno what he's talking about. I built a really nice system with 4 SATA-150 drives, 384MB of ram. Is that a lot? And it worked fine. But make sure you pay attention to cooling. I used a generic case and when the AC went out in the server room, it only took half an hour for a drive to fail! After replacing it, I made sure there was enough cooling and everything worked fine, even when the AC failed again, and was off overnight.
Then I install the OS on a RAID1 set and the data I care about on a RAID5. One system I have has dual SCSI 2xRAID1 for the system partitions and 3xRAID5 IDE drives.
The CF is /dev/hda and is mounted read-only. If I need to upgrade the kernel I remount it RW and then do what I need and remount back to RO.
This is the most reliable configuration I've come up with so far. CF has no moving parts so it's fairly reliable. Writes are limited to about 1 million. Reads are unlimited.
Compact Flash Readers
http://store.yahoo.com/ituner/cfdisk1c.html
http://store.yahoo.com/ituner/cfdisk5d.html
Some boards don't like the 5D type Converter, since they detect using the cable type (40 or 80). In which case type 1C works just fine.
If you have one large partition and impending drive failure wipes out any cylindar on that drive, all the data on it is shot. That drive won't be used at all during the rebuild... a rebuild of 250Gb. You are at risk if, during any time of the long rebuild, a 2nd drive fails completely or even coughs up a bad cylindar which can't be redirected.
If you have 6 partitions, only the "damaged" one has to be recovered immediately. Obviously you would want to recover them all as soon as possible since that first drive is probably going to bite the dust soon. Even if you do lose the first drive completely and then a 2nd drive during a rebuild, you at least may not lose everything. Any of the 40 Gb blocks which were rebuilt before the 2nd drive died would have been saved.
Getting a slightly different sized drive for an RMA can also be a problem. What if your original 250Gb drives were actually 250.3Gb and the replacement is 250 Gb even? You aren't going to fit that single 250.3 Gb partition onto the replacment drive. And are you going to call the drive manufacturer and complain that your original drives were too big?
I've had issues with this on hardware RAID. I had to back up 600Gb over the network, wipe the entire array out, rebuild it and restore the data. If it had been software RAID, I could have backed up the data from the last partition into one of the others just to be safe, resized the last one, reformatted and copied the data back.
LVM with multiple partitions would have made it even easier.
Assume 4k blocks. I write 4k all in one block, now I need to read the other disks so I can computer the checksum to write to the lsast one. I only wrote 4k so I only have 4k in ram.
Firstly, you aren't dealing with sectors; you are dealing with blocks. But we'll use your terminology and stick to "sectors" for this discussion.
Assuming you start with blank sectors, you don't have to worry about the sectors corresponding to the stripe which are empty. Why? Because 0 ^ 0 is still 0. So, no dreaded reading of the 6 sectors in this case.
Now, if you are updating a sector that has already been written, then you need to just read the parity sector(P) as well as the contents of the sector being overwritten (O). Assume the new data is (N). Then, your new parity sector becomes P' = (P ^ O ^ N). So, what would have been just a write gets translated into 2 reads (of P and O), an XOR (trivial) and 2 writes (of N and P').
We have 4 backup servers currently all either 120gb x 8 or 200gb x 8, on anywhere from 600mhz-800mhz processors w/ 256mb ram each.. they are under heavy i/o 24/7 and have performed great. hdparm tweaks are definitly helpful. Throughout our various drive problems in the past software raid has been great to recover from. We've even managed to recover from loosing 3 drives already at once in a raid5 array w/ 1 parity and 0 spare. The systems often have a fairly high load so they are only useful as a fileserver-only type setup but work great. software raid runs incredibly stable and we've been very happy with it
http://interserver.net/
The one really cool feature that's been added to linux recently is the ability to grow your raid. My original drives at home were 120 GB each. I have been slowly buying replacement 200 GB drives for my raid set over the past few months (as my budget allows). Each time I'd buy a new drive I'd mark it as failed in the raid, shutdown linux and replace the drive, and then power back on and hot-add the new drive. After all 4 were replaced I ran the new version of mdadm with linux 2.6.8 and grew the raid set:Suddenly my raid set had grown to cover the full 200 GB of each drive. I just have to say, that is really cool!
-David
There. Now go play some cool javascript games!
Lets say that so little disk activity is going on that it needs to flush buffers before it has buffered 24k of data. (I'm making an assumption how it really behaves now) I would expect it would calculate the XOR across as many 4k blocks as it had and write some blocks as blank. When it went to write more data it almost certainly has those last few 4k blocks still cached in RAM.
It should be extremely rare that it actually has to go back to disk to read data for creating the parity blocks.
What if the house burns down, you're robbed, or a tornado destroys your house? Bye bye to all your data. Don't forget you need to back up your most important data, if not the whole array.
'maxtor rules': 118,000 hits
Not that Google searches are an accurate gauge of anything.
M
This logic doesn't hold. Let's first talk about the performance.
Also, on any reasonably modern system, the software RAID will be faster. You just have a much faster processor to do the RAID processing for you. The added overhead of the RAID5 processing is nothing compared to a 1-2GHz processor.
The actual RAID processing is relatively easy, and any RAID solution, be it hardware or software, that is worth anything will not have any trouble doing the logic (perhaps the cards mentioned are indeed not worth anything). The processing isn't your limiting factor; it is data thoughput. This is where hardware shines. A lot of extra data has to be shipping in and out to maintain and validate the RAID. This can easily saturate busses. A hardware solution allows the computer to communicate only the "real" data between itself and the hardware device, and then allows that device to take the burden of communicating with the individual drives on their own dedicated busses. Sure, that device can become overwhelemed, but I submit to you that if it does, it was poorly designed.
I am not saying that one shouldn't consider software RAID solutions. Just don't consider them because you think the performance will be better.
Now lets talk about data recovery.
I've lost 4 drives out of a 12 drive system at the same time, and Linux has let me piece the RAID back together and I've lost nothing. Was the machine down? Yes. Did I lose data? No. Compare that with a 3ware hardware RAID system where I lost 2 drives. Even thought I probably could have salvaged 99% of the data off that array, the 3ware just would not let me work with that failed array.
Let us be clear: we are talking about RAID5. In RAID5, you simply cannot lose more than one drive without losing data integrity. And it isn't like you can get back some of your files; the destruction will be evenly distributed over your entire logical volume(s) as a function of the striping methodology. So it is quite impractical to recover from this scenario. I don't know what kind of system was being employeed with this 12-drive array that can withstand a 1/3 array loss, but it certainly wasn't a straight RAID5. I can come up with some solutions that would allow such massive failure, but then we aren't comparing apples to apples. I'd be very interested in knowing what the solution was in this example case. It should also be noted that we don't know how many drives were in the system that lost 2 drives, much less what kind of RAID configuration was being used. No conclusion can be derived from the information provuded.
As an aside, more often than not, when we as individuals want a large cheap array, we are less concerned about performance than reliability. We put what we can into the drives, and we hope to maximize our data/$ investment while minimizing our chances for disaster. A software RAID5 is a good solution. Some posts have said that if you can spend so much on the drives, what's stopping you from spending on a nice hardware controller? I submit that perhaps he's broke now! And besides, a controller that can RAID5 8 drives is quite the expensive controller indeed. This has software RAID written all over it.
Linux software RAID5 has a considerably better chance to work nowdays. There are very few controllers out there that have unresolved bugs. Off the top of my head here are a few:
As far as controller duty roster is concerned we should also mention Via. From being the worst controller for Linux once upon a time in 1997-9 it has become the best. I have been getting better IO performance on ITX with C3 then on Xeons with server controllers for some time now (starting from around 2.4.23).
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/
... implement the RAID-5 part of the step-by-step howto in this article. It worked great for me.
I find your ideas intriguing and I wish to subscribe to your newsletter.
Your behavior is offensive.
What is worse is that you are pretending to know something about the subject, and thereby retarding a true discussion.
The "signal ground" is important because there are wires from the power supply that allow the motherboard to control the power supply. That's the "signal". Probably it would be best not to connect those lines to the motherboard, but to install a separate switch for the hard drives.
I'm hoping someone with some experience with this will discuss it.
Also, it's partition based, not disk based (under Linux, at least). This means that with just two drives you can create one two-disk RAID-1 array (for safety) and one two-disk RAID-0 array (for performance). Just create two partitions on each drive, pair the first partition on each drive in a RAID-0 config and the second partitions as RAID-1.
You can't do a single RAID-1/0 array with only two disks though. You could try, but you wouldn't gain anything (in fact, you'd lose).
Hardare raid has reasonable performance, works as it should in simple cases, but when the shit really hits the fan, you're totally screwed.
As a hardware RAID array doesn't have a console, and no option to "ask questions" to the user, it has to automatically DO things as drives are replaced.
With Linux software raid, you end up with the option of TELLING linux what the config is, and possibly recovering some part of your data, should things go seriously wrong. In a hardware RAID, you're totally dependent on the firmware in the card to "do the right thing".
Suppose you configure 7 drives as a RAID, and the 8th as a hot spare. The firmware will mark the drives with a config version and drive number:
1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7 and hot spare 1.0
Next drive 3 breaks.
1.1, 1.2, XXX, 1.4, 1.5, 1.6, 1.7 and hot spare
1.0
and the stuff is reconfigured:
2.1, 2.2, XXX, 2.4, 2.5, 2.6, 2.7 and 2.3
If now the person handling the drives makes a mistake and removes the 4th drive: 2.4 instead of the the third:
3x.1, 3x.2, XXX, 0.0, 3x.5, 3x.6, 3x.7 and 3x.3
so now whe have a broken raidconfig: "3x". No amount of replacing 2.4 will make it fit in, as the array has determined that the raid has been broken.
If you get this far, "datarecvery" is the only thing that can help. Don't write a byte on any drive and try to get things right again. Drives that can be jumpered to "readonly" would be a help. We comercially would just make an image of all the drives and work from there....
I don't care how good your RAID is, or if it's hardware or software. I have seen lightning fry every single component in a box. RAID makes it easy to recover from individual disk failures (which probaably account for most, but not all disk failures), but I'd still be doing some form of backup. Put some of the disks elsewhere and mirror your data to completely different site (if possible).
From that perspsective (that RAID lets you treaat most hardware failures as a nuisance rather than a catastrophe), hardware RAID might be attractive.
Xix.
"Everything is adjustable, provided you have the right tools"
I bought one of these from Newegg. I had a lot of problems with it. I called Silicon Image technical support. They told me that particular chipset did not work correctly, and they would not release working firmware for it.
I told Newegg about this, but they continue to sell them.
Fry's sells them also. I told a Fry's manager that Silicon Image told me they know they don't work correctly. Fry's still sells them.
I would love to find a technically knowledgeable and honest distributor.
I've got a 4x160GB SATA software RAID-5 array (about 450Gb usable) serving up files on my home network right now, running under the 2.6 kernel.
:).
These drives are all crammed into an old Dell that was my Wintendo a couple of years ago. A few months back, the grilles on the drive-bay coolers I installed got clogged up and I lost one of the drives to overheating. Upon replacing the drive, the rebuild took the better part of an evening (but didn't need to be attended). No lost or corrupt data.
The only major problem I had was that the RAID was dirty in addition to being degraded (insert "your mom" joke here), because I brought my machine down hard before realizing what was going on. In theory, I could have done a raidhotremove on the bogus drive and brought things down normally
I ended up having to do some twiddling to get it to rebuild the dirty+degraded array. I don't remember what that was, but as long as you don't do something boneheaded like ignore kern.log messages about write errors to a specific drive, get annoyed that it's taking so long to cleanly unmount the filesystem, and hard-reset the box, that shouldn't be an issue
Stay away from hardware RAID at home that's my advice. Software RAID has the luxury of always being able to see the filesystem while you move data.
And at Best Buy.
So not too long ago there was a good sale on drives at Best Buy. Went into one store and the boxes all looked like they'd been in a Samsonite advertisement.
You know the one with the angry sysadmin that smashes everything.
Well at least it looked like one of our sysadmins! :-) Needless to say, I didn't buy any of the drives.
No, you're not "done period". You'll lose a lot of data, but may still be able to recover some. Likewise when losing one disk in a RAID-0 setup.
Any file that resides entirely outside of the gap in the array can be recovered. How likely that is depends on the details of the filesystem, the striping, and the size of the file (the larger the file, the more likely that a part of it fell into the bit bucket).
Also, not all drive failures are total. You may have a RAID-5 array with one drive that completely failed, and another drive that just has some bad sectors. In that case you should be able to recover most of your data. Or you may have two disks with just a few bad sectors, which is even less bad.
This all depends on being able to force the array to allow access to the device, so that you can mount the filesystem (in read-only mode) and sift through the remains. Some (many? most?) RAID implementations may just give up if two disks in a RAID-5 array (or one disk in a RAID-0 array) are flagged as bad, in which case you really are screwed, even though your data is still there. From what people have been posting here I would guess that Linux SW RAID will let you force it, though I've never needed to try it myself.
More ram is always a good thing:)
Not quite. In my experience, bad sectors are only remapped by the drive firmware on write. Attempts to read bad sectors will return errors. This makes sense if you think about it; you might be trying to recover data, and the sector might be readable once in a hundred tries, but if you're writing to the sector, then obviously, you don't care about the data that's there already, so it's an opportune time to remap it.
--
I recently attempted to set up a software RAID mirroring scheme on my Linux box (late 2.4 kernel; I forget which and I don't have access at the moment). Because I already had data on one of the disks I wanted to use, this is the technique I used to set it up:
/dev/md0 as an array with the newly created partition and a 'missing' device /dev/md0
- partition new, empty disk to create a partition the size I wanted my mirrored array to be
- set up
- copy the data onto
- repartition the old disk
- add the partition on the old disk to the array
The system then did a rebuild, copying the data from the new partition to the old one as desired. It worked perfectly until I rebooted, at which point it forgot about the second disk.
Any suggestions what I might have done wrong?
i'm curious, i've been looking for a hardware raid solution, what are your thaughts on these cards:
p
http://www.3ware.com/products/serial_ata9000.as
~500 USD for the 8 port one though=/
Save yourself the trouble and don't do it.
Build yourself a bunch of mirror sets, and merge the volumes using LVM/LVM2, and a journalling filesystem.
Trust me, RAID5 corruption SUCKS FUCKING ASS.
At least with RAID 1 mirrors, I can pull half the drives and continue working.
your the only person i'd head of haveing any trouble with 3ware, usualy if a product has troubles like that, 9/12 fails, you'll hear about it.
but with 3ware alls i hear is good things witch leads me to belvie its not 3ware but your drives or MB or something else.
with software raid its stuck in a OS spec format...
from what i hear 3ware has good customer support. SO it seems do your research and find a company you can depend on
Back in 2003, I ran a Software RAID5 setup under Redhat 8.0 which I later scrapped.
The server had four Seagate Barracuda IV 80Gb drives (RAID optimised versions) connected to a Highpoint Rocketraid 404 controller. The system was a dual Athlon SMP configuration on an ASUS A7M266-D SMP motherboard. The system had 512MB of registered DDR ECC memory
When extracting large tarballs, I would often experience CRC errors at random points within the archive.
When I compared the copy of the archive on the RAID5 filesystem with an original copy, I found differences. This really freaked me out as it may have meant that errors were being randomly introduced into newly written files.
The problem disappeared when I manually disconnected one of the drives, placing the array in degraded mode, and then recopied the original archive.
This led me to suspect that the authors of the RAID5 code in the kernel did not get the implementation of the read/reconstruction algorithm quite right or that the code had nolt been tested properly under SMP configurations. I did not have time (or the patience) to investigate this further.
I abandoned RAID5 and switched to a nested software RAID-1+0 configuration. It works, but I had a bad experience reconstructing the array when one the drives gradually started failing.
Fortunately, my workplace has set aside some money to replace this machine with a new Dell server with inbuilt hardware RAID5 and hot-swappable drive enclosures.
My final advice is: if you have the money, don't settle for second best.
One tricky bit is booting. LILO can't find your kernel if you use Software RAID 5, which leaves you in a bit of a chicken and egg scenario (you want to put your kernel on a RAID 5 partition for the same reason as everything else, presumably). What I did was buy a 16MB compact flash card and a $15 CF TO IDE adapter from Here. I then put the kernel on the CF card and used syslinux as the boot loader. Works like a charm.
Also, as others have said, you may be SOL if you're using hardware raid, the controller dies 3 years down the line, and the manufacturer isn't willing or is unable to get you a replacement. eBay can be a life saver in this situation, but having gone through that I don't care to do it again.
I'm using Software RAID 5 with 5 SCSI disks and I've been VERY happy with the performance and reliability.
Try reading other posts in this discussion.
I've scanned through about half the main posts here and haven't seen the following mentioned. Assuming you're talking about a software raid 5, you can't boot it. You need to set up your root partition on a mirrored software raid or no raid. This includes, /, /boot, /etc, /bin, and /sbin. As far as hardware raid goes, dont. Hardware IDE RAID controller drivers are notorious for their instability, so I'd strongly suggest the software raid.
/, and then another for /home, /usr, /var, and /tmp. It would be even better though if you created a seperate partition for each of those. When I'm dealing with an excess amount of disk space, I generally create a 2 gig root partition (though 1 gig should be enough). Now you're probably going to want to put this mirror'd setup on 3 of the disks (1 as a hot-spare), then that partition space across the other 5 disks for whatever else, i'll just say swap, though 10 gigs of swap is completly rediculous. Also a note on swap space, swap is an automatic RAID, so if you create 2 50 meg swap partitions, the kernel will combine them and access them similar to a raid0.
So basically the setup you're going to want is to create at least 2 partitions per disk, one for
Here's another important note. Those cheap drive's from Fry's seem to have a high failure rate. I bought 4 of the 160 Gig maxtor boxed drives from $59 each about 9 months ago from there, and within a month 1 drive failed. A friend of mine had bought a few of them too and he also had one fail within a short time. The other 3 drives for me haven't given me any problems since then though. I had set mine up on my existing workstation as a RAID 5 with an XFS filesystem on dm-crypted partitions which turned out to be a horrible combination. Whenever I had a power failure I would lose all the data due to the way XFS stored its MBR. Switching to ext3 solved this problem. Anyways, even after the disk failure though, the data was fine (didn't even notice the disk failure until about 2 months later). Oh yah, the reason I mentioned this is because if you're getting 8 cheap disks from Fry's, there'll be a good chance 2 or more of them will fail within a short period of time, so I would definitly think about using one or two of them as a hot-spare.
Anyways, for details on how to set up a software raid, check out the linux software raid howto (google it), and it'll give you step by step instructions.
3ware raid controllers kick ass, they are the best on the market especially for Linux
frickin expensive, though... if you need that kind of performance it'd probably be speedier and more cost effective to do a software RAID 0+1
The unofficial
The most important thing when setting up such a server is to have sufficient cooling. I have 7 drives mounted in this server, and apart from the PSU fan I installed one extra fan. I used a midi tower case, with 4 drives installed in the top 3 5,25" slots, and 3 more (with atleast 0.5" space between them) in the 3.5" bays below. I left the top 5.25" slot and the floppy drive slot open for better ventilation (I removed the CD-ROM and floppy drive as they're not needed).
Cooling is very important, without keeping some of the slots open, temperature of the top drives could easily go above 55C, which is the max operating temp of the drives I use. Operating them above this will significantly reduce their life span. With the slots open I can keep them all below 40C.
Temperature is easily monitored with SmartMonTools. This package can also be used to run regular self tests of your drives, which will help detecting drives that are about to fail (happened to me once already and as it was within warranty I got a free replacement). I didn't lose any data because the drive was only about to fail, but hadn't failed yet when the replacement arrived. After replacing the drive, the raid array automatically reconstructed the drive I took out, working just as expected.
I run a 6 drive software raid, and put encryption on top of it, and I STILL get 10-15 MB/sec performance... plenty to feed a 100 mbit HOME network.
Looks like you didn't quite get it installed properly. I'm running software RAID5 with 6 drives on a Linux 2.6.7 kernel (have been for over 6 months, migrated kernels several times now). None of the data is damaged, not even during a rebuild of the array when I had to replace a drive.
My advice is: If you are not sure about sofware raid5 do as I did: Try on vmware (virtual computer software). You'll be able to see if your system runs well with little of ram, lot of ram, drive recovery...
You'll be able to test anything and restore images if somethin goes wrong.
Some hardware RAID controllers can do this too - I run a server with an HP NetRAID 3i (or something) controller with five HDD's. Two are split in one RAID1 partition and one RAID0 partition, the rest are a RAID5 array.
If you need a really safe solution, stick with software RAID:
1) Linux SW-RAID is very well tested. It's very unlikely that a software bug will trash your data. HW-RAID Controllers have a firmware that can be buggy. It does not happen often, but if it does you're hosed (dataloss, no solution: closed source).
2) Consider the situation where your RAID-Controller fails. If it's an older model you won't get a replacement easily. No one guarantees that other controllers are compatible with the data format your defective controller wrote on your disks. So maybe all data is lost if you don't get an exact HW replacement.
To my mind HW-RAID has the advantage to be faster (but only with expensive controllers) and does have a battery buffered RAM which prevents dataloss in case of a power failure.
Maybe you forgot to change the partition type of the 'old partition' to RAID Autodetect? Or maybe you forgot to change the root= kernel parameter in lilo.conf or grub.conf?
HAND.
We just use Nagios. We had a drive failure recently, and we got paged immediately letting us know the drive had failed. It worked exactly as we expected.
Hardware, software, and blinking lights!
just joking, but being an expert in one area does not automatically make you an expert in another area. (Not that his advice about trusting brand names and doing you own testing wasn't good, I'm just saying...)
HAND.
About the only time now that a hardware RAID card makes sense is when the number of drives and total thoroughput would saturate the PCI bus. The hardware RAID card would be a lot less likely to encounter this situation, since it uses only internal busses for drive communication.
That said, hardware RAID cards generally have only a very limited cache RAM onboard, and a properly designed software RAID that integrates with the block cache could potentially make big wins in performance in systems with lots of RAM.
Hardware, software, and blinking lights!
http://ohlssonvox.8k.com/fdd_raid.htm
are out there that are decent, and will work as simple IDE controllers?
I've had difficulty finding anything that has more than two controllers that doesn't force you to use the half-baked raid.
All I'd like is a good ata-133 controller (that's what I have, but others may want to know about sata) that actually has at least 4 IDE chanells that show up as normal IDE to the Linux kernel, rather than forcing me to use their weird raid.
It might be off topic, but it seems you could buy SATA drives and say a 3ware card for just a little bit more and get hardware raid with hot swap (and yes there are linux drivers).
I was crazy back when being crazy really meant something. (Charles Manson)
Well, FYI then, 3ware is not software raid, not at all, nothing like winmodem. The one I've got in use is a 64-bit pci card that has quite nice performance! It's got excellent cross platform support (their phone techs will actually answer questions about linux and freebsd for instance!)
also it has some quite nice management tools, including a very nice www based monitoring daemon.
Why don't you like 3ware? I've admittedly only put two systems into production with 3ware hardware, but this was after extensive reading and searching, in which 3ware received great reviews.
You are absolutely right. I hadn't considered that. An XOR of the parity block would provide the same results as XORing all current data blocks against the new one.
I've been doing RAID in various ways for about a decade. The hardware vs. software landscape is very different than it was a few years ago:
...) happen to work nicely for RAID calculations as well. This is why you'll see some kernels run through trials to see which of several different parity algorithms is faster on the current CPU. This combined with other tuning done in the RAID code has reduced CPU overhead to who-cares levels.
(1) Hardware RAID -- this used to be the obvious choice; why load up the main CPU doing XOR loops? However, we are now seeing a lot of 'raid' cards that are in effect little more than multi-interface cards with some code in a BIOS rom, forcing the CPU to do all the work anyways. Putting them aside -- and those will be most of the low-end cards -- the higher-end hardware RAID cards just don't provide the bang for the buck that they used to. This is because...
(2) Software RAID has improved dramatically, especially in Linux (and I presume but cannot confirm, in the BSD's). The processor instructions that were added to the CPUs to accelerate graphics (3Dnow, SSE,
The current software RAID implementation on Linux does hot sparing and rebuilds quite nicely. I haven't had a drive fail on a software RAID box, but I've simulated it, and the recovery was exactly according to spec. The only standard gotcha is booting when the boot drive has failed -- software RAID is little help there.
The ultimate solution for many systems (imho!) is LVM on top of RAID -- providing flexible partitioning, the ability to migrate to new devices, and snapshot backups in addition to the peace-of-mind of RAID-- and all for less than 3% CPU utilization.
Been there with a Mylex DAC 960 and believe me, you don't wanna be there. In a server with redundant CPU's, redundant PSU and redundant disks trust the only non-redndant thing, the RAID card, to go pop.
The Mylex stores its RAID stuff on the card itself (not on the disks, as it should) so if the card is screwed, and your people never wrote down the friggin config, you are well, screwed. On a real RADI card the array config is stored in the first few sectors on the disk so when the card goes, you stick the disks on to the new card and you are just fine.
Fortunately we got another old Mylex on EBay and switched the NVRAM chips and it worked....one of the worst days of my life. And I was never happy to see NT 4.0 boot until that day.
Oh well, working for a non-profit tends to be an adventure. My servers stands on bricks to protect them from the occasional flooding.
The dangers of excessive individualism are nothing compared to the oppressiveness of excessive collectivism
Or do both.
only use six drives live in the raid and keep on as a hot swap and one as a spare in a box. I would suggest that if you want maximum reliability I would go for a hardware raid card.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
You are trying to work around bad sectors causing total disk failures. MD should create a 'bad sector table' for each drive, and when block 0 is referenced, it will write to block 9999999 instead.
And vice versa for reading. Then you don't need to do the hack you are describing.
Kashif
Whats your backup plan for the raid card?
Five years after the hardware card is discontinued I'll still be able to run my software raid...
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Here is an example:
One disk read and two writes... exactly as grandparent poster said. No need to read any of the original data blocks. Only the original parity block. Even then only if it isn't still cached in RAM which is quite possible.
I got my sx4000 for under 100 and that was over 2 years ago.
Food not Bombs is a nice platitude but it breaks down when you notice that the Bombees are usually well fed
Many have posted about the loss of more than one drive being v. bad. So my question is, does SW RAID support RAID 6 or 7? If not, can you do 10 or 50?
Realize that this thread has come and gone, but that makes this comment approriate - RAID systems toward end of life. Dell gives warranties for effectively five years (initial three year period + 2 one year renewals), this means that their equipment actuaries see the writing on the wall rather than unquestioningly accept the 2 million hour numbers spouted by the disk drive manufacturer. Have a replacement strategy in place as you approach the end, factor in how long it will take you to transfer the data that you've accumulated during the years of successfully operation. And as an aside, my biggest problems resulted from the ON-OFF cycle. Implement a source of constantly supplied power as well as some system to actually be able to "hot swap" the drives and the cooling fans, cycling the drives to replace a failed or failing drive may push additional drives to failure.
NFS doesn't serve partitions, it serves files.
My other first post is car post.
Absolutely true, in my experience.
The fans are not balanced correctly even when they are new! We've bought 50 of one model of fan recently and all the ones we've tried, 25 maybe, are unbalanced.
A fan mounted near or on a hard drive MUST be vibration insulated. We use nylon plastic straps.