Slashdot Mirror
RAID Controller Shoot-Out
mikemuch writes "ExtremeTech has a comparison with benchmarks of three RAID controllers from Adaptec, LSI Logic, and Promise, and along the way gives you a little refresher course on RAID in general and why you want to use it: faster throughput, longer uptime, and improved data security. Motherboard RAID controllers do well when there's 'very little or no load on the CPU, I/O bus, and memory bandwidth. But with heavy traffic and processor loads, the limitations of the shared bus and the benefits of intelligent RAID's integrated IOP and memory cache have a more significant impact.'"
On-board RAID is good enough for most everyone.
"Some days you just can't get rid of a bomb."
Dumbass.
Print version here
Annoying "next page" articles...
Anyone know of a hardware raid card with a external SATA port that supports the port multiplier feature? I have only been able to find software raid solutions.
A closed mouth gathers no foot.
It's even worse. Now you can buy even a server with these fakeraids.
They put in an add-on PCI card as RAID "controller" and add a "hot-plug" cage for disks to fool you. They also claim it is linux compatible.Imagine the joy when you find out that you can run only one ancient version of linux kernel to even access the disks.Oh and the RAID capability is supported only in Windows...
[rant end]There's a stigma associated with host-based controllers that trying to rebuild an array with them is tantamount to masochism. I think it comes from the fact that an intelligent controller can rebuild an array through BIOS-only intervention.
Anyone care to shed some light on how rebuilding arrays compares when using intelligent vs host-based controllers?
*blinking cursor*
Linux, BSD, or Solaris?
How about calling it the Windows RAID controller shoot out?
ExtremeTech should just change it's name to Mainstream tech and get it over with.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
According to the OpenBSD i386 supported hardware website, out of the cards reviewed, only Adaptec and LSI cards are compatible with OpenBSD.
However, Adaptec has refused to provide documentation so that the OpenBSD project may improve the drivers.
"Note: In the past year Adaptec has lied to us repeatedly about forthcoming documentation so that RAID support for these (rather buggy) raid controllers could be stabilized, improved, and managed. As a result, we do not recommend the Adaptec cards for use."
Other *nix variants might support the Adaptec and Promise cards a little more, but the hardware fully supported by OpenBSD is generally well-supported across all *nix variants.
Out of the cards reviewed, only the one from LSI is worth buying. Adaptec may have a little support, but it's not a good idea to purchase any RAID cards from them until they start providing better documentation.
LSI has a much newer (and presumably) faster SATA/300 card now. If you were buying a new RAID card, that's what you would buy. Why did they use the old generation?
Where's Areca? They're the performance leader in this market, and their pricing is now in line with competition.
Where's 3Ware?
What about other host-based RAID solutions? Broadcom? Marvell?
Don't even get me started about what they tested. This is just not a serious RAID review. I strongly urge folks who are interested in this subject to Google for better.
I've used some RAID controllers myself, and I have friends with a lot of experience with them. A key factor in what makes a good RAID controller is not throughput. It's long-term reliability. How long can you hammer your RAID array before you get unrecoverable corruption? A RAID array is supposed to prevent that, but if you have some weird bug in your RAID controller, or it's susceptible to EM interference from surrounding components, you will get data corruption. And I don't mean for reads; I mean that the data gets corrupted on the way from memory to the disk (at least that's our theory), where no RAID controller can protect you.
Of ATA controllers, our experience shows that 3ware controllers are the least unreliable. That is, they generally suck, because they have demonstrated performance problems and other weird failures that 3ware couldn't help us resolve, but they suffer from the least data corruption.
For whatever reason, the on-board controllers are the worst. They seem nice and perform well enough, but they have the highest rate of data corruption.
It may or may not surprise you that software RAID is relatively reliable. With a RAID1, you'd think you're twice as likely to corrupt data on writes, because you have to send the same data twice to two different drives. Sure, having them both bad is unlikely, but at a later time, how do you know which copy of a given sector is correct? But we think that removing an unreliable hardware RAID controller from the data path and just having the relatively simple ATA controller in the way reduces chances of a problem. Just a guess.
If you want truly reliable hardware RAID, you need to spend your life savings on an industrial-strength SCSI RAID controller.
The moral of the story is that there's really no such thing as 100% reliable data storage. If you want speed and don't care about reliability, RAID0 is for you. Other RAID levels add redundancy, which is nice in theory, but add hardware complexity that offsets some of the advantage. For my critical data, I store to CD and DVD ROM. And I make multiple copies of those, because those aren't all that reliable either.
This isn't a shootout, it an advertisement for the cards. They are at different price levels, and ExtreemTech is just showing the difference in performance you get for spending more money. Wow!
If they tested multiple series of the Adapatec, LSI Logic, and some 3ware cards, I would be more impressed, but this just seems like an all out advertisement to me.
Its not what it is, its something else.
Exactly.
The article may be an advertisement disguised as an article. Possibly they don't want to benchmark 3Ware because it would win. Judging from the article, possibly this is an Adaptec ad.
--
U.S. Taxpayer Karma: If you contribute money to kill people, expect your own quality of life to diminish.
I've got a coraid array that can saturate the host PCI bus running on ATA-over-Ethernet technology, which is faster & simpler than SCSI-over-IP. Performance comparable to my giant expensive fiber channel SAN at a tiny fraction of the cost.
These guys are behind the technology: http://www.coraid.com/
If you don't like Open Source, you won't like it yet. Wait a few years and there will be a version you'll like, the economics of it are compelling. But right now you need to be able to write your own init scripts.
Why buy a $600 RAID controller when I can get the same performance from a $60 gigabit ethernet card?
Do any of those RAID cards support deleting an unwanted logical drive without deleting all of the logical drives? That's been a royal pain for me with both the Adaptec raid controllers and LSI's Megaraid controllers. What use is it if you can't remove old smaller disks and add new larger disks without rebuilding the entire system?
Also, have they gotten any better about continuing to run during single-disk failures? With the megaraid cards I've used its about 50/50 whether the raid subsystem crashes during a single-disk failure under Linux. Sure I can reboot and it comes back up but half the point of a raid card is to keep running despite the disk failure.
So far the only controller I've found that's worth a damn is HP/Compaq's Smartarray. Sadly those cost boku bucks unless you get them built in to HP/Compaq servers.
The old Mylex DAC960-based cards used to kick ass but then the Linux driver author died in a helicopter crash, LSI bought Mylex and when the kernel changed from 2.2 to 2.4, the dac960 driver malfunctioned badly.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Ok, I realize it's a bad metaphor because you actually can throw a motherboard quite a distance. But here's another example of where things can go horribly wrong: How do they handle error conditions? On my desktop system, I'm running RAID-0 (with WD Raptor drives) for speed. Yes, I know what I'm doing (famous last words). No, I don't store any important data on my desktop (it's on a RAID-5 array on a server). Originally, I was using the Silicon Image 3114R on-board RAID controller included on my Asus A8N-SLI "Premium" motherboard. Eventually one of the drives died. The SI3114R responded to the problem by freezing and becoming unresponsive when a disk error occurred. Under DOS, Linux, or WinXP - the problem is not OS specific. The rest of the system works fine, but once it hits an error the SI3114R just stops working and returns nothing but errors to the OS. Now, since Asus doesn't update the SI3114R BIOS in their mobo BIOS updates (and I'm too lazy to hack my own), I don't know whether it's bad silicon, bad BIOS, or a bad design (my guess would be the latter). Accessing the drive's S.M.A.R.T. data indicates that the warning numbers were screamingly bad and probably were for some time.
So apparently the SI3114R doesn't monitor S.M.A.R.T. data, and it's error-handling capabilities fall somewhere between "shitty" and "non-existant". No big deal for me; I was only inconvenienced by having to re-install operating systems and applications.
The moral of this long-winded story is that you generally get what you pay for. This isn't the first bad experience I've had with on-board RAID controllers. If your data is important, then spend the appropriate money (think in terms of data replacement cost), do the appropriate research, and invest in a RAID setup that's right for your situation. If your protected data consists of anything more important than your Oblivion saved games, your mobo's RAID controller (or the $39 Fry's special) is probably the wrong choice.
And if anyone cares to know, I'm now using the NVRAID on the mobo (we'll eventually see if it handles failures more gracefully), and I use an Areca ARC-1110 on my server. I can attest that the Areca card does handle failures extremely well, albeit noisily.
Help save the critically endangered Blue Iguana
I was expecting some hardware to actually be shot
I second the Areca recommendation. Their cards are very capable of detecting a failed disk, taking it offline, mailing the operator, and sounding the buzzer, all without skipping a beat as far as the host operating system can tell. And their RAID engine is bleeding fast, too. I just wish the kernel folks would try harder to get their driver into the mainline. Areca is the rare example of a manufacturer who undertook the cost to write their own Linux driver and release it under the GPL, and the kernel maintainers have spent more than a year whining and bitching about how the code doesn't fit in their 80-column terminals.
A review of SATA RAID controllers that have open source Linux drivers would be very useful to me.
The ExtremeTech article was a complete waste of time.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Strong words, but pretty accurate.
I manage a few small data centers and we depend on RAID 1 and RAID 5 (and redundant servers) to keep our business running. Down-time is expensive, but rebuilding a machine from scratch is expensive too. So we don't use any software RAID or software-assisted on-board "controllers" - it's hard to call them controllers. RAID Filters? RAID bridges? RAID-like adapters? I love real RAID controllers. Everybody I interview has to explain RAID and something about how it works. If someone tells me they use RAID-0 for performance, they'll need a really impressive story about why that's not plain stupid.
People like to say "RAID-0 is great for gamers and testers - it's cheap and fast," but think about the scenarios where this applies:
1. Cheap!!
You have one computer, you use it (mostly) for gaming. Your critical data exists elsewhere (GMail? cd-rw's?). A second, identical hard drive runs you about $100 US. You have on-board RAID, which relies on your processor for all the heavy lifting, but it's paid for already. When you're loading your multi-gigabyte game, the two spindles go pretty fast together and your processor, while not idle, is mostly waiting for the disks. So you can spare the CPU cycles the RAID "controller" uses, great. But this computer is twice as likely to crash and burn as a single-HDD machine, so you can't rely on it to hold any persistent data (personal finance, schoolwork, download-only software, etc, etc). You just diminished the value of your computer, and you're trying to save money.
2. Fast, but just for games!!
You have a dedicated computer, with two-disk RAID-0, and it's your gaming rig. It's optimized for speed. The on-board RAID controller isn't fast enough when you've got to load textures from the hard drive and track hundreds of independant 3d objects in (insert blow-up-stuff game title here). So you have a hardware-based controller, with its own BIOS and OS-independant management tools. You could buy three RAID cards like this for the cost of a "mainstream" video card, the cost isn't a big deal - you can afford to spend $2k on an unstable drag-racer of a gaming rig.
Someone in the world thinks your time isn't worthless, but you're willing to burn a whole day, now and then, to rebuild your computer? Why not spend another ~$200, get two more hard drives (I know you've got the fans/power supply/drive bays/liquid-nitrogen-cooling to handle it), and run RAID 1+0? You get the same speed boost, but now you don't need ***a whole other computer*** to store your "real" data.
Don't get me wrong, big, resource-intensive games are great stuff. And faster loading is great too. But how many computers do you really want to own and maintain? I know, for many people, having six machines to manage them is cooler than one, but my time is more valuable than that (I know, we're mostly talking about Windows, and you can maintain thousands of *nix boxes with perl-scripts and ssh, thanks, got it). I'd rather just get a faster drive and controller - SATA2 is pretty damn fast for a non-SCSI system.
3. Fast for testing!! No real data!!
You're not testing performance, right? Because you're not depending on RAID-0 to handle real data when the test is complete... If you have a grid computing farm, and the time to rebuild a single box is only a few minutes, and you can stand some data loss, this might be a good solution. Maybe your application can withstand individual machine failures and the raw speed is more important... In this kind of setup, you're much more likely to have network bottlenecks than disk bottlenecks -- your processor-intensive tasks are passed around the network and not too much data has to be stored on disks. But if this is the case, and you can afford a grid/server farm, I doubt you're going to solve a disk-IO bottleneck with RAID 0 instead of external storage (SAN, big SCSI array, big SATA/SAS array, etc).
There probably are situations where the additional data-loss risk is offset by performance gains, but usual pro-RAID-0 scenarios in this debate just don't make sense.
PS - mod parent up!
I'll second the advice on Adaptec. I have the ATA Raid 5 cards. The software they provide for Windows is ok. The BIOS software is ok. The Linux software? 2.4 kernel software was a nightmare for anything other than Red Hat, SCO, and possibly one more distro. Everything else was a fucking nightmare of patching the kernel and rebuilding, then hoping it worked. Debian, it didn't. As for 2.6, forget it. No support for 2.6 yet, don't think there's any planned unless you are buying a current SATA card. The ATA cards with 2.6 kernel? There's no way of monitoring the array while the system is up. The only way to check the status of the array is to shutdown and boot up again, then get into the RAID BIOS. Last time I took one system down just to check the array, I found one dead drive, with the hot spare having kicked in to replace the dead drive. Since its a production system, its now running raid 5 without a hot spare. With the luck we've had on drives, between deathstars, bad batches from more than one manufacturer, etc., the production system with the missing hot spare and no way to monitor it is keeping me tense until I can schedule some downtime to replace the hot spare. The way it is now, I lose two drives over 100-150 days (has happened before), then I have a dead production system until it can be restored from backups.
I saw the situation with OpenBSD recommending against Adaptec, and I ignored it, crossing my fingers. Big mistake. OpenBSD is correct. Only stick with companies and hardware which OpenBSD and others in the community can get documentation on. Documentation is what OpenBSD is looking for, so they can write their own drivers. They aren't looking for the source code to the firmware, just the documentation to the registers and other internals, so no company is giving away trade secrets, precious ip, or competitive info.
Raid cards are used for a very specific reason. The safety of your data. Reliability. Especially when you consider how much the RAID hardware cards cost. Listen to OpenBSD. Stay far away from Adaptec. Far, far away from Adaptec. They are flipping the community the bird. And I don't mean penguin.
Help save the critically endangered Blue Iguana
I call 100% Shenanigans. Ok, you said "most ATA RAID..." so I call 99% Shenanigans.
/. onslaught, I'd post links to the project pictures off this very same box. :)
I built a sizeable (by 2003 standards) 4-drive RAID5 array using a cheap P3 motherboard and an Adaptec 2400A RAID controller on Feb. 16, 2003. It's still running swimmingly, serving up my media and user directories, and has never had any glitches -- not even a failed disk. Data corruption? None. If you have data corruption on a RAID5 controller, you need to seriously reconsider your parts list. (Not saying the ATA-2400A is the way to go - it's since been discontinued, and I'm not sure I'd go with the SATA version of Adaptec's RAID5 next time.)
I built it after getting tired of hardware failures -- I was bitten by both the IBM Deathstar drives and Abit's bad capacitors. It's not intended to be 100% available, or 100% infallible. I have a second 300GB single drive that does nightly file copies of critical data off the array, and I do regular "other" backups to DVD, CD-R, etc. RAID5 is not the answer to all your storage problems, but it's served me better than any other system I ever built before or after (so far).
Uptime on the array is over 99.85% over a 3 year period, and that's including 2 days worth of ice-storm induced power outage (January 2004), another half-day of hurricane-inflicted power outage (Isabel, September 2003), and my son disovering the power button more than a few times (grrrr...). Even with all that, it's never flinched once.
It's been my web server, mail server, FTP/TFTP server, print server, Shoutcast server, and audio/video media server (XBMC) for years. Best computer project I ever embarked on, and if I thought my 60K/sec uplink would stand up to the
Is it as powerful as a full-blown SCSI array? Definitely not, but it's much quieter, much cheaper (total cost: $320 for the adapter, which was expensive even then; $95 per 80GB WD800JB drive), perfectly adequate (I/O at ~56MB/sec, which is only ~5MB/sec slower than a single SCSI Ultra320 15k RPM drive). And best of all, it's 100% Ghetto Brand quality, so you know it's legit...
And on a quick check, they even wrote a FreeBSD driver under 3-clause BSD license. No signs of a blob. Got included into the FreeBSD tree around 5.4-Release.
RAID in PCs has been a hot topic and I suspect a lot of marketing has gone into products which the vast majority of people have no real use for (having a RAID setup being bragging rights).
If you really do *need* a RAID setup, it seems stupid to ignore the SCSI angle on things, because SCSI RAID controllers are much more mature in features/performance/reliability and obviously aimed at a market which is less tolerant of cheap'n'nasty.
I know that SCSI is a lot more cost and, compared to SATA, not all that much more performance for the extra money. However....
More than a year ago I picked up 2x Intel SRCU32 (these are two-channel ultra160 scsi) from ebay for £50 each. I've found these to be very good cards. In addition to the usual BIOS interface they also have software consoles for both Windows AND linux (the latter written by intel), are supported natively in linux (2.4 - driver is called "GDT").
The software console means you can configure your logical drives without your PC being stuck at the BIOS/POST stage. From what I recall a sync of a 3x72GB in R-5 mode took about 4 hours.
Another nice thing about this controller is that while being 64-bit 66MHz PCI it is compatible with (and I am using it in) 32-bit 33MHz PCI. However a downside of such older controllers is memory. It requires unbuffered ECC NON-registered PC133 (also found on ebay - 4x128MB for around £20) which can be hard to find.
I'd also like to add that while I was getting these on buy-it-now, loads of other people were bidding £100+ on Adaptec 2100S auctions, which is a 32-bit 33MHz only, single-channel card!
Summary - please think before you go buying this sort of thing, and really consider all the alternatives.
Which means that on Friday at 4PM when your RAID controller smokes, you either have an on-site spare or you start panicing. Maybe you'll be lucky to find someone who has one in stock and can ship for Saturday delivery.
If you can get onsite service contracts in your area this is a very good selling point for them. If you're > 100 miles outside of a "major" metro area, good luck.
I've never had an SMP server slow down with software RAID-1 mirrors that I could notice.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
No, I'm not kidding. Keep in mind I run on empirical, not theoretical. I'm not sure why you think it's reasonable to compare "a well designed SCSI system" which you admit requires multiple cards and channels with a single-port AoE system.
I recently replaced a ~2 TB duplexed u320 SCSI RAID array with a ~8 TB AoE array. Same hosts - same OS - no changes except from SCSI RAID to AoE RAID. My bus is PCI-133 and I have two GB ethernet ports on the motherboard, four more on two Intel PCI cards (not all of those are used for Aoe, obviously).
I have a data store that contains well over 13 million files that I need to back up regularly. Base backups that took five days now complete in 2 days, which means I can get it in a weekend, and that rsync --link-dest backups can be done overnight every night during the week.
I haven't rigorously analyzed the situation, and I never will, because I've solved my problem cheaply and effectively. So, to you this is anecdotal - but to me, it is rock-solid and reproducible, and I really don't care about theoretical numbers published by companies that want to sell me SCSI.
I've used Mylex, HP (mostly LSI), Tekram, Adaptec, DCA, etc. etc. probably every SCSI controller and subsystem out there including DEC (both real SCSI and DSSI) and Sun and Apple. AoE is cheaper and gives me empirically better performance for the buck. It's not ready for non-professionals yet, perhaps, but that's not a big deal to me since it's Open Source and the coders are reasonably easy to deal with.