Slashdot Mirror
Compelling Alternatives to RAID Setups?
jabbadabbadoo asks: "Our software shop has about 30 Linux servers and 15 NT servers running enterprise applications for our customers. Since we have service level agreements with most of them, uptime is crucial. One of the things we've done is to use RAID setups extensively, using products from well renowned disk- and controller vendors. However, we have discovered the paradox that introducing RAID controllers actually reduces overall uptime! Not only does more 'steel' increase the probability of failure, but what fails first is usually the RAID controllers. What is your experience? Have we been having bad luck?"
"A related problem, especially on Linux, is that setting up RAIDs is actually a quite costly process. There seems to be endless problems with library versions, and upgrading existing servers simply takes too many hours. To keep the customers happy, we routinely have to create a 'shadow' server while upgrading which in turn means we, at some point, have to synchronize data to the new server, which in turns means a bit of a downtime. Ouch. Does anyone have a good solution to these problems? Of course, cost is a major issue, but so is uptime (which also means cost if we don't provide the uptime dictated in the SLA). What setup gives the best cost/uptime ratio? Thank for any thoughts!"
I remember swapping quite a few Compaq RAID controllers in my day. They wouldn't outright fail, but get in a "compromised" mode, and you usually had enough time to schedule downtime to swap them out. This was much better than messing with software mirror or raid settings, because it's transparent to the OS - the OS just sees a single large disk.
We run about twenty systems with RAID storage devices (about half of them fibre channel). I've only had one system go down due to the storage device, ever. I think the power supply on our FC bays failed once or twice, but they have backup PSUs, so it wasn't a problem (hot swap, even!).
It's always a long day... 86400 doesn't fit into a short.
I'll agree that setting it up is a nightmare. I'm currently helping test two 4TB arrays for use on a Linux box (16 SATA drives presented as a single SCSI device). Benchmarks under linux are slower than under windows. It's a mess figuring out why. Meanwhile, vendors (who I will not name ship crappy software, and take months to act on bug reports.
As for transitioning servers, I've been there too. And yes, copying a terabyte of disk in single is a very long process. It'd have taken several days, which is of course unacceptable. This is where the magic of rsync comes in handy. Copy the data over several days in advance, sync it just before the scheduled downtime, and you'll have a fairly short downtime.
You don't list what brand controllers you are using, but your problems are not typical in my experience. We are a 100% Compaq shop and use their SmartArray controllers with Novell Netware and Debian Linux. We've never had a controller failure and have only lost about 3 drives over the last six years or so.
I'm a firm believer that you get what you pay for with enterprise-class servers. You shouldn't expect Tier-1 reliability from servers that are built with commodity hardware. There is a reason that Compaq/Dell/IBM servers are more expensive.
We also haven't had any issues installing other than the default Debian boot disks not supporting the SmartArray controller. A custom set of disks took care of that though.
Jason
"FORMAT C:" - Kills bugs dead!
In the past two years, none of the "downtime" that I've experenced has been attributed to the disk array or controller.
The biggies have been: power outage that exceeded the capacity of the UPS (3 hours), planned upgrades and an anonymous gremlin who bumped the reset button - since detached.
This is a boring sig
At least, about any except the most expensive RAID controllers out there.
In my experience, which does not include RAID controllers that cost more than the whole computer, software RAID is the most reliable RAID; however, even it is questionable if it's better than just buying good drives and having a PCI IDE controller card around in case the Mother Board controller goes out.
XSan can 'hide' the complexity of RAID, as well as providing management tools and 'intelligent' cascading failure... but that's just from reading the specs, not from actual experience. I hear XSan is based on CVFS? I should look at that too.
GPL Deconstructed
This is on a lower level than the RAID you are using, but we are having major problems with 10 Promise Technology TX2000 mirroring RAID controllers that we bought. The mirrors go critical for no detectable reason. Promise Technology technical support is unable to find the problem, and the company is unwilling to escalate the issue. The Promise Technology technicians escalate the issue, but 2nd level technical support never calls back.
Promise mirroring controllers on ECS (EliteGroup) L7VTA v 1.0 motherboards have the same problem. When we call ECS tech support, there is a recorded message saying they are busy and to call back later.
We've been supplying computers with Promise mirroring RAID controllers since the company began doing business, and we've had very few problems until now.
Possibly the problems are associated with newer, faster motherboards, or with AMD VIA chipset motherboards. We've never had problems with RAID controllers on Intel chipset motherboards.
Another possibility is that the RAID controllers are incompatible with DVD burner drivers that are installed with Roxio or Nero DVD burning software.
It's hard for me to believe that RAID causes more downtime than single drive setups, unless you have a really bad raid system and a really good backup system.
:-) Soon to go away... :-(
The only time RAID should ever be down, is during initial setup. Thereafter you should replace bad drives while it's running and you should never have cause to shut it down due to a RAID issue.
If you are experiencing RAID hardware problems then take a good look into these areas:
RAID Hardware --> Are you using cheap stuff? It honestly isn't worth it. Perhaps you're just discovering the 'real' value of 'cheap' hardware.
RAID Software --> If you're using unsupported drivers (ie, vendor doesn't supply or support them) then ditch the hardware and get hardware with supported drivers - make sure they support them on your configuration. You've already proven that you can't support them yourself.
System Hardware --> If the system is generally cheap (cheap power, bad airflow, cheap components, etc) then you simply can't expect the RAID card to work 24/7.
Server Room --> Make certian your server room can handle the power and ventilation needs of the servers. This should go without saying, but all too often it is the problem.
The reason people go with cheap components is the lower initial cost. They only work for a few thousand hours of heavy operation. You must get server rated components if you want them to operate for more than a year or two. There really is a difference.
Lastly, I use 20+ Promise FastTrack ATA RAID cards in 20+ Novell networks. I use cheap components, and they work in harsh conditions. They are not set up for hot-swap, as that's not a need in this situation. I have to replace the cheap hardware every 2-4 years, powersupplies every year, hard drives every 2-3 years. The only time the RAID cards have gone bad is when a power supply failure (usually due to a power outage/surge/brownout) fries the motherboard and usually most of the components in the case.
I have never had a failure where both HDs completely failed simultaneously, though usually when the rest of the computer goes I replace the whole thing and get the data off one of the old hard drives. This is not an advertisement for Promise. They simply are the only one's with supported Novell 3.12 drivers.
I'd be surprised if you've covered all these bases and are still having problems.
-Adam
There is an old saw in the aviation industry: "A twin engine aircraft will have twice as many engine problems as a single engine aircraft."
However, which would you rather be in, a twin engine aircraft that just lost one engine, or a single engine aircraft that just lost an engine?
Yes, RAID cards die - I've been shocked at how often that happens. And 5 disk RAID will have more failures than a 4 disk JBOD (just a bunch of disks) array.
But the question is, are you seeing a reduction in UPTIME, or just in mean time to failure? Maybe the RAID system throws an error once a month and the JBOD system throws an error every two months, but if you can recover in 5 minutes by swaping cards or drives rather than 5 hours for restoring the JBOD from backup, you are better off.
Perhaps what you might look at would be using RAID software on the server's processor, coupled with Firewire drive bays, disks, and multiple Firewire cards. If you have a card die, move the disks to another card until you can schedule downtime. A disk dies, hot-swap and rebuild in background.
www.eFax.com are spammers
The answer is SysAdmin 101 stuff.
..
1. Buy quality hardware.
IDE RAID for critical servers is a bad idea.
In my experience, RAID hardware tends to be very picky and suffers from subtle and often bizarre hardware conflicts. In general, using a RAID solution that is packaged with the hardware is the best idea.
If you cannot afford good RAID hardware, stick to conventional JBOD configurations.
2. Configuration
Design your the configuration of your systems around consistency first, performance second.
You need to document your procedures for building servers, allocating storage, etc. Create scripts whenever possible.
If you are not confident that you could not talk a marginally qualified technician through a server rebuild over the phone, your docs aren't good enough. If you don't have the time to write docs, make the time or work late.
3. Backups
You need documented, tested backup AND restore procedures. All of your oncall staff need to be able to restore a server.
With 50 servers, disk controller or disk failures should not be a common event. We work with approximately 400 datacenter and 200 field servers (varying in age from 1-9 years), and replaced 3 controllers and 19 disks last year.
Look for electrical issues, you may have crappy electrical service.
Conformity is the jailer of freedom and enemy of growth. -JFK
What Promise FastTrak RAID controllers are you using? As I said in the comment just above yours, we are having major problems with FastTrak Tx2000 mirroring controllers.
What RAID controllers would you recommend?
What hardware is "quality"?
I spent the past week and a half trying to set up a 4x160 SATA Raid-5. It was a huge excercise in frustration because every time I'd try to build a volume, my machine would promptly freeze after a few percent. I changed out IDE emulation for SCSI emulation in kernel... same thing... I changed SATA controllers, same thing. I changed SATA cables, same thing. I changed power supplies, same thing. I added 4 80 mm case fans, same thing. In the end, it turned out that the culprit was raidtools. Nobody had ever bothered to post that raid-5 + raidtools + kernel 2.6 locks up a computer. I changed to mdadm, and I had a working array 50 minutes later.
If your bandwidth requirements are not too high you may be able to use a distributed file system on many redundant (cheap IDE & G ethernet) nodes and allow for replacements. Your uptime should be constant, given enough UPS and redundancy of nodes.
--
"we live in a post-ideological world..." - Billy Bragg.
They're systems are probably 80% auctioned desktops and such from busted dot coms.. and I suspect that many of them are not RAID at all. I have yet to hear of a redundant raid controller either. Your best bet is just replication of data on you backend servers and using something in the nature of a Cisco CSS or some other services balancer device to handle keeping alive servers available while redirecting away from dead servers.
You can still do RAID with this setup but you'd have the added security of 2 or more systems making up your entire functional system so if one is down the other can continue normally. Then it's trivial to repair the dead machine and bring it back into the cluster.
All things being equal in terms of build quality, the thing most likely to fail is the thing with the most moving bits.
You say you've had more raid controller failures than disk failures. Did any of the raid controller failures require a restore from backup? A non-redundant-disk failure would have.
Add up the total time you were down due to raid controller failures and the total time you would have been down for disk failures if you didn't have raid. That's a better measure than instances of failure.
PC Chips motherboards? How so?
What do you recommend?
You can't slap a buzzword like RAID onto whatever you were doing before and expect results. Reliable systems have to be carefully engineered correctly.
From the sound of your posting, I'm assuming when you say you're using RAID, you mean internal RAID cards inside a server with internal disks attached, and relatively small amounts of it. In these types of scenarios, the highest performing, most reliable, and most cost effective option is to put two seperate scsi controllers in your boxes, buy twice as much storage as you need, and mirror between the controllers using the OS's software mirroring capabilities. You are now indepedant of controller failure, the controllers themselves are less likely to "fail" (which doesn't always mean hardware frying) than a complex raid controller by their simpler nature, and you're getting the performance benefit of full mirroring instead of that clunky raid5 business. If you have enough storage to warrant four or more internal disks of some size, use mirror+striping. Always mirror at the lowest level, and then stripe on top of that (in a 4 disk design actually it doesn't matter which way you layer them, but in 6+ disk designs it gives higher data availability in the unlikely event of multiple disk failures). Or in other words - raid5 and hardware cards = bad, mirroring/striping + software raid = good.
Your goal is not to be buzzword compliant by slapping in a raid controller, your goal is to carefully analyze your systems, your options, your requirements, and your budget, and eliminate single points of failure everywhere that it's feasible and desirable to do so, starting with the lowest MTBF items in the system and working your way up. There are no magic bullet answers of course - change the situation and the "right" answer can change dramatically.
11*43+456^2
The performance we get with Promise controllers (when they work) has been satisfactory. The application is a cash register; the computer is always faster than the operator. We only need a mirror copy of our data.
3Ware told me they cannot boot from one drive, after one fails. A 3ware formatted drive cannot boot from the IDE controller on the motherboard. Promise can do both. We need features, not performance, in this case.
Do you have a link to an Adaptec IDE mirroring RAID controller you would recommend?
The Adaptec ATA RAID 1200A is about $55, about $30 less than the price of the Promise controllers. We have no experience with them; I found the info by Googling and Froogling.
Question: Are Adaptec ATA RAID 1200A cards the same as HighPoint RocketRaid 133 cards? I notice the BIOS setup screens look identical.
If you want uptime for an enterprise, you have to use enterprise class storage products, or distribute the data the way Google does. There is a reason EMC and Hitachi and others can charge what they do for storage - you can't match the performance, uptime and features.
Shadow copies? Look at SnapView and SANCopy in EMCs CLARiiON line - no downtime to create a copy. I would expect Hitachi and others to have similar features. There are a lot of used EMC disk arrays on Ebay and other places - just make sure you can re-license the software.
We have small "hand-built" RAID arrays in our lab totalling about 1TB, and 20+TB of various kinds of EMC disk. The 1TB of cheap RAID is more work to maintain than all the EMC disk put together.
I don't see why setting up the RAIDs under any OS should be more time consuming than on other OSs. Certainly if you use the right hardware-based RAID things should be very simple and very fast.
.8 TB or so at a very reasonable price and very reliable. So far it has worked well.
Bang for the buck, you can't beat the Apple Xserve RAID. They are IDE, but almost as fast as the fastest scsi arrays, and seem to be very reliable. The array can be easily partitioned into a variety of raid types with hot spares. The unit can then connect to Windows or Linux via standard fibre channel interface and look like simple scsi drives. The RAID is administered via an ethernet connection using a nice java gui tool.
We set our Xserve RAIDs up such that each array (each Xserve RAID box has 2 arrays with separate controller logic for each) is RAID 5 plus a hot spare, and then the array is mirrored with the other one. This gives is
Perhaps doing RAID over network block devices would solve your reliability problem. NBD is designed for RAID, you distribute over partitions that are physically separate from each other on different machines and segments, you can do heartbeat, etc. Don't assume that this is necessarily the "cheap way out with cheap hardware". You can do this with fast hardware that's backed by hardware RAID too and use it in a network RAID 0, 1, or 5 scenario for example.
I maintain a large number of Dell servers and I have NEVER seen computers malfunction so often before in my life. Our desktops seem to be far more reliable. Try RAID-10 if you want belt and suspendors (two hardware RAID 5 arrays put together in software as a mirror set). Even beter, try some kind of server clustering (Reduntand Array of Inexpensive Servers?)
Compaq's newer SmartArray have seemed to be more stable.. however only time will tell.
-un1xloser
v4sw6PU$hw6ln6pr4F$ck 4/6$ma3+6u7LNS$w2m4l7U$i2e4+7en6a2X h
If Mylex cards are failing, that's important! If RAID cards fail, then the company, and all its employees, are out of business. And that's what apparently happened to Mylex. It's now owned by LSI Logic.
At the low end of the scale, we seem to be having the same kind of problem. We are having a high failure rate with Promise Technology FastTrak Tx2000 controllers. Promise Technology seems to have lost the will, or maybe ability, to deal with problems.
When I read through the comments to this story, there are a lot of situations where RAID cards are failing. But why?
The problem seems to be industry-wide. I talked to someone in technical support at HighPoint and he said the mirroring controllers sold by HighPoint have random mirror breakage failures, also.
This is a new problem. Did Microsoft do something to break mirroring controllers so that customers will buy Microsoft's far more expensive solution? Is there some problem with modern hardware no one has discovered?
One thing I can say is that, in the past, these cards worked reliably, or the companies could not have stayed in business. Promise Technology mirroring controllers were reliable for us for many years. Now they often cannot even be installed without failing during installation.
The Promise FastTrak Tx100 cards always worked for us, too. The only Promise cards that fail for us are the Tx2000 cards. Since we have been unable to get help from Promise for this problem, I presume they know there is a problem, and are unable to fix it.
If your having problems with controllers, drives, enclosures, etc., going bad, then maybe you need to buy better hardware (i.e. more expensive).
i an tml530/index.htmlu cts/servers/prolian tstorage/arraycontrollers/index.html0 4.www1.hp.com/products/servers/prolian tstorage/drives-enclosures/4300enclosure/
I have been working with compaq proliant servers for several years (support for RedHat Linux is good) with nary a hardware problem.
http://h18004.www1.hp.com/products/servers/prol
http://h18004.www1.hp.com/prod
http://h180
I know that expensive is not always better or more reliable but failing (or nearly failing) to meet a SLA should get management buy-in to buy just about anything in the $K range.
[I haven't tried either of these products.]
Gateway 840 Serial-ATA RAID Enclosure is cheaper per GB than Xserve RAID. It has 12 bays and uses U320 SCSI instead of Fiber Channel for the connection to the system. Currently the cheapest config you can do is $4,749. That's with 4 250GB SATA drives and their cheapest 3yr warranty (another nice thing is you can increase the warrany to 4 or even 5 years and they have a variety of response times you can choose). Gateway gives you all 12 carriers no matter how many drives you buy from them. So you buy 8 more 250GB drives for $225/ea. ($1800) for a total of $6,549. Apple won't sell you drive carriers, you have to buy the carriers with their drives. They currently charge $450 for the driver carrier + 250GB ATA drive. Xserve RAID with 12x250GB drives and a 3yr. warranty costs $10,998.
The cheapest way to go is to build you own using a PCI RAID controller and drive cages in a large PC case. There are drawbacks to the DYI method but a 12x250GB SATA RAID system would cost you about less than $5000 ($2700 for drives, $750 for 3ware 8506-12 RAID card, ~$450 for 3 drive cages, the rest is for a big-ass case, mobo, etc.). Note that includes the cost of the computer which the above OEM options do not include.
I can count on one hand the number of RAID controller failiures I've had. If I remove external factors (like power failures), then I think I've had only 1 controller fail.
If uptime is so absolutely crucial how about a duplexed mirror of RAID 5 arrays. Two controllers and a RAID 5. When in doubt throw more money at the problem. :)
Everybody dies frustrated and sad and that is beautiful
When that slow 250GB ATA class drive is dead, and while its fellow drives are chugging their little hearts out (and probably maxing out that 3ware controller), how long will it take to rebuild your array?
Have you tested how long it takes? Probably better than 24 hours if your system is moderately loaded.
Guess what you have now? The marvelous opportunity for a CASCADING FAILURE!
That's right kids! Because you just had a drive fail, and all the other drives are doing double the work to rebuild from parity data, you have a higher chance of getting a second drive failure.
Consider that you bought all of the drives in that array at the same time. They've all been running the same amount of time. What if there was a minor manufacturing defect that caused that First drive to fail? How soon before it takes out the other 4?
A 'resume generating event' waiting to happen.
Best of luck.. and I agree with the comment upthread. SATA drives are for Workstations. Maybe for storing what we call 'reference data'.
Not much more.
There's a few choice terms in the industry- 'Economy Enterprise'
'Garbage RAID'
'Ghetto SAN'
Good luck
Striving to achieve a lower state of conciousness
What you said is what I would expect. However, I called and talked to someone in 3Ware technical support, and he said it would not boot with only one drive; it would be necessary to rebuild the array to boot the computer. Maybe you are using a different controller.
In addition, here are questions and answers from a session on the 3Ware chat system:
Request:- 10th January 2004 at 8:23
[Irrelevant questions removed here.]
We've been using Promise RAID controllers with our cash register software, and experiencing excessive failures. We are considering moving to 3Ware 7006-2 mirroring controllers.
Can we rebuild the mirror using only the 7006-2 firmware, without booting? Can we clone hard drives with the firmware?
Can we boot from one hard drive of the mirror, when that hard drive is temporarily attached to the motherboard IDE controller?
Does the mirroring controller store anything on the hard drive that would interfere with other devices or software?
Response:- 12th January 2004 at 11:31
Michael,
You can build/rebuild a mirror in the bios before booting to the OS.
You cannot attach the drive to the motherboard controller and boot from it.
The controller allocates a small space in the hard drive less than 1MB to store the raid information
Sincerely,
3ware Customer Support.
Microsoft's solution is that everyone should buy Windows 2003 server and use software RAID, available only on that Windows OS.
That's all we need is software RAID mirroring, but it doesn't make sense, for this application, to support a much more complex system and much more expensive system to get it.
If (disk)space and performance is not a problem (i.e. HD below 200GB, non-fancy single CPU), you could simply go with two (or three) cheap PC boxen instead of one "data center quality" RAID machine (for the same total price). If you mirror data+setup over from "production" to "standby" daily, any downtime due to any failure (HD, controller, mobo, OS, filesystem) can be minimized to 1-2 minutes (switch service over to the standby) - continuing with yesterdays data, which should be sufficient for most cases.
Integrating a backup/backlog (e.g. 3 months data) into a mirror setup is possible in several ways - my company does offer such a solution (managed service, that is).
Continuing with current data instead of yesterday's status is quite a bit more challenging, though...
Buy a NetApp Filer, mount it and use it for all your variable data. Get rid of the RAID arrays attached directly to the servers.
+++OK ATH
You just don't know what you're doing. Go paint a picture or something.
So, did you "post" that raid-5 + raidtools + kernel 2.6 locks up a computer, to save somebody else going through what you had to ?
The Internet's nature is peer to peer - 20050301_cs_profs.pdf
Especially if you are dealing with an IBM 4758. They detect a casual battery replacement as an attack and clear their memory, which is a good thing. Point is, you had better know what you are doing.
Lasers Controlled Games!
Ahh the sign of a true nerd - That depends question springs up flushing you out.
You dont have a depends in the question you have a statment you answer with single engine or twin engine. Which would you rather be in when one engine fails. Your not flying around in a simulator your answering a rethorical question - So go crash and dye.
We are using power supplies that seem like the best (KingWin), although not expensive.
We have tested these units by putting a re-boot program in the Win XP startup folder. This causes continous reboots. We have run several computers more than 12 hours continously rebooting. This should show problems with the power supplies.
We do NOT see problems, usually, with the Promise Tx2000 controllers when continously rebooting. The problems come after the units are delivered to the customer, a terrible situation.
try IBM Total Storage Solutions, and solve your problems with information management, administration, backup and/or archive
the great thing about the newer proliants (in addition to their exceptionally reliable raid cards) is their management cards. you can setup tftp, put your custom kernel/modules there and boot strap complete debian systems in like 5 minutes.
once you like that idea you can build in dynamic fault tolerance. have a server go down? take your hourly rsync of the production data (and config) and dynamically build a new replacement box.
poor man's on-demand.
having dealt with these in very high profile environments, i can --assure-- you that just about ANY alternative would be preferable. maybe you meant f5?