Slashdot Mirror
Compelling Alternatives to RAID Setups?
jabbadabbadoo asks: "Our software shop has about 30 Linux servers and 15 NT servers running enterprise applications for our customers. Since we have service level agreements with most of them, uptime is crucial. One of the things we've done is to use RAID setups extensively, using products from well renowned disk- and controller vendors. However, we have discovered the paradox that introducing RAID controllers actually reduces overall uptime! Not only does more 'steel' increase the probability of failure, but what fails first is usually the RAID controllers. What is your experience? Have we been having bad luck?"
"A related problem, especially on Linux, is that setting up RAIDs is actually a quite costly process. There seems to be endless problems with library versions, and upgrading existing servers simply takes too many hours. To keep the customers happy, we routinely have to create a 'shadow' server while upgrading which in turn means we, at some point, have to synchronize data to the new server, which in turns means a bit of a downtime. Ouch. Does anyone have a good solution to these problems? Of course, cost is a major issue, but so is uptime (which also means cost if we don't provide the uptime dictated in the SLA). What setup gives the best cost/uptime ratio? Thank for any thoughts!"
I remember swapping quite a few Compaq RAID controllers in my day. They wouldn't outright fail, but get in a "compromised" mode, and you usually had enough time to schedule downtime to swap them out. This was much better than messing with software mirror or raid settings, because it's transparent to the OS - the OS just sees a single large disk.
I'll agree that setting it up is a nightmare. I'm currently helping test two 4TB arrays for use on a Linux box (16 SATA drives presented as a single SCSI device). Benchmarks under linux are slower than under windows. It's a mess figuring out why. Meanwhile, vendors (who I will not name ship crappy software, and take months to act on bug reports.
As for transitioning servers, I've been there too. And yes, copying a terabyte of disk in single is a very long process. It'd have taken several days, which is of course unacceptable. This is where the magic of rsync comes in handy. Copy the data over several days in advance, sync it just before the scheduled downtime, and you'll have a fairly short downtime.
You don't list what brand controllers you are using, but your problems are not typical in my experience. We are a 100% Compaq shop and use their SmartArray controllers with Novell Netware and Debian Linux. We've never had a controller failure and have only lost about 3 drives over the last six years or so.
I'm a firm believer that you get what you pay for with enterprise-class servers. You shouldn't expect Tier-1 reliability from servers that are built with commodity hardware. There is a reason that Compaq/Dell/IBM servers are more expensive.
We also haven't had any issues installing other than the default Debian boot disks not supporting the SmartArray controller. A custom set of disks took care of that though.
Jason
"FORMAT C:" - Kills bugs dead!
In the past two years, none of the "downtime" that I've experenced has been attributed to the disk array or controller.
The biggies have been: power outage that exceeded the capacity of the UPS (3 hours), planned upgrades and an anonymous gremlin who bumped the reset button - since detached.
This is a boring sig
XSan can 'hide' the complexity of RAID, as well as providing management tools and 'intelligent' cascading failure... but that's just from reading the specs, not from actual experience. I hear XSan is based on CVFS? I should look at that too.
GPL Deconstructed
This is on a lower level than the RAID you are using, but we are having major problems with 10 Promise Technology TX2000 mirroring RAID controllers that we bought. The mirrors go critical for no detectable reason. Promise Technology technical support is unable to find the problem, and the company is unwilling to escalate the issue. The Promise Technology technicians escalate the issue, but 2nd level technical support never calls back.
Promise mirroring controllers on ECS (EliteGroup) L7VTA v 1.0 motherboards have the same problem. When we call ECS tech support, there is a recorded message saying they are busy and to call back later.
We've been supplying computers with Promise mirroring RAID controllers since the company began doing business, and we've had very few problems until now.
Possibly the problems are associated with newer, faster motherboards, or with AMD VIA chipset motherboards. We've never had problems with RAID controllers on Intel chipset motherboards.
Another possibility is that the RAID controllers are incompatible with DVD burner drivers that are installed with Roxio or Nero DVD burning software.
It's hard for me to believe that RAID causes more downtime than single drive setups, unless you have a really bad raid system and a really good backup system.
:-) Soon to go away... :-(
The only time RAID should ever be down, is during initial setup. Thereafter you should replace bad drives while it's running and you should never have cause to shut it down due to a RAID issue.
If you are experiencing RAID hardware problems then take a good look into these areas:
RAID Hardware --> Are you using cheap stuff? It honestly isn't worth it. Perhaps you're just discovering the 'real' value of 'cheap' hardware.
RAID Software --> If you're using unsupported drivers (ie, vendor doesn't supply or support them) then ditch the hardware and get hardware with supported drivers - make sure they support them on your configuration. You've already proven that you can't support them yourself.
System Hardware --> If the system is generally cheap (cheap power, bad airflow, cheap components, etc) then you simply can't expect the RAID card to work 24/7.
Server Room --> Make certian your server room can handle the power and ventilation needs of the servers. This should go without saying, but all too often it is the problem.
The reason people go with cheap components is the lower initial cost. They only work for a few thousand hours of heavy operation. You must get server rated components if you want them to operate for more than a year or two. There really is a difference.
Lastly, I use 20+ Promise FastTrack ATA RAID cards in 20+ Novell networks. I use cheap components, and they work in harsh conditions. They are not set up for hot-swap, as that's not a need in this situation. I have to replace the cheap hardware every 2-4 years, powersupplies every year, hard drives every 2-3 years. The only time the RAID cards have gone bad is when a power supply failure (usually due to a power outage/surge/brownout) fries the motherboard and usually most of the components in the case.
I have never had a failure where both HDs completely failed simultaneously, though usually when the rest of the computer goes I replace the whole thing and get the data off one of the old hard drives. This is not an advertisement for Promise. They simply are the only one's with supported Novell 3.12 drivers.
I'd be surprised if you've covered all these bases and are still having problems.
-Adam
There is an old saw in the aviation industry: "A twin engine aircraft will have twice as many engine problems as a single engine aircraft."
However, which would you rather be in, a twin engine aircraft that just lost one engine, or a single engine aircraft that just lost an engine?
Yes, RAID cards die - I've been shocked at how often that happens. And 5 disk RAID will have more failures than a 4 disk JBOD (just a bunch of disks) array.
But the question is, are you seeing a reduction in UPTIME, or just in mean time to failure? Maybe the RAID system throws an error once a month and the JBOD system throws an error every two months, but if you can recover in 5 minutes by swaping cards or drives rather than 5 hours for restoring the JBOD from backup, you are better off.
Perhaps what you might look at would be using RAID software on the server's processor, coupled with Firewire drive bays, disks, and multiple Firewire cards. If you have a card die, move the disks to another card until you can schedule downtime. A disk dies, hot-swap and rebuild in background.
www.eFax.com are spammers
The answer is SysAdmin 101 stuff.
..
1. Buy quality hardware.
IDE RAID for critical servers is a bad idea.
In my experience, RAID hardware tends to be very picky and suffers from subtle and often bizarre hardware conflicts. In general, using a RAID solution that is packaged with the hardware is the best idea.
If you cannot afford good RAID hardware, stick to conventional JBOD configurations.
2. Configuration
Design your the configuration of your systems around consistency first, performance second.
You need to document your procedures for building servers, allocating storage, etc. Create scripts whenever possible.
If you are not confident that you could not talk a marginally qualified technician through a server rebuild over the phone, your docs aren't good enough. If you don't have the time to write docs, make the time or work late.
3. Backups
You need documented, tested backup AND restore procedures. All of your oncall staff need to be able to restore a server.
With 50 servers, disk controller or disk failures should not be a common event. We work with approximately 400 datacenter and 200 field servers (varying in age from 1-9 years), and replaced 3 controllers and 19 disks last year.
Look for electrical issues, you may have crappy electrical service.
Conformity is the jailer of freedom and enemy of growth. -JFK
I spent the past week and a half trying to set up a 4x160 SATA Raid-5. It was a huge excercise in frustration because every time I'd try to build a volume, my machine would promptly freeze after a few percent. I changed out IDE emulation for SCSI emulation in kernel... same thing... I changed SATA controllers, same thing. I changed SATA cables, same thing. I changed power supplies, same thing. I added 4 80 mm case fans, same thing. In the end, it turned out that the culprit was raidtools. Nobody had ever bothered to post that raid-5 + raidtools + kernel 2.6 locks up a computer. I changed to mdadm, and I had a working array 50 minutes later.
If your bandwidth requirements are not too high you may be able to use a distributed file system on many redundant (cheap IDE & G ethernet) nodes and allow for replacements. Your uptime should be constant, given enough UPS and redundancy of nodes.
--
"we live in a post-ideological world..." - Billy Bragg.
They're systems are probably 80% auctioned desktops and such from busted dot coms.. and I suspect that many of them are not RAID at all. I have yet to hear of a redundant raid controller either. Your best bet is just replication of data on you backend servers and using something in the nature of a Cisco CSS or some other services balancer device to handle keeping alive servers available while redirecting away from dead servers.
You can still do RAID with this setup but you'd have the added security of 2 or more systems making up your entire functional system so if one is down the other can continue normally. Then it's trivial to repair the dead machine and bring it back into the cluster.
You can't slap a buzzword like RAID onto whatever you were doing before and expect results. Reliable systems have to be carefully engineered correctly.
From the sound of your posting, I'm assuming when you say you're using RAID, you mean internal RAID cards inside a server with internal disks attached, and relatively small amounts of it. In these types of scenarios, the highest performing, most reliable, and most cost effective option is to put two seperate scsi controllers in your boxes, buy twice as much storage as you need, and mirror between the controllers using the OS's software mirroring capabilities. You are now indepedant of controller failure, the controllers themselves are less likely to "fail" (which doesn't always mean hardware frying) than a complex raid controller by their simpler nature, and you're getting the performance benefit of full mirroring instead of that clunky raid5 business. If you have enough storage to warrant four or more internal disks of some size, use mirror+striping. Always mirror at the lowest level, and then stripe on top of that (in a 4 disk design actually it doesn't matter which way you layer them, but in 6+ disk designs it gives higher data availability in the unlikely event of multiple disk failures). Or in other words - raid5 and hardware cards = bad, mirroring/striping + software raid = good.
Your goal is not to be buzzword compliant by slapping in a raid controller, your goal is to carefully analyze your systems, your options, your requirements, and your budget, and eliminate single points of failure everywhere that it's feasible and desirable to do so, starting with the lowest MTBF items in the system and working your way up. There are no magic bullet answers of course - change the situation and the "right" answer can change dramatically.
11*43+456^2
I don't see why setting up the RAIDs under any OS should be more time consuming than on other OSs. Certainly if you use the right hardware-based RAID things should be very simple and very fast.
.8 TB or so at a very reasonable price and very reliable. So far it has worked well.
Bang for the buck, you can't beat the Apple Xserve RAID. They are IDE, but almost as fast as the fastest scsi arrays, and seem to be very reliable. The array can be easily partitioned into a variety of raid types with hot spares. The unit can then connect to Windows or Linux via standard fibre channel interface and look like simple scsi drives. The RAID is administered via an ethernet connection using a nice java gui tool.
We set our Xserve RAIDs up such that each array (each Xserve RAID box has 2 arrays with separate controller logic for each) is RAID 5 plus a hot spare, and then the array is mirrored with the other one. This gives is
If uptime is so absolutely crucial how about a duplexed mirror of RAID 5 arrays. Two controllers and a RAID 5. When in doubt throw more money at the problem. :)
Everybody dies frustrated and sad and that is beautiful
When that slow 250GB ATA class drive is dead, and while its fellow drives are chugging their little hearts out (and probably maxing out that 3ware controller), how long will it take to rebuild your array?
Have you tested how long it takes? Probably better than 24 hours if your system is moderately loaded.
Guess what you have now? The marvelous opportunity for a CASCADING FAILURE!
That's right kids! Because you just had a drive fail, and all the other drives are doing double the work to rebuild from parity data, you have a higher chance of getting a second drive failure.
Consider that you bought all of the drives in that array at the same time. They've all been running the same amount of time. What if there was a minor manufacturing defect that caused that First drive to fail? How soon before it takes out the other 4?
A 'resume generating event' waiting to happen.
Best of luck.. and I agree with the comment upthread. SATA drives are for Workstations. Maybe for storing what we call 'reference data'.
Not much more.
There's a few choice terms in the industry- 'Economy Enterprise'
'Garbage RAID'
'Ghetto SAN'
Good luck
Striving to achieve a lower state of conciousness
If (disk)space and performance is not a problem (i.e. HD below 200GB, non-fancy single CPU), you could simply go with two (or three) cheap PC boxen instead of one "data center quality" RAID machine (for the same total price). If you mirror data+setup over from "production" to "standby" daily, any downtime due to any failure (HD, controller, mobo, OS, filesystem) can be minimized to 1-2 minutes (switch service over to the standby) - continuing with yesterdays data, which should be sufficient for most cases.
Integrating a backup/backlog (e.g. 3 months data) into a mirror setup is possible in several ways - my company does offer such a solution (managed service, that is).
Continuing with current data instead of yesterday's status is quite a bit more challenging, though...