Slashdot Mirror
Recovering a Wrecked RAID
Dr. Eggman writes "Tom's Hardware recently posted an article specifying how the professionals at Kroll Ontrack recover data from a RAID array that has suffered a hard drive failure, allowing for recovery of even RAID 5 arrays suffering two failures. The article is quick to warn this is costly, however, and points out the different types of hard drive failures that occur, only some of which are repairable. Ultimately the article concludes that consistent backups and other good practices are the best solution. Still, it provides an interesting look into the world of data after death."
RAID 5 is great, though expensive when done right. RAID 6 is better, though has less performance, as well as additional cost. Many controllers will not do RAID 6, and you lose 2 drives to parity. If your data is truly critical, you should have backups done VERY often, as well as a RAID 50. This way you are far less likely to lose data, though you have to have a stripe of at least 3 drives, in a mirror. This requires at minimum, 6 drives. There are also VRAIDs, which allow for you to lose drives until you hit the watermark of your data. This technology is usually reserved for SAN systems.
For DB's and home use a mirror set is usually best. For homes because it is simple, for DB servers because it is fast.
My home setup is a pair of 300 gig drives in a mirror, with another 1.6TB for other storage. Stuff that is important is on the mirror, and is differentially backed up to DVD regularly.
Stuff on the mass array is available in original form (my DVD and CD library that's been ripped) or is backed up whenever it changes, which is not often (my code library, for example). Active code and my wife's thesis are on the raid. Supporting documents for the thesis are on DVD and mass storage, as is old code projects that I may borrow from for functionality in a new project. The old project (and likely several versions of it) are off on DVD in a safe deposit box, with the rest of my backups.
Safe deposit boxes are awesome. I have one that can store 600 cds in cake boxes and it only costs $120/year. Dirt cheap for climate controlled fireproof storage.
-nB
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
RAID 50? Why not RAID 10? If you're already mirroring, the RAID 5 will probably not afford you much additional protection, and it has the effect of making writes slower.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
It takes far too many pages to say what could actually fit in a page or two.
Never put all of your eggs in one little basket (RAID or otherwise)! For the love of God, if your data is critical, you need a backup *and* an offsite backup. At least one of each. There are no exceptions to this rule.
Curiosity forces me to ask: If I'm looking into a series of cheap 80GB hard drives for $42, is it worth the extra ~$84 to go with a RAID 60, requiring 8 drives minimum, rather than the 6 drive RAID 50? I ask because in recent research, I noted that with RAID 50, if you suffer two HD failures in the same stripe, the entire array goes out, but with the RAID 60, you have to lose 4 HD (literally half the array) before the array is unrecoverable. Do you or anyone else have some preformance comparisons between RAID 50 and RAID 60? What sort of software and/or hardware options does one have with each type?
Demented But Determined.
People often poopoo software RAID (it is more of a pain to manage). But when it comes to recovery, it's what you want. You know the disk format and have the tools. Of course, you really shouldn't have to recover, you should keep good backups or another mirror if its that important.
Could these articles be any more annoying to read?
They painstakingly
NEXT PAGE
pull data
NEXT PAGE
off the
NEXT PAGE
damaged drive
And tell them to cause more damage next time or I will tell everyone they secretly admire Steve Ballmers commitment to developers.
Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
What truth?
There is no dupe
OK, this is for the very extreme (and rare) cases where the disk is physically very damaged. Most of the time, you'll find that available tools are enough. See http://en.wikipedia.org/wiki/SpinRite, for example. Has worked for me, but 1. Copy the entire disk contents first. 'Low-level' disk-to-disk dup utilities (Seagate...) can work fine here. 2. Be prepared to wait. Of course, if your disk is on its way out, the intensive reading, (and writing, in the case of SpinRite) may accelerate its demise. Keep the disk at a constant, cool temperate, (stick it in a domestic freezer if you've no aircon).
http://www.tomshardware.com/2007/02/14/raid_recove ry/print.html
I don't know why TH has printer friendly pages that they don't ever link to.
[Fuck Beta]
o0t!
I have a concern with the recommendations given in the introduction:
We assume that all hard drives will be handled with care, so they should be installed in suitable drive bays. If you use multiple drives, we recommend removable drive frame solutions, which help reduce vibration transfer onto the computer chassis and even back to individual hard drives. Make sure that your system has sufficient ventilation, so high speed hard drives won't overheat.
I've found that the removable drive frames available for cheap consumer hardware to be total crap. The metal enclosure keeps heat close to the drive, and the tiny fans used don't move nearly as much air past the drive as when it's inside the case, being cooled by the airflow of the case fans. The drive temperature is therefore higher even under the best conditions. In addition, the smaller fans fill with gunk quickly and as a result wear out faster than larger ones, leading regularly to a drive trapped in an uncooled box.
I've used enclosures from Promise, Enermax, and several other companies whose products were so bad I tried to forget their names; all had fans that instantly became the least reliable part of the entire system once I installed the drive frame, and I wasn't happy with the drive's temperature from day one.
I don't think the person making this comment at Tom's ever keeps systems running long enough to realize the long-term issues that come with anything cheaper than server-grade drive enclosures for hard drives. I'd welcome suggestions for a better quality product in this category. It's a hard subject to cover, because by the time you've had several units setup for a year or two to gather useful data on how rugged they are, the product is obsolete; not something any review site I'm aware of is setup to cover.
Sony ha
Guys, if you're doing regular backups and have a cold spare handy then RAID5 is typically more than enough. Two drive failures are exceedingly rare unless you have some sort of controller fault (and that will typically hit all of your drives anyway). Don't forget about the write penalty either, RAID 5 has a fairly stiff penalty, but RAID 6 is even worse. If you're talking about RAID5_0 or RAID6_0 you're almost certainly doing it wrong or planning for a day when you can't buy replacement hard drives (nuclear holocaust?).
To put it another way: What do you think your chances are of having a second drive failure in the few hours it takes you to replace the drive and rebuild it? Even if that does happen you just lose the data up until your last backup (a day at most).
Most professional installations I do are RAID1_0, because people are building the RAID array for the performance, not the cost. Since you're using crappy 80GB HDDs, I'm guessing you're going for cost, which makes it strange that you're thinking about a RAID6_0 solution at all (the controller alone won't be cheap for that). If you work the odds I think you'll find that it's just not worth it to build a RAID6_0, especially given the write penalty and complexity (complexity is your enemy with this, complexity means bugs, which can undermine your entire effort).
I read the internet for the articles.
SpinRite is a Steve Gibson product. Steve Gibson is a pompous blowhard with few real skills. There are plenty of other ways to do a low level copy of a disk.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
Mirroring is an expensive waste of disk space unless you really need the fast write times.
I think a 4 disk RAID-5 is a better sweet spot, since you lose less storage to parity/recovery data and it seems cheaper to buy 4x smaller drives than 2x larger drives. 2 300GB disks looks around $200 and 4x 120GB disks is around $220, but you end up with 60GB more space for the $20.
RAID 11? Or, more to the point, how would I implement a mirror, but with 3 drives? Does linux 'md' do this? How about any controllers?
After all, we're supposed to replicate data 3 times, right?
A sun D1000 loaded with latest-generation 300GB disk drives? Not a bad solution, slow, and not the cheapest.
Apple X-serve RAID? Cheapest - does it work reliably with Linux or Solaris? Word in the street is that it does, but I have not seen a demo yet.
We're actually going with recycling our ancient D-1000s and A-1000s with no-name 300 GB SCSI drives. Pretty old school, but reliable.
Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
It's not that expensive with the price of drives these days. The nice thing about a mirror is that if your controller (or something else if you have a software raid) dies you can mount one of the drives on its own. After dealing with a failed controller, I'm glad to fork out a little more money for the piece of mind.
Because maybe you've got a dozen drives in the array. RAID 10 would be a serious hit on capacity while RAID 50 wouldn't be so bad.
Some file storage doesn't need to be super fast but you need a lot of it and you need reliability. Hence, RAID 50. You'll learn that the first time you have a budget on a hardware deployment.
With recent articles on HDDs not being very good for redundancy (because they often fail at the same time if they are from the same batch, or fail because of things like electrical spikes which affect all drives in an array) it is clear that HDDs are not an ideal backup medium. I use an external 2.5" HDD which is totally disconnected from the PC and everything else when not in use (to avoid power surges etc), but only for critical data as my machine has 1.2TB of HDD storage.
Optical discs are a joke - 4.3GB is just not enough. Larger formats exist but are relatively expensive. Tape is expensive per MB and slow, plus it isn't random access and not suited to anything but slow full backups. MO is too small and expensive.
It seems like the best bet is something like a Century Tower - basically a USB enclosure that can take up to 4/8 drives. Keep it totally disconnected when not in use, and use RAID 0 mirroring with drives from different manufacturers.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
Bingo,
Plus the controller is cheaper, saving the 60 gig worth of money.
-nB
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
Oooh, and this just happened to me a few weeks ago. well, not quite, but close enough.
I had an LVM container that sat on a RAID-1 volume go bad.
the lvm tools couldn't reconstruct the container, so I effectively 'lost' my partitions.
There wasn't any program I could find which would scan the raid volume for the data partitions,
so I ended up cobbling one together on my own, out of the sources in the ext2-tools distro.
And yes, I did get my data back, and no, i'm no longer using LVM containers.
Support FSF: Stop thinking with your wallet, and think with your imagination. (cc/non-commercial)
Oh, I'm just trolling. Ignore me. It was mostly an excuse to make a "hack the gibson" joke. It was a dumb joke anyway.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
RAID 50 is pretty fast for reads, and not very reliable. Fail the wrong two drives, and down it goes. With a dozen drives in the array, that's getting pretty scary: odds are low, but still significant, that if one drive fails, another will fail before you can resilver, and almost a half chance that it's the wrong one.
I'd go for RAID 6 or RAID 10 for a big storage cluster, if it needed to be decently reliable. RAID 10 is a lot smaller, but it's faster and about 5 times less likely to fail than RAID 50. RAID 6 is about as fast as RAID 50, has the same storage capacity, and is much more reliable, but you'll probably need a RAID controller.
I hereby place the above post in the public domain.
Linux md RAID-1 allows you to replicate to n number of drives, PLUS set m more drives as spares that will be automatically substituted for failed drives without intervention. You can spread the drives among as many controllers as you want.
Of course you need off site backups too (fire, theft, lightning, human error).
Depends on what we're talking about.
For SMB customers, i usually use 2 seperate RAID1 sets.
Why? Because SMB customers usually use their stuff looong beyond the expected lifetime, and don't have any support contracts. If a RAID controller dies, you're usually in trouble - because a replacement without a contract costs more than a new machine. If you use RAID1, you usually don't have to worry about that, since you (usually) can just plug them into a normal controller.
Mirrors have slower writes and faster reads. This is because the controller can pull different chunks of data from both/all hard drives. The writes are not faster as there has to be twice as many. Striping has faster writes but no tolerance for failure
I'm a big fan of the hard drive->freezer method. It has been alleged that putting a broken hard drive into a freezer can sometimes make the data readable again for a short period of time.
This is good reading:
http://storagemojo.com/?p=383
Short synopsis for those who don't want to read it: The rebuild process is intense enough to cause secondary failures in many more cases than you'd think. Because you haven't seen it yet is not indicative of the overall population, and sysadmins are payed to be prepared.
The rest of your post is arguable, but it's more a matter of opinion and practice than anything else.
With the two drives on separate channels, mirrored writes can be done in parallel.
Intron: the portion of DNA which expresses nothing useful.
what i always wanted to ask a guy who clearly knows a lot about this stuff is ...?
what physical media do you recommend for regular snapshots/backups: tape archives, DVD's (it would take a hell lot of dvd's to backup 1 TB i think)
or just another machine (with or without RAID), NAS, SAN,
Besides having a backup not connected to system, i found simply having a spare disk to steal the circuit board off of to be a life saver :)
I miss the old bigfoot drives we had, everyone said they had problems with them but it was always (in our case) the board that died NOT the disk. I saved a couple of those by swapping in a board for a 1 hour recovery.
If you buy several HD for RAID or whatever buy one more and stick it on shelf for a rainy day. Along with a few utilities you can do 3/4's of what they do for $100 instead of $1000+
Really, a wrecked raid can be fixed pretty easily if you have enough warlocks to get everyone a soulstone.
Your mind is clear / The things that you fear / Will fade with how much you / Believe what you hear
When a disk fails in a RAID, it becomes an AID.
Or in the case of RAID-1, it becomes just an ID.
Most people don't even think inside the box.
I attended a small conference where the Kroll VP of Data Recovery was speaking. He came in, his assistant set up his power point stuff, made sure the projector was right etc. He then gave a very interesting talk about what Kroll could pull off of a drive, despite what had been done to it. By way of example he showed a slide of a burnt and bent hard drive - that came out of the sky when the shuttle broke up. They recovered 99% of the data on that drive. He also mentioned that they do the data recovery for all of the spook organizations in D.C.
When we broke for lunch I got to sit at his table and we got to ask him all sorts of questions about their processes. He mentioned they have things they use that they have never patented because it would be too much of a leg up for both the competition and those that seek to destroy data. We tried to get him to tell us what we would have to do to a drive to make it unreadable. Mostly his answers to our "Surely this would make the data unreadable" queries were "You would think that would work wouldn't you?" Someone referenced his assistant who was sitting next to him and the VP said:
"Him? No, no, no. (laughs) He is not my assistant, in fact he doesn't work for me at all. He is a lawyer for the company and is here to make sure I don't say anything I am not supposed to." The assistant then gave us one of those 'I could eat you alive' lawyer smiles.
I walked out secure in the knowledge that short of melting the platters down the data can *always* be recovered.
Sera
Slashdot, where armchair scientists get shouted down and armchair theologians get modded up.
As long as you know how the RAID config was setup(striping size), most disk recovery programs will do the job just fine. GetDataBack NTFS is functional and simple tool to use as long as you know how the disks were setup. Including RAID5...I've rebuilt 3 RAID5's and a shitload of 0's, 1's, and 01's. You should see the look on some of these people's faces after your done(with all 18+hrs of it...)The problem usually I find is that if you recovered the data then the customer is usually under the impression that you *fixed* the disk and they can keep on using it without replacing it...so yeah, it's not a big deal it's just a question of how much time you want to spend and how much time you have to finish the job.
Completely depends on your environment. Large tape changers (robots) are pretty much your only choice with big datacenters. For home use removable hard drives are often good enough, especially since most home users don't do offsite backups like they should. You can even used fixed hard drives for home use, depending on how much you value your data. DVD backups aren't worth it in my experiance, they're too finicky and streaming across multiple disks is a pain at best.
I read the internet for the articles.
``Still, it provides an interesting look into the world of data after death.''
Death before data!!!
How often do you see raid controllers actually fail? I'm not sure I've ever seen one fail since I started working with them in the early 90s.
Many complete RAID 5 failures can be "forced" back online by a good hardware RAID card BIOS, saving the money that would have been paid to Ontrack.
That said, RAID is no substitute for a backup.
All you need is the ability to read and write from at least one of the two failed drives, even if the drive has some bad sectors.
Install a new drive in place of the drive that is completely dead, and force the "only half dead" drive to work.
Many times you can recover your data this way.
Newer RAID cards can even fail "stripes" in an array, to "write around" bad sectors on hard drives while not losing the entire array.
You may lose the data that was in the "bad" stripe, but the rest of the array still contains useful data.
I am the unwilling control for my Origin.
It doesn't.
An alternative to a drive in a removable bay frame is in a SATA external enclosure. It has the advantage of a separate power supply (I've known PC power supplies to fail in a way that sends mains through the DC lines, frying everything). An external RAID1 (mirror) disk on hot-plug SATA is also something you can quickly grab in case of a fire, or to take somewhere else to mount read-only.
Last week, i did a data recovery on a client that had multiple disk head crash from a power outage, or a kick or something. The drives were resulting in a click-seek, which for the most parts is unrecoverable.
Popped in a Helix disk, and checked what the MFT was doing. Low and behold, no MFT, no boot sector, and a huge list of bad sectors. Basically, the crash had resulted in a bad sector in the bad sector table, and all over the first portion of the disk.
These were 200GB disks, but eventually I was able to get a sector repair program to read through and do a non-destructive repair. Data was safe, but was now corrupt. Next step was to repair the data, and I was finally able to just use chdisk to repair.
Eventually, it was back to real data, and was able to push the data over to a new replacement hard drive.
Told the client to invest in RAID 1, but seriously doubt they would be willing to spend that $100 for the RAID. Instead, they prefer to pay $1000 for a repair.
BACKUPS. make lots of BACKUPS. RAID your stuff, and get those backups offsite. Do them regularly. Seriously, it would save your ass if something happens. For example, I have a LAN HD that is parked out in a shed in my backyard. Total cost $200, and has already saved my ass 2x.
TFA is just an advertisement for Kroll Ontrack. Basically "you can phone Kroll Ontrack ..", "the Ontrack data recovery specialist will ..", WTF?
I use RAID 5 at home just for a little more piece of mind.
When your making a terrabyte server you dont want to lose a terrabyte of data overnight.
At least in Australia around June last year the sweet spot was 4x320gig 16mb cache drives.
Not sure if prices have changed since then but that gives you 1.2TB for the least amount of money.
I did get that setup and RAID 5 is running on the four drives. Works well.
Not if one of the hard drives is reading data when the write command comes in.
Then the read command has to complete first.
If the motherboard fails and is replaced, won't the disks be overwritten when reconfiguring the array?
If you use a reputable controller (i.e. one that costs more than your entire motherboard), it will read the configuration off the disks instead of overwriting them.
The issue is that there are 2 writes that must be done, even in parallel, is not as fast as reading from multiple platters. You don't get to cut the write in half and use the speed of the spinning platters to your advantage. In a RAID 1 Read, you can read partial data from either or both drives.
I'm looking at a SCSI RAID controller right now with one host channel and 6 drive channels. If I mirror a drive on two different drive channels I can run full speed writes from the host with no drive contention. Both mirrors are getting written in parallel.
It might be possible to read partial data from each mirror, but I've never seen a RAID controller do that. This one queues each read to the least busy mirror. Otherwise it has to process two SCSI commands instead of one.
Intron: the portion of DNA which expresses nothing useful.
After all, we're supposed to replicate data 3 times, right?
:^)
That's why my server has "New Folder", "New Folder(2)", and "New Folder(3)" on it.
As much as this stuff is cool, it's going to be insanely expensive to restore data from these guys.
Data integrity and uptime are served by RAID5. If it's not good enough, then it should be backed with mirroring (RAID5+0) or some form of dual-parity RAID (RAID-DP from NetApp, etc.).
But data gets lost or corrupted, even without disk failures. Backups are the place where data recovery is done. DO YOUR BACKUPS!
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
According to:
r /schroeder_html/index.html
http://www.usenix.org/events/fast07/tech/schroede
The chances of a double disk failure in a RAID 5 are significantly greater than we think. A friend of mine had a double failure just last week.
I have 3TB of storage here. 1TB of that is in a 5 x 250 hardware raid-5. In this case it's a stand-alone enclosure with firewire/usb ports on the back. I chose this because it's easily portable to another machine, so if the server buys the farm, I simply unplug it and walk it to the next server and jack it in. Also if the computer crashes, there is less risk of corruption of data since the raid box is handling parity calculation. It also does not hammer the server if I have to do a rebuild.
The other 2TB is the large, more easily replaceable data, mostly video media. Those are arranged as single drives. I am not stupid enough to try to stripe them, I don't like the idea of any 1 of 8 drives failing leading to total data loss on all drives.
I have a script that runs once a week and reads 100% of all blocks on all drives, and emails me if a bad block was found. I replace the drive immediately. To date I have yet to lose a byte.
Closest call I had was a few years ago before the raid5, I had a pair of software mirrors. I had a server crash that wiped one drive's partition table and wiped the other drive's directory. Neither alone was fixable by any utility I tried. I ended up doing some really scarry things with DD and XXD in terminal to reconstruct the partition table from scratch on one drive and install firewire drivers on it so I could get at my data. I am very thankful for having a very high level of knowledge on partition table and basic directory layouts, most people would have had to cough up a stack of benjamins to get that fixed.
My near future plans are to buy two 1tb lacie bigdick (they have a horrible failure rate, never use them for critical anything) and use that to back up the loose drives. Long term I plan to get a larger raid5 box to phase out the old raid as primary storage, and convert it to cover some of the less critical storage. Right now the cheapest and simplest backup plan for us seems to be to buy large cheap drives and keep 1 or 2 complete clones.
I work for the Department of Redundancy Department.
http://docs.sun.com/app/docs/doc/819-5461/6n7ht6q
For example:
But pretty much agree with everything else you've said... Though a hot spare is always nice.
Deleted
I've replaced two failed RAID controllers in the last year. Both of them were Compaq SmartArray boards, somewhere in the neighbourhood of 4-5 years old.
"...always new atoms but always doing the same dance, remembering what the dance was yesterday." -Richard Feynman
-b.
I had one fail on Thursday :(
In a Raid 10, a two-drive failure evebt can result in data-loss.
In a Raid 50, it cannot.
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?