Slashdot Mirror
Recovering a Wrecked RAID
Dr. Eggman writes "Tom's Hardware recently posted an article specifying how the professionals at Kroll Ontrack recover data from a RAID array that has suffered a hard drive failure, allowing for recovery of even RAID 5 arrays suffering two failures. The article is quick to warn this is costly, however, and points out the different types of hard drive failures that occur, only some of which are repairable. Ultimately the article concludes that consistent backups and other good practices are the best solution. Still, it provides an interesting look into the world of data after death."
I have a concern with the recommendations given in the introduction:
We assume that all hard drives will be handled with care, so they should be installed in suitable drive bays. If you use multiple drives, we recommend removable drive frame solutions, which help reduce vibration transfer onto the computer chassis and even back to individual hard drives. Make sure that your system has sufficient ventilation, so high speed hard drives won't overheat.
I've found that the removable drive frames available for cheap consumer hardware to be total crap. The metal enclosure keeps heat close to the drive, and the tiny fans used don't move nearly as much air past the drive as when it's inside the case, being cooled by the airflow of the case fans. The drive temperature is therefore higher even under the best conditions. In addition, the smaller fans fill with gunk quickly and as a result wear out faster than larger ones, leading regularly to a drive trapped in an uncooled box.
I've used enclosures from Promise, Enermax, and several other companies whose products were so bad I tried to forget their names; all had fans that instantly became the least reliable part of the entire system once I installed the drive frame, and I wasn't happy with the drive's temperature from day one.
I don't think the person making this comment at Tom's ever keeps systems running long enough to realize the long-term issues that come with anything cheaper than server-grade drive enclosures for hard drives. I'd welcome suggestions for a better quality product in this category. It's a hard subject to cover, because by the time you've had several units setup for a year or two to gather useful data on how rugged they are, the product is obsolete; not something any review site I'm aware of is setup to cover.
With recent articles on HDDs not being very good for redundancy (because they often fail at the same time if they are from the same batch, or fail because of things like electrical spikes which affect all drives in an array) it is clear that HDDs are not an ideal backup medium. I use an external 2.5" HDD which is totally disconnected from the PC and everything else when not in use (to avoid power surges etc), but only for critical data as my machine has 1.2TB of HDD storage.
Optical discs are a joke - 4.3GB is just not enough. Larger formats exist but are relatively expensive. Tape is expensive per MB and slow, plus it isn't random access and not suited to anything but slow full backups. MO is too small and expensive.
It seems like the best bet is something like a Century Tower - basically a USB enclosure that can take up to 4/8 drives. Keep it totally disconnected when not in use, and use RAID 0 mirroring with drives from different manufacturers.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
what's wrong with popping in a livecd like sysreccd http://www.sysresccd.org/Main_Page/
and to use dd to take an image of the disk or ghost (but iirc ghost uses dd) ?
i have been able to successfully recover 99% of a crashed, broken, badly partitioned hard drive that way numerous of times
offcorse i do not claim i have the expertise as ontrack but seeing as i've done this for quite
a few friends and since well not everybody can pay what they ask for their service, i can understand
why they get drives that have been subjected to a DIY recovery at first
But why do we need all these expensive consumer disk recovery tools, that often do not work correctly
i must agree on the issue that this article is mainly advertising but that is to be suspected
i mean the dude works in that company, he's kinda obligated to praise the so called 'superiority' of their own proprietary tools.
granted i don't have a clean room but the area in that so-called clean room doesn't seem so clean
and well the platters hanging on the top right on the other picture doesn't strike me as a good idea neither
that obviously wasn't a clean room and i would at least encourage the use of a static bag anyway.
I attended a small conference where the Kroll VP of Data Recovery was speaking. He came in, his assistant set up his power point stuff, made sure the projector was right etc. He then gave a very interesting talk about what Kroll could pull off of a drive, despite what had been done to it. By way of example he showed a slide of a burnt and bent hard drive - that came out of the sky when the shuttle broke up. They recovered 99% of the data on that drive. He also mentioned that they do the data recovery for all of the spook organizations in D.C.
When we broke for lunch I got to sit at his table and we got to ask him all sorts of questions about their processes. He mentioned they have things they use that they have never patented because it would be too much of a leg up for both the competition and those that seek to destroy data. We tried to get him to tell us what we would have to do to a drive to make it unreadable. Mostly his answers to our "Surely this would make the data unreadable" queries were "You would think that would work wouldn't you?" Someone referenced his assistant who was sitting next to him and the VP said:
"Him? No, no, no. (laughs) He is not my assistant, in fact he doesn't work for me at all. He is a lawyer for the company and is here to make sure I don't say anything I am not supposed to." The assistant then gave us one of those 'I could eat you alive' lawyer smiles.
I walked out secure in the knowledge that short of melting the platters down the data can *always* be recovered.
Sera
Slashdot, where armchair scientists get shouted down and armchair theologians get modded up.
Last week, i did a data recovery on a client that had multiple disk head crash from a power outage, or a kick or something. The drives were resulting in a click-seek, which for the most parts is unrecoverable.
Popped in a Helix disk, and checked what the MFT was doing. Low and behold, no MFT, no boot sector, and a huge list of bad sectors. Basically, the crash had resulted in a bad sector in the bad sector table, and all over the first portion of the disk.
These were 200GB disks, but eventually I was able to get a sector repair program to read through and do a non-destructive repair. Data was safe, but was now corrupt. Next step was to repair the data, and I was finally able to just use chdisk to repair.
Eventually, it was back to real data, and was able to push the data over to a new replacement hard drive.
Told the client to invest in RAID 1, but seriously doubt they would be willing to spend that $100 for the RAID. Instead, they prefer to pay $1000 for a repair.
BACKUPS. make lots of BACKUPS. RAID your stuff, and get those backups offsite. Do them regularly. Seriously, it would save your ass if something happens. For example, I have a LAN HD that is parked out in a shed in my backyard. Total cost $200, and has already saved my ass 2x.
I have 3TB of storage here. 1TB of that is in a 5 x 250 hardware raid-5. In this case it's a stand-alone enclosure with firewire/usb ports on the back. I chose this because it's easily portable to another machine, so if the server buys the farm, I simply unplug it and walk it to the next server and jack it in. Also if the computer crashes, there is less risk of corruption of data since the raid box is handling parity calculation. It also does not hammer the server if I have to do a rebuild.
The other 2TB is the large, more easily replaceable data, mostly video media. Those are arranged as single drives. I am not stupid enough to try to stripe them, I don't like the idea of any 1 of 8 drives failing leading to total data loss on all drives.
I have a script that runs once a week and reads 100% of all blocks on all drives, and emails me if a bad block was found. I replace the drive immediately. To date I have yet to lose a byte.
Closest call I had was a few years ago before the raid5, I had a pair of software mirrors. I had a server crash that wiped one drive's partition table and wiped the other drive's directory. Neither alone was fixable by any utility I tried. I ended up doing some really scarry things with DD and XXD in terminal to reconstruct the partition table from scratch on one drive and install firewire drivers on it so I could get at my data. I am very thankful for having a very high level of knowledge on partition table and basic directory layouts, most people would have had to cough up a stack of benjamins to get that fixed.
My near future plans are to buy two 1tb lacie bigdick (they have a horrible failure rate, never use them for critical anything) and use that to back up the loose drives. Long term I plan to get a larger raid5 box to phase out the old raid as primary storage, and convert it to cover some of the less critical storage. Right now the cheapest and simplest backup plan for us seems to be to buy large cheap drives and keep 1 or 2 complete clones.
I work for the Department of Redundancy Department.