Hardware For Bulk IDE Hard Drive Burn-In?

r0gue_ asks: "I work for a mid-size OEM hardware manufacturer. We ship approximately 300 to 500 IDE HDs every month across all our units. Currently we experience about a 4% failure rate (Maxtor and WDs), though in recent months it has been a couple percent higher. The problem is our systems are dedicated boxes with a non end-user friendly form factor. Virtually every physical HD failure results in an RMA. What we are looking for is a hardware based IDE HD burn-in platform. Something that we could drop a dozen or so drives in at once, stress test them for a day or two, then put them into inventory for builds. I know the HD manufacturers and larger OEMs use them but I have not been able to track down anywhere we could purchase one. Right now moving to SCSI or a form factor that supports externally removable drives is not an option. I was hoping that the Slashdot community could point me in the right direction."

Comments

Here's a solution:

[Swift Guru]

For the love of God, don't use Western Digital or Maxtor drives. It's like you're asking for that 4%.

Re:Here's a solution:

[MarkusQ]

For the love of God, don't use Western Digital or Maxtor drives. It's like you're asking for that 4%.

Yep. Use only Maxtor. That way you should be able to get 8%.

-- MarkusQ

Re:Here's a solution:

[tomoe27]

Western Digital: I've owned several western digital drives over the past decade, and none of them have ever failed me. At my workplace, I've found old WD drives in Pentium I PC's that have been in service for 6+ years without a single problem.

Maxtor: I've been plagued with problems from maxtor drives over the years. From one original Maxtor i've bought (and it's RMA replacements), i had 2 that had spindle motors that became abnormally loud, one catastrophically fail (IDE Auto-detect had problems even detecting what the drive was), and then the last one i had started failing (with SMART warning, at least) about 5 days after my warranty period ended.

Seagate: I've never used any of the newer seagate offerings, but my older seagate drives lasted for years before i replaced them with higher capacity drives.

Quantum: Most of the quantum drives (standard 3.5" form factor) i have encountered at my workplace have performed reliabily over the years, recently however we've started seeing a bunch of them failing, but considering most of them are 4+ years old, it's not a bad track record. I recently had a quantum bigfoot die last fall, but even that was close to 4 years old also. (Just what exactly prompted quantum to make this strange hybrid form factor bigfoot anyways?)

IBM: The older drives were great, and I used to love their service, i would contact their support and they would do an advanced RMA with no problem. However on my most recent experience with them in Fall '02 getting an RMA on a drive less than a year old, they told me they couldn't Advance RMA me a drive, i would have to use standard RMA, which including shipping times, would take almost a month, which was quite unacceptable. Not to mention how their newer drives had a problem with failures.

Re:Here's a solution:

[mmontour]

You can't go by brand alone - at some point every manufacturer has had a line of bad drives.

StorageReview has a Drive Reliability Survey that lists statistics for many drive families. For example, WD 205Bx drives are near the top of the rankings (99th percentile) while the 600Ax is near the bottom (10th percentile).

Re:Here's a solution:

[afidel]

That 99th percentile is based on barely enough drives for it to be rated (just over 60) so it doesn't mean much, most of the drives have hundreds of units in the database. Besides I think the database is in many ways flawed as most people who list their drives will do so because they have had a failed drive. The best way from my perspective is to look at what companies with hundreds or thousands of drives are doing. Rackspace switched out all of their IBM IDE's for Maxtors, google uses Maxtor's, and the recent IDE backup unit featured here used Maxtors. But maybe I'm just biased because as an OEM I had tons of drives from almost every manufacturer die on me except for Maxtors.

Re:Here's a solution:

[slaker]

I'd suggest Samsung. Yes, I'm being serious. Even the best of their drives is slow, but "slow" just means that the 7200rpm 80GB Samsung brings up the tail of the pack of _current_ ATA drives, performing better than current 5400rpm entries from WD and Seagate and just a hair slower than current Seagate and Maxtor drives. Before someone jumps on me about performance, do TRY to keep in mind that any current ATA drive is going to be substantially faster than any two-year-old ATA drive, mainly due to the benefits of increased platter density.

My main reason for suggesting Samsung, aside from the joys of a real 3 year warranty and the fact that Samsung drives really are value-priced, is that my return rate, and the return rates of several other resellers I know, has been exceedingly low.

How-to

[m0rph3us0]

Put 8 IDE controllers into a box (more than this maxs out the PCI bus bandwidth) .
write bash script that checks dmesg for how many drives are in the system and invoked the follwing perl script for each drive.

Write perl script that does this.
formats and partitions drive to max size,
copies a kernel or some other large file onto the disk until it is full.
monitors syslog for IDE errors
md5sums the files to make sure they all match.
reports an error if the MD5 doesnt match.

unless you get hotswap controllers you will have to reboot everytime you want to test another batch of drives.

if you dont wish to write this perl script i can be hired to do it for you.

Re:How-to

[m0rph3us0]

one more thing, dont put more than 1 drive on each channel as it massively slows down the operation.

Re:How-to

[torpor]

It's kinda stupid to only do *8* disks at a time, when you can easily do 64 ... using Firewire.

My advice would be to investigate into as many Firewire->IDE convertors as your company can afford, and then use a Firewire-friendly OS to do the burn-in. Something like OSX or Linux would work very well in this case - actually, a cheap Apple machine would be perfect for this application.

There's no need to start things up in batches with Firewire, either. You can plug in a disk, and your 'stresser' program can be written in a way that it just picks up that disk, stresses it, and reports failures along the way.

Would be a very simple project. If you want specific help, feel free to contact me ...

Re:How-to

[vadim_t]

Yeah, that'd be real fast. The bandwidth of Firewire is less than PCI. But okay, suppose you get several cards. In this case, the bandwidth is still 133MB/s. Assuming that you have all that for your disks, which doesn't include the network card, sound card, overhead, and whatever else you have, that gives 2MB/s per disk. Real fast.

Now, my motherboard supports PCI 64 at 66Mhz, with a bandwidth of 532MB/s, this would give 8.3MB/s per disk. Still not a lot, and you'd have to find a PCI 64 Firewire card with a lot of connectors, because at least my motherboard has only two slots.

My 80GB disk can do 40MB/s quite easily according to hdparm, so with my available bandwidth I could support about 13 drives, let's say 10 to compensate for overhead and other things on the bus. I think that 40MB/s is quite near the limit of Firewire, so I might need ATA instead. With two serial ATA cards with at least 5 connectors on each I suppose it'd be possible. Parallel ATA would also work, I guess, but the wiring would be really complicated with so many drives, especially because you want to have a drive per cable for maximum performance.

Re:How-to

[dubl-u]

That's a good approach. At 8 drives a day that's 250 a month for a station that you can build for well under $1000. I'd only add that you may need to tune this based on the failure modes that you are seeing.

For example, if it's just bad spots, then you'll want to do as many reads and writes as possible. For that, the fastest thing would be a little C program that reads and writes different patterns to the raw device linearly.

On the other hand, if the failures are tied to seeks, you'll want to write to semi-random locations on the device, to force maximum seeks. Or if you see a mix of both, then your best bet might be to follow m0rph3us0's plan, perhaps tweaking it a bit to better simulate normal filesystem efficiency (and you can just do bit compares rather than md5sums if CPU is an issue).

You should also keep an eye on heat issues. The burn-in should happen at temperatures that are like what they will be in the end systems. If you pack 8 seeking drives into some cases, they'll cook. If you leave them in the open air, they might not trigger the failures you are seeing in the field. Try to match measured operating case temperature.

Oh, and don't forget to measure whether this burn-in is really helping. Take stats now, and keep tracking causes of return. It could be that the drives are sensitive to noisy power or vibration or something else that your burn-in won't catch.

Re:How-to

[torpor]

How do you - at a consumer'ish level - fix the ugliness of the IDE cable, and non-hotswappable capabilities? I would think being able to load disks on and off without having to do a full system re-boot would be pretty advantageous... more important than speed concerns, anyway.

Can your PC really do sustained writes to >8 drives without getting into performance issues?

Actually, speed is a good point. I guess I hadn't thought about that so much as part of the setup, but I guess if you're exercising disks to make them fail, you want to do it as fast as possible... even though, in your end-system where the disks will be used, a Firewire stress-test may be more realistic in terms of disk behaviour.

Either way, there's tradeoffs.

I used to repair disk drives...

[dfinster]

In the late '80s I did hard drive repair and we used Wilson and Flexstar equipment for testing and burn-in. I can't find any links to Wilson equipment right now. Flexstar had a more extensible architecture and sounds like what you need. I've used the 2550 series RLL and EDSI Flexstar modules (this was the late '80s, we all thought that IDE was a passing fad at the time) and I can verify that the programming language for this equipment was very straightforward. The Flexstar equipment was very reliable. The only trouble we ever had was the cable ends that would naturally wear out from constant plugging and unplugging. We just replaced all the cable ends every two or three months.

Western Digital and Maxtor are equal?

[Futurepower(R)]

Slashdotters! If you don't find a story interesting, please don't complain and call Slashdot lame. Just ignore the story. Do you complain to your local newspaper that they should not publish recipes because you don't cook?

Comment about the Slashdot question: The wording of the question seems to imply that you believe that Maxtor and Western Digital hard drives have an equal failure rate. That has not been my experience. My experience has been that Western Digital are the most reliable hard drives. I'm very interested to know the experience of other readers.

Western Digital went through a bad stretch in which they experienced a problem that caused high failure rates several years ago, but that was cured.

It's shocking that you are in the computer business and knowingly shipping products with a 4% failure rate. That's very expensive and annoys the customers.

However, you are on the right track. Electronic products have what is called "infant failure". Most failures occur in the first week. During 192 hours (one week), the failure rate falls typically by a factor of 100 or even 10,000. At the end of one week most failures have already happened.

It's very easy to write a program that exercises a hard drive. Just copy files back and forth from folder to folder. It is easy to write a program that fills a hard drive with files, then erases them and starts again.

The Promise Ultra133 TX2 supports adding four more hard drives to the 4 already supported by modern motherboards. Eight is enough for one test computer, usually, because the power supply won't support more. Be careful to use delayed start. Maybe you will need more powerful power supplies than you normally use.

Make SURE that you are not having troubles with heat. Are your drives cool when they are installed in your product? High heat will cause high failure rate.

Re:Western Digital and Maxtor are equal?

[GigsVT]

Addendums to your message:
With a true 400 watt power supply, you can easily power 16 drives reliably. For reference, 8 drives pull a total of about 5-6 amps on 12v spin up, for about 1 second, then together use less than an amp on 12v, and very little 5v. This is based on testing with Maxtor 5400rpm drives, 7200 probably use a little more, and other brands may vary.

Power specs given in hard disk spec sheets are mostly boilerplate and do not reflect actual power consumption, the actual consumption is usually much lower than the spec.

ATA doesn't support delayed start, your power supply has to be able to take the full startup. 3ware makes controllers that support up to 12 drives, and hot swap when you use thier hot swap bays. A setup like that isn't cheap compared to the Promise card, but it may be worth it if you are testing hundreds of disks.

Re:Western Digital and Maxtor are equal?

[GigsVT]

You can do that, just have to make sure the grounds don't float and are tied together.

As far as powering them up in a sequence, there is no need to do that really, you can just turn on all the power supplies with the same switch. That's a little trickier to do with ATX, but cyberguys sells a adapter to make an ATX power supply act like an AT one with an external switch, and AT style motherboard connector. Or, since you aren't using the motherboard connector, you could just send the power supply the on signal the same way the motherboard does (read the ATX spec).

Most redundant power supply systems are really parallel power supply systems, so if you really need something like 800 watts, just look for a case that can take redundant supplies and put two 400s in, or two 480s, or whatever you can afford. :)

Case solution

[lusername]

We built some disk arrays using a front-loading IDE case with drive trays. This one is pretty pricey but it's _nice_ hardware:

https://www.rackmountplus.com/spec.asp?ID=RMAC4D -IDE

That, plus a couple RAID cards (like 3ware's new 12-port cards) in a 64/66 PCI slot and bonnie++ would do a pretty good job of burning in your drives. You could flip drives in and out in a few seconds.

Why not

[Froze]

buy a few IDE raid cards and set them up raid one? This impliments a full mirror of data on the raided devices. Then perform burnin on the raid device.

Note: I have never implimented raid and am not an expert, so this idea would need to be independently verified.

3Ware card + Removable Hard Drive Bays

[MetricT]

We have several IDE fileservers at work. Each box is equipped with two 3Ware 8-port controllers, and 16 removable drive bays. Stick a 17th drive in there as an OS drive, install Linux, and run benchmark of your choice. Once you're happy with the drives, just pull the bays and swap in new drives.

Re:3Ware card + Removable Hard Drive Bays

[afidel]

Yep and use software from Extreme Protocol Solutions or someone like them. Yes you can put together your own testing software, but why bother when there are others out there who have already gone through all the variables and problems. They explicitly support 3Ware cards for IDE testing see This link.