Slashdot Mirror
Annual Hard Drive Reliability Report: 8TB, HGST Disks Top Chart Racking Up 45 Years Without Failure (arstechnica.com)
Online backup solution provider Backblaze has released its much-renowned, annual hard drives reliability and failure report. From a report on ArsTechnica: The company uses self-built pods of 45 or 60 disks for its storage. Each pod is initially assembled with identical disks, but different pods use different sizes and models of disk, depending on age and availability. The standout finding: three 45-disk pods using 4TB Toshiba disks, and one 45-disk pod using 8TB HGST disks, went a full year without a single spindle failing. These are, respectively, more than 145 and 45 years of aggregate usage without a fault. The Toshiba result makes for a nice comparison against the drive's spec sheet. Toshiba rates that model as having a 1-million-hour mean time to failure (MTTF). Mean time to failure (or mean time between failures, MTBF -- the two measures are functionally identical for disks, with vendors using both) is an aggregate property: given a large number of disks, Toshiba says that you can expect to see one disk failure for every million hours of aggregated usage. Over 2016, those disks accumulated 1.2 million hours of usage without failing, healthily surpassing their specification. [...] For 2016 as a whole, Backblaze saw its lowest ever failure rate of 1.95 percent. Though a few models remain concerning -- 13.6 percent of one older model of Seagate 4TB disk failed in 2016 -- most are performing well. Seagate's 6TB and 8TB models, in contrast, outperform the average. Improvements to the storage pod design that reduce vibration are also likely to be at play.
Every time Backblaze publishes a report the HGST drives always come out on top.
It's a little more expensive to fill your NAS with them but in my experience it's been worth it.
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
Is aggregate usage even a meaningful metric?
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
Is aggregate usage even a meaningful metric?
It tells you the MTBF for right now, but it's not useful to predict MTTF unless you know the shape of the bathtub curve. It takes a few years to build that curve.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
Arstechnica just borderline copy&pasting from the source. See the actual article at: https://www.backblaze.com/blog...
Shame on Arstechnica for not even bothering to link their source material.
This space is not for rent.
Yes, aggregate usage is a meaningful metric, if you know what it defines. MTBF can be tricky, in many cases it is converted to Annualized failure rate (AFR) to obtain a meaningful metric.
However, it makes no sense to employ metrics based on an exponential distribution model (which does not have memory) to compare different sets of disks. In particular, the summary says 13.6 percent of one older model of Seagate 4TB disk failed in 2016... If such drives are older (and thus present a longer uptime) their age induces a higher failure rate, which is not observed in the model since it only considers the uptime hours in the given year but not the previous ones.
45 years spread over a bunch of drives without a failure doesn't mean that we can expect any individual drive to last 45 years.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
And "few years" is approximately half of MTTF, or do we now know enough to determine failure distribution indirectly?
The bathtub curve is real, and if you follow BackBlaze tips, they show that years 2-4 are usually exceptional in terms of reliability.
My recommendation is to buy the NAS/SAN/POD/Whatever and spin it up for 3 months, then put it into production and then wait 42 months. After that, start planning and when the next drive fails in the 42-48 month range, start the purchasing process (depending on lead time needed), get it installed, wait 3 months to get early failures out of the way than transfer data ... wash/rinse/repeat. You'll get close to five years between purchase and retirement, with a bit of overlap between versions.
If you have several decks of drives, you can get a reasonable cycle going, and it becomes second nature. Data loss is not an option.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
45x3 = more than 145?? Who worked this out?
One you see the error rate start to rise, it can be effective to fit to the expected curve shape, but not always. Crystal balls are unreliable.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
Interesting, wouldn't manufacturer be incentivized to minimize early failures, as it would be most expensive for them?
Also, how do you account for bad batches/production runs or do they always show up during initial 3 month period?
Aggregate years are not years.
"Nine women can't make a baby in one month."
In another study, Ford took 1,000 cars and ran them for a year without a problem. This translate to all Ford cars will last 1,000 years without a problem.
Sheesh!
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
Is aggregate usage even a meaningful metric?
I find it hard to believe. It isn't measuring 45 years worth of things like metal fatigue, material decay or degeneration, wear and tear etc.
What its really saying is that early failures are at a very low rate; they've measured lots of disks for a few years and can show that these disks don't typically fail in the first few years of use. Totally different from saying that one of these disks can last 45 years of continuous use. To represent it as that seems like something doomed to litigation.
In the free world the media isn't government run; the government is media run.
In a server, always on environment these are great numbers but in power conservative desktops/home NAS situations I'd love to see CSS numbers. Again though, Seagate still sucks.
Harrison's Postulate - "For every action there is an equal and opposite criticism"
The bathtub curve is real
This Backblaze report, previous Backblaze reports, and the Google logitudinal disk reliability study, have all found that the "bathtub curve" is a myth. HDDs do not have high early failure rates, nor does the failure rate suddenly rise after a set period of time.
Another myth that these studies have debunked is that HDDs do better if kept cool. Actually, failure rates are lower for disks kept at the higher end of the rated temperatures. This is one reason that Google runs "hot" datacenters today, with ambient temps over 100F.
>have all found that the "bathtub curve" is a myth
I don't believe that. I can believe the length of the bathtub curve is much longer than the useful life of the disk drive.
It's real in board products, cars and silicon.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
you've inherited your grandfather's old farm plot in Stardew Valley. Armed with hand-me-down tools and a few coins, you set out to begin your new...More this Website http://bit.ly/2kfQKaX
In my job we sell tens of millions of each product. We warrant for X years (E.G. 8-10 would be typical for something with a natural replacement cycle of 4-5 years), so we then design the things such that the curve is at a low point at time X. Component aging is heavily modeled and measured so we don't mess up. It would indeed get very expensive if there were lots of early failures. You find bad batches through testing.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
I find it hard to believe. It isn't measuring 45 years worth of things like metal fatigue, material decay or degeneration, wear and tear etc.
For spinning disks, factors that do not materialize in the first few years and thus can't be determined from an aggregated short time test include (but are not limited to)
- demagnetization of fixed magnets (leads to write failures)
- magnetization of paramagnetic materials (leads to bit rot)
- wear on ball bearings (leads to all kinds of fatal crashes)
- accretion of and contamination of lubrication (leads to sticktion)
Based on my own experience as a long term sysadmin, the quality and longevity of drives go up and down. Late 1990s drives were bad, early 2000s were good, late 2000s were bad, early 2010s were good, and now it's pretty bad again. It's not just vendor specific, because vendors seem to adjust to each other to arrive at common price/quality point. Sure, there are exceptions, like the Deathstars, but overall, I think the drives tend to be similar in longevity not so much based on brand, but what generation they are.
Personally, I get tired of Backblaze numbers. It's great that someone takes the time to compile data but everyone in social settings points to them as a reason to buy/not buy a product. Drive failures will vary due to conditions, who you purchased them from, etc.
Always run diagnostics before shoving a drive somewhere. Always have redundancy if it must be up 24/7. Always have backups.
Personally WD Red's have been my source of pain over the years.
FYI, Toshiba does not provide diagnostic tools for their drives. If you have or suspect a failure, you must RMA the drive through them.
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
It won't, but may. The only thing that we know for certain is that ... you don't understand reliability figures.
The aggregate data shows the combined effect of random failures and infant mortality. It has nothing to do with wear out failures which limit the ultimate life of a drive. These combined effects are also what is most relevant to most HDD use cases, mainly those drives that aren't abused, and don't find a home in some bank's basement where they have to sit for 20 years without fault.
What its really saying is that early failures are at a very low rate;
Not just early. Early and random failures. When you include both of those, providing your equipment has a relatively short mission time compared to wear out it gives you a good indication of how reliable it will be over-all in your server.
The bathtub curve is certainly real, but most drives aren't kept in service long enough to see the far wall - Google's study only goes to 5 years, for example. As failure rates start climbing you tend to replace the lot of them, rather than keep them in service until you reach 50% failure/year. (Note that the PDF you linked does show high infant mortality for drives in heavy use.)
I used to work with very old HDDs, though, and even with a busy used market, the supply of old drives would fall off a cliff at a certain point. When everyone is seeing 50% failure/year, it doesn't take long until spares just can't be found.
(If you're curious why anyone would put up with that sort of thing - the software that works only works on a machine old enough that only very old drives can attach to it. And since demand at the time was maybe 1% of the peak, you'd be using old drives until about 90% ever made had failed.)
Socialism: a lie told by totalitarians and believed by fools.
MTBF has little to do with MTTF, Film at 11.
Socialism: a lie told by totalitarians and believed by fools.
because that's how I've started to think of them.
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
Is aggregate usage even a meaningful metric?
It tells you the MTBF for right now, but it's not useful to predict MTTF unless you know the shape of the bathtub curve. It takes a few years to build that curve.
Is that a standard tub or a "garden" tub?
is superior. WD may have bought them up, but it's the same engineering departement, thus the highest quality drives. WD's own breed, as we can often see from reports like these, are churning out mechanical turds as ever.
Looking at data from both Backblaze and Google, what's apparent to me is that all brands have some good models and some bad. Google made sure to point that out in their report. Something like "the most reliable model and the least reliable model are the same brand. While reliability is somewhat consistent within samples of the same model, there is little to no correlation between any brand name and reliability".
In other words, these studies show that HGST Model #12345678 is a good drive. They don't show that HGST (or any other company) consistently makes good drives.
Are you currently using a drive built in the 1970s? The 1980s or 1990s even? If not, you probably don't care about a drive that may last 45 years. You care, probably, about how likely it is that a drive will fail in the ~3 years before you upgrade it.
I'm no hosting/cloud service provider. I have a pretty solid rig outfitted with a boot SSD, 6 Mechanical drives, and a BluRay burner.
I bought six of the 4TB Seagates when they first came out and all but one had died by the fall of last year. I replaced the 4Tb Seagates with 8GB HGSTs the first day I could, and all have been working fine since installation. (though I did have a problem getting one of the HGST drives recognized by the OS (Win 8.1 Pro x64)
Sig Follows: "Suppose you were an idiot. And suppose you were a member of Congress. But I repeat myself." -- Mark Twain
I can believe the length of the bathtub curve is much longer than the useful life of the disk drive.
The "bathtub curve" has two ends. Neither end is valid for HDDs. If a HDD spins up and formats, then it is no more likely to suffer an "early death" in the first few months than it is to fail in subsequent months. Likewise, the sharp rise in failures after 3-4 years doesn't appear valid. There is certainly a rise, but it is not that sharp. Also, HDD failure is more strongly correlated with accumulative spin time than with calendar age.
It's real in board products, cars and silicon.
These are different issues. Boards often fail because electrolytic capacitors dry out, and less often because of tin whiskers. Those are both aggravated by age and heat. I am not a "car guy" so I don't want to comment on that. I very much disagree that there is a "bathtub" failure rate for actual silicon (rather than chip to chip connections). I have 40 year old TTL chips that still work just fine.
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
Is aggregate usage even a meaningful metric?
It tells you the MTBF for right now, but it's not useful to predict MTTF unless you know the shape of the bathtub curve. It takes a few years to build that curve.
Is that a standard tub or a "garden" tub?
Neither. It's a rub-a-dub-dub-tub.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
If manufacturers do their job, consumers should never see the leading edge. If the HDD study says there's no leading edge then that's good enough for me.
For modern silicon, electromigration has much less distance to travel than in 40 year old TTL chips. E.G. for a 10 year old chip, the distance to travel is a lot less that 1/4 of the distance to travel in a 40 year old chip that is 4X as old. You won't see the leading edge because manufacturing test is effective.
I should use this sig to advertise my book ISBN-13 : 978-1501515132.
In other words, these studies show that HGST Model #12345678 is a good drive. They don't show that HGST (or any other company) consistently makes good drives.
It doesn't matter if it's good when I can't find model #12345678 sold anywhere I search.
Hitachi is by far the most consistent though. With other brands a few models have really high failure rates, while Hitachi varies from excellent to just very good. They seem to test their designs much better, and of course you pay for that.
I'd be interested to see a comparison of SMART data between models and manufacturers too. I strongly suspect that some are much better at warning you of impending failure than others.
const int one = 65536; (Silvermoon, Texture.cs)
SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
one 45-disk pod using 8TB HGST disks, went a full year without a single spindle failing... 45 years of aggregate usage without a fault.
Oh, yeah? I have a 1,000 disk pod that went a MONTH without a single spindle failing... 83 years of aggregate usage without a fault!
These error margins put the actual failure rate (within the confidence interval) well within the range of most other drives tested. So you can't say with confidence that these particular drives with zero failures were the most reliable. (Looking over their data, it does seem HGST drives are statistically more reliable than Seagate and WD drives. Goody for me - I've been a big fan of the IBM/HGST/Toshiba drives ever since they went overboard improving them following the "DeathStar" fiasco, and have been using them predominantly. Sometimes an embarrassing product failure is the best thing for a company.)
In other news, in 2062 they will have time travel, otherwise how could you possibly know that just-released 8TB drive would last 45 years?
You know damn well that's unlikely and you're purposefully misunderstanding this.
It's quite obvious to *anyone* with an ounce of common sense that it refers to an 8TB drive they've been running continuously since 1971. Occam's razor, see?
"Slashdot - News and Chat Sites Deviant". (Click "homepage" link above for details).
When everyone is seeing 50% failure/year, it doesn't take long until spares just can't be found.
(If you're curious why anyone would put up with that sort of thing - the software that works only works on a machine old enough that only very old drives can attach to it. And since demand at the time was maybe 1% of the peak, you'd be using old drives until about 90% ever made had failed.)
At what point do you look at emulation of the system?
I supported an *old* customer tracking/billing system for a local oper for a while. I was able to move him off the 80286 to a new Pentium 4 (at the time) and was able to tune a QEMU system to support him correctly. Hardest part was supporting the printer (app was hardcoded for a positively ancient HP Laser, or an Oki dot matrix).
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
> I'd be interested to see a comparison of SMART data between models and manufacturers too. I strongly suspect that some are much better at warning you of impending failure than others.
That *would* be interesting. That might be more consistent, with some manufacturer normally providing good data. Of course no matter what SMART does, if a model with 5 platters is subject to catastrophic failure, SMART can't do anything about that.
People who know more than I about hard drives have written that different manufacturers calculate SMART data *differently*, so to make predictions based on SMART, you need to know how to interpret the data from that specific manufacturer. HGST data may not be *better* or *worse* than Toshiba, just different, so if you use Toshiba drives you want to know how to understand Toshiba data.
Ps a clear example of this is that all manufacturers make drives with different numbers of platters. A drive with 5 platters is FAR more likely to fail than a drive with 1 platter. They may be made by the same manufacturer, but the 5-platter model is at least 5 times as likely to fail (platters interfere with each other).
Show me a report where HGST drives are not in the top 20% in terms of reliability.
At what point do you look at emulation of the system?
We were doing hardware-assisted emulation for new systems, but old systems were in the field. Eventually they were replaced, but they were expensive enough that we didn't until we finally couldn't get drives.
Socialism: a lie told by totalitarians and believed by fools.
if I put 100 tires on a track and ran them for a year and got 10000 (normal year) of wear on them.
It would not mean that any of them would be good for 10 million miles.
Once you get past the first burn in period, they should all run about the same distance.
Run them until failure and let me know the MTBF.
Even Backblaze warns these numbers shouldn't really be used by the average consumer to justify their drive purchases, and for very good reasons.
The numbers lie.
They lie because you don't use drives in the manner that they do, Backblaze starts a pod, it fills with data and then primarily sits IDLE from that point on. In other words, they fire it up, does a ton of writes then does nothing, whereas your drives write, read erase, spin up, spin down constantly. Your drives sit in a box that may be in a warm closet, lack air flow, or sit by your feet getting bumped all of the time.
Check out the Google reports over the years. I think you'll find they have some good and some bad. Specifically, the more platters a drive has, the greater the chance of failure - regardless of manufacturer.
If you want to be a fan of any one brand, that's fine, doesn't bother me.
But...modern HP LaserJet printers still grok PCL, and Oki still makes dot matrix printers.
Wouldn't printing have been the easy part?
Kid-proof tablet..
With lower tolerances, more precise engineering, variance drops. Mean time to failure may remain the same, but instead of one disk working a month, and another ten years, you have twenty disks failing within three months of each other, five years from now. Bearings wearing the same, grease drying up at the same rate, springs losing flexibility at the same rate - the date of failure ceases to be a random factor, and becomes highly deterministic. And the fact that not a single disk failed within a year means only that they are performing very similarly - not that they are performing extremely well.
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
I checked the specific 8 TB hard drive referenced in the article, and it's helium filled.
That's not the type of hard drive I'd want to rely on for any more than a few years, at least until they've perfected helium technology.
Mainly I wonder how they plan on keeping the helium sealed inside the hard drive given that seals degrade over time.
3 * 45 is 135?
The ars article even seems to lack the link to the primary source.
So stop linking secondary sources here!
I do similar, using LVM and mdadm. I've found it works well. Reliability is much increased by a) automatic monitoring of SMART data which warns me of impending failures via email and b) weekly scrubbing, checking that all blocks are consistent.
> As long as I replace the drives when it fails
The above monitoring and scrubbing lets me replace drives shortly BEFORE they fail, and mostly ensures that the remaining disks don't have hidden errors. A rebuild is intensive, so it can certainly cause a "working" disk to fail at the worst possible time, if you're not verifying the health of those "working" disks weekly.
Based on my own experience as a long term sysadmin, the quality and longevity of drives go up and down. Late 1990s drives were bad, early 2000s were good, late 2000s were bad, early 2010s were good, and now it's pretty bad again. It's not just vendor specific, because vendors seem to adjust to each other to arrive at common price/quality point. Sure, there are exceptions, like the Deathstars, but overall, I think the drives tend to be similar in longevity not so much based on brand, but what generation they are.
I agree. And even within a vendor and within a model there can be huge variation. I worked on a server farm where we had hundreds of 'identical disks' (same make, model and vendor) except some were made in Hungary and some were made in Thailand. The Thailand disks were failing at an enormous rate.
In the free world the media isn't government run; the government is media run.
That's exactly what I thought.
no. Trying to get this thing to print was a *bitch*.
and when I say hardcoded I mean hardcoded. Parallel port output only, puked on the windows virtual port, total pain in the ass. I hinestly think they didn't actually use the printer drivers, but rather bit bashed the port output.
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
Another myth that these studies have debunked is that HDDs do better if kept cool. Actually, failure rates are lower for disks kept at the higher end of the rated temperatures. This is one reason that Google runs "hot" datacenters today, with ambient temps over 100F.
Funnily though, I've had very good success with IBM Deskstars (back during the infamous era of "Deathstars" click death) simply by running them cooler.
Though again, I did only have a few. So I only have anecdotal evidence.
Maybe I was just lucky to have the few Deskstars that didn't went "Dearthstar".
But maybe thermal management was indeed exceptionnally a problem on these old drives.
"Sufficiently advanced satire is indistinguishable from reality." - [Tips: 1DrYakQDKCQ6y52z6QbnkxHXAocMZJE61o ]
MS-DOS didn't have printer drivers. Bit-banging was unusual because MS-DOS did provide a character interface to the printer port (typically as a device called LPT1:, which you can easily parse as the equivalent of /dev/whatever, plus or minus some CPM-ish),
But even early Windows releases were half-fucking-decent at capturing LPT1: output and spooling it appropriately for MS-DOS applications, but you said this shit the bed, too.
That said, doesn't QEMU (and friends) provide a properly-virtualized parallel port -- bit-banging and all?
(And if not, it should.)
Kid-proof tablet..
The use-case for B.B is very different than NAS users. Pay more server NAS drives to reduce failures. Learned my lessen losing media files with no warning due to low level sector failures. SMART is not smart. I am using WD RED drives now in my QNAP with very good results.