Data Center Study Reveals Top 5 SMART Stats That Correlate To Drive Failures

Lucas123 writes Backblaze, which has taken to publishing data on hard drive failure rates in its data center, has just released data from a new study of nearly 40,000 spindles revealing what it said are the top 5 SMART (Self-Monitoring, Analysis and Reporting Technology) values that correlate most closely with impending drive failures. The study also revealed that many SMART values that one would innately consider related to drive failures, actually don't relate it it at all. Gleb Budman, CEO of Backblaze, said the problem is that the industry has created vendor specific values, so that a stat related to one drive and manufacturer may not relate to another. "SMART 1 might seem correlated to drive failure rates, but actually it's more of an indication that different drive vendors are using it themselves for different things," Budman said. "Seagate wants to track something, but only they know what that is. Western Digital uses SMART for something else — neither will tell you what it is."

Comments

Skip the blogspam, here's the real link

[Anonymous Coward]

https://www.backblaze.com/blog/hard-drive-smart-stats/

Goes into a lot more detail too.

The measurements in question:

[Immerman]

for those who are only passingly curious and don't want to read the article.
        SMART 5 - Reallocated_Sector_Count.
        SMART 187 - Reported_Uncorrectable_Errors.
        SMART 188 - Command_Timeout.
        SMART 197 - Current_Pending_Sector_Count.
        SMART 198 - Offline_Uncorrectable

Re:The measurements in question:

[SpaceManFlip]

I read the article to find those "5 Top SMART Stats" they refer to, but I'm replying here because it's the relevant place.

Those 5 SMART stats match up exactly with what I habitually look at on the job monitoring lots of RAID arrays' drives. Those are the stats that tell you if the drive is going bad most often in my experience.

Re:The measurements in question:

[omnichad]

And I can confirm. Reallocated Sector Count rarely goes above zero when the drive is fine. It's possible to have a few sectors go bad and get reallocated, but it's usually part of a bigger problem when it happens (this number is reset to zero at the factory, after all initially bad sectors have been remapped). If the Current Pending Sector Count is non-zero, it's likely over.

I always clone a drive immediately with ddrescue when it gets to this point, while the drive is still working.

Re:The measurements in question:

[koinu]

Reallocated_Sector_Count sectors that the drive successfully replaced Reported_Uncorrectable_Errors errors that could not be recovered by ECC Command_Timeout controller hanging and had to be resetted Current_Pending_Sector_Count sectors to be replace by the next write access Offline_Uncorrectable sectors that the drive tried to repair, but failed (try offline test, maybe it is not dead yet)

Re:The measurements in question:

[omnichad]

Also, generally you don't need to panic over this attribute. You should panic when it increases steadily.

True, I've had a few drives hold steady at 1 sector reallocated. But if Current Pending Sector count remains non-zero for very long, it's a headache at the very least and probably a failure. Generally, it seems like as soon as you crest zero, it's over. I've had the next symptom be a totally unresponsive drive. But doing the backup when you hit 1 (admittedly overly cautious) will force the drive to read off all the sectors and you'll at least get your backup while you verify the rest of the drive still reads OK.

Re:The measurements in question:

[swillden]

Yes. This article isn't exactly news as it pretty much confirms what the global peanut gallery has already said about this stuff.

Still, data is better than emergent collective perceptions from distributed anecdotes.

Re:Correlation != causation

[Immerman]

Nope. When looking for warning signs you don't care about causation, it's enough to know that the presence of A indicates an increased probability of imminent B.

Re: Seagate OEM?

[corychristison]

I buy whatever is cheapest.

I know it's a toss up no matter what or when you buy hard drives, so the only thing I have left to guage is price, capacity, and speed (RPM) depending on the intended use.

About a year ago I took a gamble on an SSD for my primary workstation. I bought an ADATA SX900 64GB drive. I had never heard of the brand before. It was ~$120 at the time, and the cheapest for that capacity. I've been looking at getting a 128GB (or so) SSD for my laptop. Prices right now look like I will be getting another ADATA... but I am holding out for Black Friday/Cyber Monday deals to decide.

Oddly enough, over the past 10 years, I've never had a hard drive die in any of my computers while in use. I have a stack of 4 or 5 drives, ranging in capacity from 100GB to 500GB, 3 different different brands, that I'm not using right now. A while back, I plugged one in just to see if it still worked and it didn't. I recently found out it was the hotswap bay that quit working, so as far as I know it still works.

Conversely, I have some servers in a datacenter. Had a drive fail on reboot after a kernel upgrade the other night. Sent a ticket to the DC and they plugged a new one in. Good to go again. In case you're wondering, it has 4x600GB SAS drives in RAID-10.

TL;DR: Buy whatever is cheapest, the odds are always the same.

Re:Uncorrected reads

[ls671]

I have had drives fail. I took them off line and wrote 0 and 1 to them with dd until Reallocated_Sector_Ct stops raising and Current_Pending_Sector goes to zero then ran e2fsck -c -c on them 2 or 3 times then, I put them back on line!!!

Most people would say this is crazy but in my opinion, the surface of the drives often have bad spots while the rest is perfectly OK. Some on those drives are still on line without reporting any new errors after more than 5 years, some almost 10 years. Those are server drives with very low Start_Stop_Count, Power_Cycle_Count and Power-Off_Retract_Count. All lower than 250 after 10 years. Those drives are spinning all the time.

Newer drives will relocate bad sectors to free reserved space they keep for that purpose. As long as you don't run out of free spare space, IMHO, it is worth a try.
 

Re: Seagate OEM?

[brianwski]

> TL;DR: Buy whatever is cheapest, the odds are always the same.

Disclaimer: I work at Backblaze. I'm going to completely agree with you wholeheartedly, and say in addition you must have a backup. You don't have to use us, I'm just saying if a drive has a 1 percent chance or a 30 percent chance of failing, the actionable item is the same - keep a backup and buy the cheaper drive and restore from backup when it happens.

> over the past 10 years, I've never had a hard drive die in any of my computers while in use.

Professionally we lose something like 10 (?) drives every single day at Backblaze, but *PERSONALLY* I had a LOT of luck for a number of years, but about 3 years ago I finally lost one drive. I'm more backed up than most people, so it was a completely relaxed event. Not a bit of stress. Replace the drive, re-install the OS, and restore the data. Yet something like 95 percent of people never backup their data. IT professionals backup up their family computers, but once you are out there in "normal computer user" land, it's a horror show.

Re:Put the SMART stats to the test

[brianwski]

Disclaimer: I work at Backblaze. Essentially this is what we did. We don't care at all if one drive dies, so we left it in an environment where we can read and write them all day (the storage pods with live customer data) and when they failed we calmly replaced them with zero customer data loss and produced this blog post. :-)