Seagate Firmware Update Bricks 500GB Barracudas

Voidsinger writes "The latest firmware updates to correct Seagate woes have created a new debacle. It seems from Seagate forums that there has yet to be a successful update of the 3500320AS models from SD15 to the new SD1A firmware. Add to that the updater updates the firmware of all drives of the same type at once, and you get a meltdown of RAID arrays, and people's backups if they were on the same type of drive. Drives are still flashable though, and Seagate has pulled the update for validation. While it would have been nice of them to validate the firmware beforehand, there is still a little hope that not everyone will lose all of their data."

Comments

At least no censoring

[amclay]

I'm glad to see them trying though. It's nice of a company to realize they made a mistake, and work to fix it.

Re:At least no censoring

[ta bu shi da yu]

Oh they are trying. Trying to bugger up systems! Surely if they validated the firmware update before releasing it the problems would have been caught in the QA process? I'd love to have been a fly on the wall in the QA meeting after the latest fix was released.

Re:At least no censoring

[berend botje]

So far, there is no indication that they even have a QA process...

Re:At least no censoring

[Karl Cocknozzle]

I am glad they've owned up to the mistake, but remember back in the day when Seagate was a trusted brand? Man, I feel reaaaally old right now...

Re:At least no censoring

[Zenaku]

Me too. But then my first (grown-up, non-dish-washing) job was with them, so my brand loyalty may have to do with that kool-aid they were always handing out. . .

I have a solution for long term data storage.

[Anonymous Coward]

clay tablets.

Re:I have a solution for long term data storage.

[Gerzel]

No golden plates and seer stones are the way to go.

Re:I have a solution for long term data storage.

[DigiShaman]

Yes, because they never get broken.

Here is a perfect example. http://www.youtube.com/watch?v=L940yIeVZzE

Re:I have a solution for long term data storage.

[beav007]

CDs work by blistering aluminium foil with a laser.

Wrong (at least for the vast majority of current cases). Manufactured CDs are pressed, while CD+/-Rs have an organic dye that the laser heats to change its optical properties.

Re:I have a solution for long term data storage.

[AlecC]

The magnetic field recorded in lava flows has lasted hundreds of millions of years.

That's it! Put a coil round an active volcano and write your long term data into the lava. No more lost backups now!

Re:I have a solution for long term data storage.

[gweihir]

CDs work by blistering aluminium foil with a laser. It's not perfect, but it works.

Not at all. Except for the stamp in stamped CDs, and that laser you cannot afford. The user writable formats use a laser-induced chemical reaction in a dye or phase change material.

Please do not post such nonsense.

Re:I have a solution for long term data storage.

[Andy Dodd]

Core memory is making a comeback (sort of) - http://en.wikipedia.org/wiki/Magnetoresistive_Random_Access_Memory

If You Can Reflash It, It's Not Bricked

[Fieryphoenix]

Ay Caramba already.

Re:If You Can Reflash It, It's Not Bricked

[Anonymous Coward]

Except that there are cases in this incident where you can't reflash it. So bricked is correct.

Re:If You Can Reflash It, It's Not Bricked

[Urza9814]

I gotta agree with the GP. I mean, the term is 'bricked' as in 'it is now worthless as anything other than a brick (paper weight, building material, etc). If you can just reflash it, it's not bricked. Now of course there are a variety of levels of not being able to flash it anymore, but I would say that if you can flash it back using the same process you used to flash it in the first place...obviously you know how and are capable of doing it, therefore it should be reasonably simple for you to fix it and therefore it is still worth more than a brick. 'Bricked' means you can't fix it, you send it in for service, and all they can do is throw it in the trash and give you a new one.

Re:If You Can Reflash It, It's Not Bricked

[ResidntGeek]

The whole point of calling something a "brick" is that's how useful it is - it can't be made to do anything better, ever again. If you can plug a cable into something, and run a program on your computer that makes it able to store data or play MP3s or whatever, it's CLEARLY more useful than a brick.

Re:If You Can Reflash It, It's Not Bricked

[ResidntGeek]

Why shouldn't it be proper to say something is a brick if it can't do anything better unless it's fixed?

Because a brick can't be fixed to do anything better. The term as originally used is wonderfully descriptive - I think I first heard it used about the PSPBrick trojan, which really turns the PSP into a BRICK. Like, you can't do anything with it anymore. There isn't a thing in the world that you can do with your PSP, ever again, except keep a table from wobbling. I like having a term for that sort of hardware-disabling software problem, and I can't imaging there's anything as evocative as "brick" for the purpose.

Re:If You Can Reflash It, It's Not Bricked

[Flentil]

What if you could send it to a 3rd party to get it working again, like one of those data-recovery specialists? What if it costs $800 to do that? Is it considered bricked then because it's 'totaled' like a car? See it's a slippery slope that easily avoided by simply accepting the current accepted meaning of something being bricked. It's not working right now. It's not good for anything but a paperweight. It's like a brick. It's bricked. Get it fixed tomorrow and it's un-bricked. See that's easy. If you want to talk about something being broken beyond repair, I'm sure there's some other word for that.

Re:If You Can Reflash It, It's Not Bricked

[smellotron]

...I'm not sure "ever again" needs to be part of the definition.

Every time I've ever heard the term "Bricked", the "ever again" has been the most significant implication. The term loses its meaning if you expand it to include any device that is currently not functioning.

Re:If You Can Reflash It, It's Not Bricked

[Anonymous Coward]

Why shouldn't it be proper to say something is a brick if it can't do anything better unless it's fixed?

The word you are looking for is "disabled". As in, "That firmware update disables the drive." As in, "You can re-enable the drive by re-flashing it with a better firmware."

"Bricked" properly means forever.

I doubt people would call my hard drive a brick if I bashed it into bits

Now you are just being pedantic. I agree that if you have rendered it into bits, it is no longer brick-like, and I for one am not prepared to start using "gravel" as a verb to mean "render into bits".

But if you were to knock over an operating PC, and the hard drive heads crashed so hard that the drive fails and can never be used again ever, you have just bricked that drive. See how it works?

No need for anybody to get huffy about somebody calling something that could be fixed a brick.

I don't think "huffy" is the word here. "Brick" as a verb has a specific meaning, and we hate to see that meaning diluted.

It may be a losing battle... we lost the one about "hackers" being a word for cool people. No reporters ever say "crackers" and darn few use adjectives like "black-hat hackers".

"Brick" as a verb is a fun bit of language. Demoting it to mean "disable" sucks the joy from it. But if we have to bicker about it, that sucks the joy too.

Re:If You Can Reflash It, It's Not Bricked

[rrohbeck]

I have to agree. The manufacturer can generally reload the firmware from scratch through a serial or diag port. After all that's what they do in manufacturing. When I worked with disk drives, we had ROMware, firmware (in flash) and Diskware. The ROM is mask programmed and has only boot code that can program the flash ROM, the flash ROM can be reloaded via the disk interface or a serial port (and can't do much more than load a track from disk), and the disk contains the actual code.
Then we got rid of the flash ROM and things became a little more exciting because the code in ROM had to be able to read and write a few sectors reliably - for the entire lifetime of the product [line], including cost reductions.

Re:If You Can Reflash It, It's Not Bricked

[adolf]

The definition of "bricked" depends on the ability of the speaker.

I once bricked a Linksys WRT54G. I say this because I was sure that there was nothing that I, given my knowledge at the time, could ever do to rescue it.

As time went on, I learned more about the problem. Eventually, I soldered a header to the 54G's board and built a JTAG cable, and was able to reflash its firmware more or less directly using my Gentoo desktop's parallel port. Afterward it clearly wasn't a brick anymore, since it was now routing packets just fine. I believe that the precise point at which the device stopped being a brick was between the moment when I finally understood how to repair it, and the final completion of the repair.

So, here's what I think: Given average knowledge and ability, there's lots of things that one might be able to brick. However, with sufficient knowledge and ability, nothing can be bricked.

Re:If You Can Reflash It, It's Not Bricked

[Jugalator]

Why shouldn't it be proper to say something is a brick if it can't do anything better unless it's fixed?

Then it's in need of service. You can call it "broken".

Re:If You Can Reflash It, It's Not Bricked

[Hognoxious]

Bricks can be useful too. E.g. you could build a house or defend yourself from The Man in a riot or revolution.

I'm hearing that in Troy McClure's voice.

Re:If You Can Reflash It, It's Not Bricked

[Gary W. Longsine]

Uhm, no, because the term "bricked" was invented to mean broken beyond repair, not good for anything but a paperweight, like, uhm, a brick. Why on earth did you keep talking after you got that far?

Re:If You Can Reflash It, It's Not Bricked

[billcopc]

You're close, but bricked really just means "you can't fix it, nor can the average layperson". There is such a think as "unbricking".

For example, you might brick a motherboard by flashing it with some hacked BIOS you found on a tweak forum. If you're as dumb as the average forum troll, you're probably not clever, resourceful or brave enough to hotflash your socketed chip on a different board, but an experienced techie could do it.

There's also a pretty large market of "unbricking services", usually just some half-breed with a special cable he bought off of some other wannabe-crook on eBay. He'll reflash your PSP, cell phone or hacked FTA receiver for ten bucks, right from his ornate Honda Civic office.

There are very few cases where a "bricked" device is truly beyond repair by a skilled and equipped technician. If a gadget sells for $100, and your staff tech costs $50/hour, then as long as he can fix more than one unit every two hours (minus S&H and markdown), you fix the gadget. In practice, you end up seeing the same problems over and over, most of them very simple, so your tech might be able to fix 5+ per hour, and I'm being conservative here.

Throwing it in the trash is not a good idea, because if you don't try to fix the broken ones, someone else will buy your trash and do it behind your back. Then you have a bunch of poorly-repaired devices bearing your brand name, floating around generating forum posts and hate mail all over the web. The cost of junking returns can be greater than the cost of repairing them.

Re:If You Can Reflash It, It's Not Bricked

[LateArthurDent]

The definition of "bricked" depends on the ability of the speaker.

Not really. It's bricked if it cannot be repaired by non-physical means. If you have to open the device up and start soldering leads, the device is bricked. You're just capable of unbricking it.

The term "unbrick" has been around even in the old days when "brick" was being used correctly. I think that may have been what caused the new definition to come about. People would go into forums and see things like, "I've bricked my router, anyone know if it's possible to unbrick it?" The people asking the question were looking for hardware solutions such as the one you've accomplished, but the ones new to the terminology started inferring the meaning of the term "brick" as "currently not functioning" since it was obviously possible to bring them back to life in some cases.

If Seagate keeps this up

[Killer Orca]

They'll be no different from other HDD manufacturers. I recently got a Seagate external because the price and 5-year warranty were a great combo. I hear they are going to lower the warranty period and now these problems; makes me wonder where I will be able to buy reliable drives in the future.

Re:If Seagate keeps this up

[afidel]

I wonder if this is coming from the Seagate side of the house or the Maxtor side? This sure seems a LOT more like something the old Maxtor would have done than the enterprise provider of choice Seagate.

Re:If Seagate keeps this up

[rrohbeck]

http://www.theregister.co.uk/2006/05/23/seagate_6000_job_cuts/

http://articles.latimes.com/2006/may/23/business/fi-maxtor23

This was practically all of Maxtor US, Longmont and Milpitas, including what was left of Quantum HDD, except Shrewsbury AFAIK.

Re:If Seagate keeps this up

[diamondsw]

The remainder will largely be made up of Maxtor's Asia-Pacific manufacturing workers, Seagate said.

The drives with bad firmware came out of operations in Thailand, if I recall. This could still easily be Maxtor...

Re:Upgrading and flashing 'untested' technology?

[fuzzyfuzzyfungus]

Arguably, when version "latest and greatest" -1 has a cool bug that causes it to permanently and (without hardware intervention) irrecoverably brick itself for no obvious reason, applying version "latest and greatest", at the manufacturer's recommendation, is a fairly reasonable thing to do.

Anybody who thinks that RAID=backup is going to learn an exciting lesson; but I don't think we can, in fairness, blame people for applying the update.

3rd time in the last few months?

[tiffany98121]

what is happening with seagate? did they downsize their qa staff or something?

Re:Huh....

[sjames]

Normally, they wouldn't, but these drives already had issues. Seagate recommended updating the firmware (with their 'handy' windows only updater). Unfortunately, that made the problem worse.

As the owner of 4 of the 1 TB drives...

[ShadowBlasko]

I would like to know where the hell the firmware update IS? I have opened a ticket with Seagate for each drive. Followed the directions (which were linked to here last week) in detail, and I have heard back NOTHING.

Not even an acknowledgment that they have looked at my tickets. I got a "your ticket was created" email, and that is it.

Seagate is getting very close to losing a lot of customers.

Re:As the owner of 4 of the 1 TB drives...

[Banichi]

Obviously, they are using their own product.

Alternate headline

[The Clockwork Troll]

Barracuda Flounders

Oh what a long, long fall.

[jd]

Once upon a great while back, Seagate was one of the première names in hard disk technology. These days, the only press I'm seeing them get is bad firmware, questionable reliability, etc. They've been around longer than Microsoft, they really have no excuse at this point for not even testing their bugfixes on their own hardware. It's not like they even have to test third-party stuff.

What leads to this sort of decline?

Re:Oh what a long, long fall.

[tono]

I don't know, you should ask IBM the same thing. Deathstar?

Re:customer support offline too ?

[Anonymous Coward]

Over 48 hours now .... Starting to think I better not hold my breath

I agree, but good job for holding your breath for so long :P

Not Windows.

[antdude]

The firmware updater uses FreeDOS from a CD image (ISO). Users had to burn it to a CD and boot from it. Here's an example when I tried it (first release that crashed while upgrading -- did not brick for people and me) under VMware to see if my CD booted: http://img403.imageshack.us/img403/7128/screenshotsa7.gif from Sunday night. I didn't bother to try the second one because that one totally bricked 500 GB HDDs which I have!

bad Seagate, bad!

[Eil]

I work for a web hosting company and we get these drives by the case. I couldn't guess how many are deployed throughout the datacenter but on some of our backup servers alone I've calculated that I have almost 100 drives that need the firmware update. Thankfully none of the disks on the systems that I admin have shown problems yet, but we try to run a quality operation and that includes preventive maintenance wherever possible.

I was all set to update the firmware on these when one of our guys found that the update rendered unusable 8 of the 8 drives he upgraded the day before Seagate pulled the update. We currently have some massive amount of Western Digital 500GB and 750GB disks on rush order as a result of this debacle. It wouldn't surprise me if management tells us to swap the Seagate disks for the WDs and decides to just sell the whole lot of Seagate disks off in bulk as defective. It would be cheaper than paying people to update each one by hand.

Before this, Seagate used to mean "quality" in my opinion as their failure rate seemed to be lower than the competition and their 5-year warranty was unmatched. For the average home user, this situation is a headache. For people running datacenters filled with these disks, it's an outright fiasco.

Re:bad Seagate, bad!

[seifried]

"web hosting company" - lots of cheap servers with lots of disk (how else do you sell 10gig VPS servers? It's not like these machines have high IO requirements typically.

Re:bad Seagate, bad!

[fuzzyfuzzyfungus]

Depends on the application; but probably a lot. With SATA drives natively supported by SAS controllers, and substantially larger and cheaper than SAS, they are quite attractive for anything that doesn't need very high speed.

Seagate's forum is on fire from this mess.

[antdude]

Go here http://forums.seagate.com/stx/board?board.id=ata_drives to see the angry users and posts in Seagate's official forum. Most of us are pretty angry and upset. Definitely read this super long thread: http://forums.seagate.com/stx/board/message?board.id=ata_drives&message.id=6272 (42 pages).

I find it ironic that our HDDs are about to be bricked EITHER way (on its own) or with the pulled firmware updater (released twice already too; first one crashed with memory dumps and stuff for everyone; second one bricks 500 GB models).

FYI, http://support.seagate.com/firmware/MooseDT-32MB-SD1A.ISO was the ISO file that was released (404 error now due to brickings) according to my download history. Seagate needs to get the next one right!

Re:Seagate's forum is on fire from this mess.

[lord_rob the only on]

Deleted already :(

Slashdotted ?

Not bricked!

[ZorkZero]

It's not bricked if you can fix it without modifying the hardware. It's a nice term -- stop destroying it.

scandal nomenclature

[adavies42]

is this seagategate?

I talked with A/S 10 minutes ago

[digirave]

I talked with A/S 10 minutes ago

After talking with Seagate A/S a few days ago and told I needed to update my firmware and sent an email on how to update, no fireware was downloadable from the links in the email provided.

Annoyed I talked to Seagate A/S again today, it seems I do not need a firmware upgrade anymore, and only some of the hard drives made in Taiwan between some date seem to be defective and updating firmware in non-defective drives seems to be causing problems. Hence they removed all links to firmware. Since they are not 100% sure of what I mentioned above yet, they told me they are going to update their site and call me back when things get finalized next week.

Meta-suggestion for eds

[Brett Buck]

Can we, for God's sake, just permanently ban the use of the word "brick" or "bricked" in the summaries. I have yet to see it used correctly.

        Brett

Re:Meta-suggestion for eds

[Brett Buck]

Anybody who uses "brick" as a verb should be bricked - with a brick.

          Brett

THE FACTS

[maxtorman]

I work for Seagate. I was there when the fit hit the shan, and I saw everything going in internally, as well as externally.
I really love my job, so please excuse the sock-puppet nature that creating a brand new account and claiming to be an authority on the subject I must seem to be. But I am a geek, and I really think you all need to know the true story behind the scenes.

This whole thing started with the 1.5 Terabyte drives. It had a stuttering issue, which at first we all thought was a simple bad implementation of SATA on common chipsets. Seagate engineers promptly jumped in and worked to try to duplicate the issue and prove where the problem was. This wasn't a massive rush as 1.5tb drives are what? 5% of the drives on the market. When it became obvious that the issue was more widespread, they buckled down and put out a couple of firmware revisions to fix it.

Now, in the 1.5tb drives, there are 2 main revisions. the the product line that gets the CC* firmware, and the line that gets the SD* firmware. They came out with firmware CC1H and SD1A to fix these issues and started issuing them.

But, seagate has always been restrictive of handing out their firmware, so such updates required calling in with your serial so that the people who had access to hand out the firmware could check a) model, b) part number, and c) current firmware just to make absolutely sure that they were giving the right firmware out. This has been a procedre that has worked for YEARS up until now.

Then the bricking issue came to their attention. It took so long because it's an issue that's hard to track down - pretty much the journal or log space in the firmware is written to if certain events occur. IF the drive is powered down when there are 320 entries in this journal or log, then when it is powered back up, the drive errors out on init and won't boot properly - to the point that it won't even report it's information to the BIOS.

This is a rare, but still obviously bad issue. Up until now, we all figured it was just some standard type of failure, as it was such a rare event, so we'd RMA the drives.

So, for whatever reason, mid management started freaking out (as it could be a liability for seagate, I suspect - ontop of the already potentially liable issue of the stuttering problem causing drives to fail in RAIDs). So, they pushed the release of the SD1A firmware to the general public. They took a few days to 'test', though it was mostly just including some code in the batch file that kicks off the firmware updater, to check that it is a BRINKS drive, and the proper model number. Then it was kicked out to the public.

Please understand, this firmware had to go through five different checks to make sure it applies to the specific conditions to qualify sending to a customer, before now. 5 chances for us to go your drive needs the other (or none) firmware update. Suddenly, it's down to ONE check, and even that was more designed for a contingency just incase the wrong firmware was sent out.

Of course, it starts bricking drives.

Right now, the engineers are crapping themselves, the firmware's been pulled, the support agents are told to say "The firmware will be released soon" and no real procedure to fix this issue is in place. Our phones are flooded so bad that it locks the system up when there are too many calls in queue, and emails are coming in at hundreds an hour.

We simply cannot keep up.

The good news is, the chance of your drive simply not spinning up one day is very low. And for those of you who flashed the wrong firmware - be patient. It's not bricked, just unable to write data to the platters properly. When they have a *GOOD* firmware out, a new flash should un-brick the drives. If not, flashing it back to SD15 should make it work again.

Seagate really pushes the idea of being open and honest as much as we can without being sued to hell. They let agents make choices and use their skills instead of scripting us to death. They worked hard to bring their support back t

Re:THE FACTS

[rossz]

There's a lesson to be learned here. DON'T FARKING LET MIDDLE MANAGEMENT BYPASS YOUR TRIED AND TRUE TEST/RELEASE PROCEDURE. Yes, the initial problem was bad, but the rush to get a fix out made it much much worse. Upper management is at fault here for allowing middle management pencil pushing idiots to do this to the company's reputation. Procedures are in place for a damn good reason.

Re:THE FACTS

[noidentity]

Maybe I'm naive, but why can't the firmware updater first check that it is being applied to the proper drive hardware? Surely it has a way to ask the drive exactly what hardware it has in it. But routers seem to be just as stupid, accepting whatever file you send to it without checking anything, so what do I know.

Re:THE FACTS

[sa1lnr]

"Right now, the engineers are crapping themselves"

Shitting bricks no doubt. ;)

Re:THE FACTS

[maxtorman]

Thank you! I did lose some very close coworkers in the last round of layoffs. i think that's another reason this is hitting so hard - we are at our highest support volume, low sales, and rough stock. I believe Seagate realizes they need to keep every customer possible. But pressure can and in this case has lead to some bad decisions by management to ignore their engineers' recommendations... but even then, this issue was hard to see coming as this firmware had a month of field proof that it works... they just didn't realize that a small group of trained people experienced and acting as an absolute filter of who gets what firmware cannot be replaced by a 10 year old batch file.

Re:THE FACTS

[maxtorman]

You say that now, but you have to admit, with such screaming and carpet-clawing that went on about the 1.5Tb issue, some of the fault rests on the mob mentality pushing Seagate management to get a fix out ASAP for an issue recently proven. I'm not saying it's okay - but the exact same situation that can force a large and lumbering company to move faster, can force management to push really hard and cause quality systems to break down. You can whip the bull to get it to run, but you may just cause it to run right off the cliff. :)

Re:THE FACTS

[maxtorman]

It was never designed to be a public release. The script checks two things.. to make sure it's a BRINKS or a MOOSE drive, and to check the model number. If you get the firmware from the torrents (it's out there) and tear it apart with uniextract, you can see the batch file and what it checks for. It's a program that was built back in the 90's and used ever since! You remove those 2 checks, and it'll happily flash that IBM or Western Digital drive with the seagate firmware as well.

Re:THE FACTS

[maxtorman]

The engineers may be shitting BRINKS.. but management is shitting MOOSE!

Re:THE FACTS

[maxtorman]

As I've noted below, it was an emergency release that shouldn't have been, and was never designed for release to the general public.

They should have redesigned the delivery system, but there was too much public pressure on them to get a fox out *now*...

But then again, it was somewhat their own damn fault - if they had just came out an explained the details of the issue to everyone instead of keeping it in-house, people would have realized quickly it wasn't as dangerous a situation as it seems at first glance. Just inconvenient to the few who run into it more then anything. But the ambulance chasing lawyers smelled blood during the 1.5Tb issue and forced management into a hole.

Re:THE FACTS

[maxtorman]

Wait a few days - Seagate will have in place a procedure to get bricked drives due to a bad firmware, in place. Once they do, you should just be able to send them the drive and it'll be reflashed with good firmware and sent back. I can't say this for absolute certain, but that's what they're telling us now.

If you have confidential data on the drive, you have two options:

a) if you send it in for a reflash, there will be a tech who flashes the drive using a serial interface, and then verifies good read/writes to the data. But he's likely unbricking a hundred drives a week, and doesn't care about what's on the drive unless he happens to maybe notice a folder when he does he read/write test labled "OMG HUGE AMOUNT OF CHILD PORNOGRAPHY". I can't even say that a person will even be doing the R/W test - but there is that chance.

or b) RMA your drive. The first thing that happens once the drive passes a visual inspection (verifying that the warranty is still valid and the drive hasn't been user-damaged physically) is the drive is thrown on a text machine. if the drive passes the physical tests, then it's firmware is flashed and the diag machine goes through a 7 pass zero-random-zero-random cycle that destroys any and all data on the drive. This not only ensures data wipe, but also helps diagnose any read/write errors on the drive. If you RMA the drive, it's not even hooked up to a human-accessable 'computer' (just diag equipment) until the next customer who received the drive as a refurb, puts it in their computer - at which point it should be so blank, not even the government could recover data from it using the most advanced tech that we know about.

Call back and push your way up to a supervisor, and see what they offer you on friday, since the agent sent you the wrong firmware.

Re:THE FACTS

[maxtorman]

I've worked tech support for a while now, and that's something that's struck me working for seagate.
Just about everyone there actually.. well, -cares- about what they do. The atmosphere inside is completely different then say, dell or cox or AT&T (from what I hear - I've never worked at any of those places, but many of my coworkers have).

The management lets support agents have license to use their experience and skills in fixing issues, instead of reading from a script. They don't heavy-hand them, and they are more concerned in fixing an issue in as few of calls as possible, then caring about the number of calls taken a day or keeping the calls as short as possible. It means that tech support can actually *fix* the problem instead of trying to get people off the phones as fast as possible.

And thanks, always glad to hear the positive feedback.

Re:THE FACTS

[omidaladini]

Our phones are flooded so bad that it locks the system up when there are too many calls in queue

You should look for a firmware update for your phones too!

Re:THE FACTS

[maxtorman]

Thank you! I wish this information would have been public and I didn't have to create a new account to avoid being fired for releasing 'confidential information' - but what can you do with jerkoff lawyers tearing at your corporate heels already?

Now, to your questions!

1) It keeps changing because the scope of the issue keeps changing. I'm pretty sure it's a range of drives within the familys noted in the KB article - but also, there are some external drives affected because they contain an internal drive with the problem, that aren't on the article yet. Your best bet would be to compare your drive to the list of models, and then wait a little while.. around friday, I *think* they should have most issues sorted out and the information accurate. But I can't promise anything.

2) That could very well be it. I'm not privy to the nitty-gritty details, as engineering clammed up pretty quickly - I'm just a geek enough to understand what I hear in passing or the few technical details I came across when I go looking for information. But the mysterious death log being a SMART self-test log would absolutely make sense, and is consistent with what I'm hearing.

3) Unofficially, I've seen more then just the 1.5Tb drives display symptoms similiar to the stuttering issue, but none so blatent or as impacting as it is in the 1.5Tb drives.

As far as the firmware fixing both the stuttering issue and the unresponsive-drive issue, yes. The changes for the stuttering issue was made in CC1H and SD1A firmwares. Any firmware equal or more recent then those two, will have the fix for both issues.

4) I have no idea. SMART characteristics can vary from part number to part number - or even sometimes drive-to-drive; so what is 'out of tolerances' for one part number could be just fine for a different p/n (even though they are the same model number).

Re:THE FACTS

[maxtorman]

First, let me apologize, I'm gong to withhold employment details such as tenure and experience mostly due to the fact that many of us at Seagate (including some in management) are Slashdot regulars.

That said, I really do enjoy my time at Seagate, and it has been an absolutely wonderful company to work for.

As far as "BRINKS" "MOOSE" "GALAXY" etc.. are concerned, they are pretty much the internal development names of the drive family. There can be overlap, but most "BRINKS" drives are 7200.11, I believe, while "MOOSE" drives are almost all 7200.10, and "GALAXY" drives are 7200.9. Generally, those names don't make it out into public, but if you were to tear into the SD1A firmware, you'll notice that it looks for the "BRINKS" drive before it flashes the firmware to the drive. There can be different internal names for different revisions of the drive itself, but generaly they stick to one revision per family - a new internal name would only be used for a MAJOR revision on the drive.

I don't have my documentation handy, but I'll look that up later in the week and try to give you a better answer.

Finally, thank you for your kind comments.

Re:THE FACTS

[maxtorman]

1 word: Lawsuits. if they gave incorrect information, it could open them up for liability if people acted o that information. When a business' data could be worth millions, one slip-up could cost them dearly. The only reason this firmware isn't such an issue, because of the disclaimers allover the place when you flash a drive.

yes, the 1.5Tb drives both stutter and are at risk of bricking due to the journal issue. The Stuttering issue is fairly recent and mostly runs in the 1.5tb drives - but the journal issue is older and exists across many 7200.11 drives. ES2 drives and Diamondmax drives.

SD1A fixes both of these problems in the 1.5Tb drives.

Re:THE FACTS

[bipbop]

Hahaha. That's so true, but the whole point of middle management is to make bad engineering decisions for political (read: "stupid") reasons, because the people who know enough wouldn't, and the people above them think it'll save money to have a political layer (that is, a stupidity) like that inbetween. The people at the top can't make those decisions directly, because when they screw up, someone has to take the blame. So these people are pushed to make "the hard decisions", then get blamed for it when they turned out to be "the stupid decisions". So, execs keep their noses clean, and engineers get bypassed. Of course, if middle management has their way, they push the responsibility on down as far as they can, but that's irrelevant from the perspective of the execs who put the stupidity layer in place.

Sigh.

Re:THE FACTS

[swilver]

It's a shame seagate is/was so secretive about the actual problems with the 1.5 TB drives. I specifically went looking on your website to get information about the problem so I could assess how bad the problem was and whether or not we'd be affected (yes, we run Linux). The only thing I found was mass censorship on the forums by moderators, and any discussion about the problem was locked away. Since I couldn't find out *easily* whether or not we'd be affected, I opted to buy a set of drives from another manufacturer.

Re:Huh....

[keeboo]

If the problem was the uncertainty of your data's integrity due to a firmware bug, then your problem is solved.

Now you are sure.

Re:Fail

[Farmer Tim]

Which means they've proved its never too late to have a happy childhood.

Re:Upgrading and flashing 'untested' technology?

[Anonymous Coward]

This whole debacle is *exactly* the reason I prefer software RAID to hardware RAID. I deliberately make my RAID arrays with disks manufactured at different times and by different brands, and when possible on different controllers as well -- having a totally homogenous RAID array has always struck me as dangerous.

This isn't anything to do with SATA per se

[igb]

This problem isn't anything to do with the drives being SATA versus anything else, and the FC lobby shouldn't get too smug. Some (with hindsight, at least) bad engineering decisions got taken in a complex product, and the result was that the product got into trouble. All disks are a mixtures of electronics, mechanicals and firmware, and although this happened today on a SATA drive it could happen equally well tomorrow on an FC drive. The answer to your question is ``anyone who wants to be power, space and money efficient''. There are products now shipping in volume --- Pillar, Sun's Fishworks boxes spring to mind --- where the performance of SATA is brought up to FC standards for many workloads (your mileage may vary, objects may be closer than they appear, etc) by the application of appropriate filesystem structures, battery-backed RAM, flash, SSD, etc, etc. There are, before anyone jumps in, workloads where nothing this side of a gazillion independent spindles of 15000rpm FC is going to work. But conversely, there are other workloads where performance isn't as much of an issue as space and power density (backup, for example) or where capacity causes the business far more issues than performance. I've got a Pillar stuffed full of SATA. There's FC available for people who need the performance bump, but I don't: my application workload saturates on other factors long before it maxes out the NAS server, and even if that were not the case, the value to my business of small deltas of performance (and the difference between FC and SATA is a lot smaller than people make out) is less than the massive difference in price. In general terms, SATA today is where FC was five years ago, and even if you end up short stroking it it's _still_ cheaper than FC. My Pillar allows me to effectively short-stroke SATA for performance and use the residue for non-critical data, which is nice, of course. Performance isn't everything, as otherwise we'd all be going to the supermarket in Formula 1 cars. There are other criteria, and SATA may be appropriate for your business, depending on what your business is. And slightly more controversially, I'm suspicious of admins who claim their application needs the latest bleeding edge of a component --- disks --- which is on a slow development curve for performance. The speed of disks scales, loosely, at the square root of the capacity times any increase in rotational speed, but seek times have only improved by a factor of four over the twenty years I've been running fileservers for. If you're seek bound, you've got deeper problems that disk technology won't always help you with. ian

Bricked Threshold

[raehl]

A brick's value is the cost of creating a brick to replace it.

So if it is less expensive to throw something out and buy a new one than it is to repair it, it's bricked.

Re:Bricked Threshold

[Oak1]

Um, no. A "functional electronic device" is one for which its cost of repair = $0.

A thank-you! (and some questions)

[Wrexs0ul]

Maxtorman, I'd mod you up if I had the points. Your comments are the first ones to alleviate a very significant knot that formed in my stomach after reading this. I'm still a little concerned though, and have some questions at the bottom I hope you could answer.

I'm a little late to the party because I only use these only for non-critical stuff like home office and family PC's, but the prospect of having all my drives inevitably die really scares me. I've bought 18 drives (ST31000340AS and ST3500320AS all w/ FW SD15) in the last half of 2008 that sound like they match those reported to fail on the forum.

Funny enough I was complaining to my vendor about 4 drives that had to be replaced because they died within a month of use. Thought it was a bad batch they were pawning-off on customers, but I still trusted the Seagate brand.

So, my questions:

1) Is this definitely fixable in firmware? Should I be buying new drives right now?

2) What are the honest chances of a drive dying before a working firmware patch? My critical stuff is in RAID5 so I can always rebuild, but the gaming rig is RAID0, and off-site stuff like mom's media PC is only a single drive.

I appreciate your comments. Good to know there's a guru in the Slashdot community :)

-Matt

Re:A thank-you! (and some questions)

[maxtorman]

I'll answer your questions to the best of my ability, and as honestly as I can! I'm no statistician, but the 'drive becoming inaccessable at boot-up' is pretty much a very slim chance - but when you have 10 million drives in the field, it does happen. The conditions have to be just right - you have to reboot just after the drive writes the 320th log file to the firmware space of the drive. this is a log file that's written only occasionally, usually when there are bad sectors, missed writes, etc... might happen every few days on a computer in a nin-RAID home use situation.. and if that log file is written even one time after the magic #320, it rolls over the oldest file kept on the drive and there's no issue. It'll only stop responding IF the drive is powered up with log file #320 being the latest one written... a perfect storm situation. IF this is the case, then seagate is trying to put in place a procedure where you can simply ship them the drive, they hook it up to a serial controller, and re-flashed with the fixed firmware. That's all it takes to restore the drive to operation! As for buying new drives, that's up to you. None of the CC firmware drives were affected - only the SD firmware drives. I'd wait until later in the week, maybe next week, until they have a known working and properly proven firmware update. If you were to have flashed the drives with the 'bad' firmware - it would disable any read/write functions to the drive, but the drive would still be accessible in BIOS and a very good chance that flashing it back to a previous SD formware (or up to the yet to be released proven firmware) would make it all better. Oh, and RAID0 scares me by it's very nature... not an 'if' but 'when' the RAID 0 craps out and all data is lost - but I'm a bit jaded from too much tech support! :)

Re:A thank-you! (and some questions)

[maxtorman]

Just remember, everything dies
I sincerely smile in relief every time I hear someone other then myself say that phrase. It's a sign of someone who truely 'gets it' as far as hardware is concerned.

Thing is, I know Seagate really does try to push for high manufacturing standards (for example, did you know that every last Refurb drive *must* go through the full new-drive qualification before it's sent out? - something only a percentage of actual new drives have to go through because it's time consuming).

But WD and Hitachi and most other HDD companies have only slightly worse failure rates, negligible difference in failures really in the grand scheme of things. No matter if you go with WD, Seagate, Hitachi, IBM.... whatever. If there isn't an outstanding issue, then you're really looking at about the same chance of failure no matter which you buy.

The real decision is more 'how easy will it be to replace this if it fails?'

Funny though, I have in my system a 250Gb WD drive, that used to be in an external enclosure, until the USB interface controller died and I ripped the drive out - and now, after about a year of use, the internal drive's starting to fail... I rolled the crapshoot on this drive it seems. :)

Re:A thank-you! (and some questions)

[maxtorman]

I've been a denizen of slashdot for many years - I just wish all these mod points were on my main account! :)
But it is nice to be able to contribute knowledge and experience back to the community for once.

Thing is, this issue -is- rare. But it manifests itself in a way that's hard to distinguish from a normal drive failure. (suddenly no detection in the BIOS; spins up but never is seen on the computer - this can happen for a dozen reasons including a loose or bad cable, physical drive failure, etc) so a whole lot of 'me toos' doesn't mean much. When this issue potentially affects millions of drives, a >.01 chance of failure still adds up to thousands of drives.

I'll say this. There is no *more* chance of it dying on your next boot up then here has been of it dying your last 3000 boot ups.

Many people have noted that drives do tend to fail in batches. If there is some slight manufacturing error that causes a drive to fail, it tends to also exist in other drives from that same lot, the closer to the same manufacture time, the more likely it is to also fail. I tend to agree with them - it would make sense to me that if a photo lithography machine got bumped or a slight bad mix in the emulsion for etching the PCBs were present, it would affect all drives around the same build time.

Who knows.

Re:A thank-you! (and some questions)

[maxtorman]

As far as I know, if your drive has the CC1G, CC1H, CC1J or any of the CC firmwares really, it is completely unaffected by this issue.
However, it may need an update if you experience 'stuttering' (the drive pausing for more then a few seconds during data transfer). The CC1H and CC1J firmwares are *fine* and will absolutely not brick your drive.

I'd still wait a little while though - support is overwhelmed and mistakes are being made as noone is used to these changes. Once everyone gets a routine down (once there -is- a routine at all), they'll be better able to help reliably.

Re:Huh....

[Dibblah]

It's not "moving the head to prevent wear". It's SMART data gathering. smartctl will soon sort you out. However, I would personally not recommend it.

smartctl --smart=on --offlineauto=off

Re:Pwnt.

[pipatron]

Raid has never been a backup. A backup is something stored outside of the running set. That way you can restore the data if your running system would, you know, break down.

Re:Legal pressure is counterproductive

[maxtorman]

I absolutely agree with you.
If we had been allowed proper development and proving time, this may not have been an issue at all.
But the moment Seagate even admitted there was an firmware issue with the 1.5Tb drives, lawyers began recruiting for class action suits.

Disgusting ambulance chasing fecal sniffers.

I saw this coming

[Britz]

Since I had not heard of massive numbers of Seagate drives failing I already suspected that this is a rare occasion in which the drives would not spin up. I was wondering why Seagate announced this bug berfore they have a fix ready. Looks like they announced at very early. Maybe they also should have put more emphasis on the fact that it is a very rare bug.

It was announced. And people were freaking out about a bug from Seagate without a fix ready. What happens when customers freak? Right: Tons of pressure on getting a fix ASAP. I thought the chance of a bad firmware would be much higher with that much pressure and upgrading the drives would pose a more severe risk than just doing nothing. So I think I will keep doing that for another couple days at least.

The sad part about that story is that companies will be more reluctant to be open about bugs in the future.

firmware update for drive = fail

[stiller]

Ok, maybe it's just me, but who the hell updates drive firmware anyway? Just because I'm a techie, doesn't mean I am suddenly willing to do more work than other customers.
Do you think a single consumer out there goes through the trouble of updating their drive firmware? (unless there's an automatic procedure in place, like probably mac and some windows manufacturers have)

To me, any drive which requires an firmware update to function (not just perform better) after purchase, is a failed product and I would surely hesitate to buy another ever again.

I used to buy Seagate drives in pretty large numbers for some of my datacenter activities and every time a drive locked up for some reason, I insisted on a new drive through EMA. Had Seagate refused, they would have taken away a large chunk of their added value, to me. I would probably never buy another drive from them again.

Re:Pwnt.

[Kokuyo]

It is the old High Availability versus Disaster Recovery question. Two completely different things aimed at two completely different problems.

The first is to make sure that your system remains available as long as possible even if some of your hardware goes belly-up. The latter is for when your DATA goes belly-up.

Re:Fail

[drinkypoo]

Seagate never played Whack-a-mole growing up.

The game you played with a server full of seagate drives "growing up" is that if it was off long enough to cool down it was a virtual certainty that at least one of those drives wouldn't spin up. The odds of another disk developing stiction while you were taking the first one out of the case and whacking it with a screwdriver approached infinity as the number of disks became large enough to make the machine or enclosure heavy. Is that enough like whack-a-mole for you?

On the plus side, most of those drives had really huge filters in them. I had a 40MB RLL disk that I opened, de-stuck, and closed with nary a data error. I just did it in my bedroom (huh huh) with no dust or static control in the environment whatsoever. That drive eventually burned a trace right off of its control board, and then burned off the jumper wire I replaced it with, but that was over a year later and I'm quite sure one had nothing to do with the other.

I don't know how Seagate actually got its good name. I've used a lot of their disks, since way back when, and I just don't see it. I used to like Maxtor in spite of the noise, but WD has pretty much been my best friend all along. Today I will hardly buy anything else.

Re:Huh....

[magarity]

it starts to randomly move the head to prevent wear
 
I have to ask what wear you think is being prevented by INDUCING activity. If the head arm DIDN'T move, that would be *preventing* wear compared to moving it. But "reducing" wear?

Re:Trashing their name

[hairyfeet]

Well unfortunately you will get the occasional clunker no matter who you go with. Any product cranked out in those kinds of numbers is bound to have at least small bad batches no matter how good the QA. Now as for ExcelStor, when I said small I was talking 80-160Gb range. I have built plenty of office machines with ExcelStor drives in that range and they are quite popular. They are VERY quiet, which if you are building a machine which is going to be on the desktop is a plus.

With WD I've always had those fail really quick(and thus be under warranty) or not at all. I've got a drawer filled with 40Gb WD IDE drives from upgrades that I'll need to figure out what to do with. Samsung I've had good luck with on very large capacity, as well as Hitachi. But if you are wanting a 1.5TB you are pretty much stuck in Seagate land-they are pretty much the only game in town. Might I suggest either the Samsung I linked to earlier or perhaps one of these, two in a RAID 0 if you really must get above the 1TB range? Because until Seagate gets their collective shit together I would be afraid if picking up one of their drives ATM. My WDs may only be 500GB each(and that strikes me as funny as hell that I can say "only" when my first HDD was 2GB) but they are VERY quiet and give me more space than I will ever need.

A victims point of view

[jupp201]

I am one of the victims and your report confirmed all the problems which I expected to occur inside your company. I previously worked with an electronic giant and the problems are just too similar.

The catastrophic problems which Seagate is facing now could have been prevented - if there would have been one single person in customer service who would have cared and pushed the issue, which was known for months, up to the right people. A little googling some months ago would have proven that this issue is far bigger than a "one time" incident.

After all it doesn't happen every day that Data Recovery companies announce with joy that they are able to handle widespread 7200.11 firmware problems. Or that the two major companies which provide recovery solutions race for being the first to have a two click solution for this cash cow.

Data recovery companies were flooded with drives. They figured out an easy way to fix the firmware and kept it secret. They made a great profit, charging prices as if it was a hardware failure.

Seagate Datarecovery did the same by quoting up to 1800 USD for a 10 minute fix. Although I am sure that they were the only ones not aware of the easy fix.

The problem with the undetectable bios drives really isn't new. Your customer service knew it for a long time, but they are paid so little and probably have such strict procedures that they don't care about Seagates customers and no one dared to report the drive failures as a major incident. Everyone shut up about it and the people which are responsible and do care only learned about it months later when (or shortly before) it got out to the press.

Seagate had months of time to fix it. Two months ago when my drive broke, there was already plenty of information about the problem on the net. The only one who would deny any problem was Seagate.

I warned your board moderator of the disaster which will strike Seagate months ago. I tried to show him that these were not normal failure rates but the poorly paid guy didn't care.

The email support who takes two weeks to respond, and the phone and live support were just as ignorant.

There were people reporting how 4 out of 6 drives broke within weeks, and Seagate would only respond that such failure rates are normal.

People on the Seagate boards were constantly reporting the problem, but your board moderator shut them up. Threads where getting deleted and locked, including a big thread where the community was working on a fix. The reason, according to Seagate, was that it added nothing to the community.

The board moderator would consistently tell everyone that there is no known problem with the drive - the same message as your customer service.

It went as far as blocking links in private messages to a posting on another board which could help the victims. So how could Seagate expect from those people now to actually believe that the company cares?

The posting on the new board had within a short time 10.000 views. That's when things started to get out of hand for Seagate.

People were pissed off for months about Seagate. Everyone knew that the firmware was broken, but the company denied any problems. We knew that it is not that difficult to recover the data if you have the tools and knowhow, but the company wouldn't give any assistance. Many would have accepted the fate if the drive would truly be broken. But not if it is inaccessible because of a firmware bug which makes every single drive a -clicking- time bomb.

People everywhere were calling Seagate harddrives junk drives which are so unreliable that they will never buy them again.

So I, as many others, went on to warn every single person we knew about the problem with Seagate drives. The hilarious/sad thing is that before, I would recommend Seagate to everyone I knew. If someone would ask me which drive to buy I would reply with no doubt: Seagate.

This could have been prevented if Seagate would have acknowledged the problem much earlier. I wasted day after day,

Why not include base firmware on a rom?

[jollyreaper]

I never understood why equipment capable of being flash-updated by users does not include the 1.0 drivers as a ROM onboard the device. This way if you completely and utterly bork the flashing, you can reset a jumper, press a recessed button with a paperclip, so SOMETHING that will cause the EPROM to be reflashed from the known good ROM. "Hey, here's baseline firmware again, people. Let's try this again."

The only possible explanation I can think of for not doing this is that the known-good ROM would add another half-cent to the manufacturing process and we know how manufacturers watch their pennies.

Re:Pwnt.

[FredFredrickson]

I do a lot of video and audio work- so with about 350 gb worth of data that is in jeapordy, I find myself doing multiple backups.

I work in computer repair, and you would be surprised how many people lose all their data, despite having a backup. No single backup is a good idea. Multiple backups are ALWAYS a good idea. So here's what I do on a small budget with a ton of data:

NOTE: Backups need to be painless. If they're not, I'll never do them. Be honest with yourself. Making a note to sit down and burn DVDs every sunday will NEVER happen. It might happen a few times, but not *just* before you need it. Plan around convenience, for your data's sake.

1. Backup number 1 is basic. I have another desktop on my network, with a raid mirror and a network share. I run a backup regularly. Incremental backups to seperate files works nicely to conserve space. CHECK YOUR BACKUPS REGULARLY. Nothing's worse than realizing your backup has been bunk for 3 weeks, and then have a system go down.

2. My second backup is in case of major disasters that burn down my house or something similar. I use MOZY to do online backups. Now, with 350gb, it's not a very fast thing. But because it's incremental, eventually you get past the hump, and it only has bits and peices to do to keep up. A major point for me on online backups was security. My life is in this data, including SSNs, Pictures, Embarrassingly bad poetry, and so on, so I was a bit nervous about starting an online backup.

Mozy offers local encryption with a private key BEFORE uploading, although it's not neccessary. Mozy states in their terms, if you decide to go with the private key, and you lose that key- they can't help you. I like the sound of that.

If you don't use your own private encryption, they say they don't look at your data, but you know how companies fold to government wishes. I'm not hiding anything illegal, but I do want my privacy.

Mozy is $5/month. With my connection (768kbps up) it's taking months to finish the first upload. But after that's done, I'll have a bit more comfort, and for $5, it's not worth not doing. They allow you to throttle the bandwidth during different times of the day, so I have it upload during the night and while I'm at work, then have it stop while I'm home, so I have the internet connection all to myself!

*Note: I do not work for, nor am I affiliated with mozy.