Slashdot Mirror


Flawed AMD Chip Can Lead To Data Corruption

Brandonski writes "Apparently AMD allowed some flawed chips to slip through their detection grid. The problem affects only a small number of chips and only single core 2.6 and 2.8 GHz CPUs." From the article: "It is believed that the glitch is triggered when the affected chip's FPU is made to loop through a series of memory-fetch, multiplication and addition operations without any condition checks on the result of the calculations. The loop has to run over and over again for long enough to cause localized heating which together with high ambient temperatures could combine to cause the result of the operation to be recorded incorrectly, leading to data corruption."

47 of 203 comments (clear)

  1. I dub thee by Anonymous Coward · · Score: 2, Funny

    Fetch Div, son of Eff Div, Heir to Count Zero, and Lord of a new generation of digital serfs, soon to be labled as having "emotional problems."

  2. Re:What? by qbwiz · · Score: 2, Interesting

    Generally, chips aren't supposed to have localized heating problems. Either it should all have a problem, or none of it should.

    --
    Ewige Blumenkraft.
  3. I Have an AMD CPU by ozmanjusri · · Score: 5, Funny

    Hey, I have an AMD 2.8Ghz. Maybe I should stop refresðN9'óI]öR9ù¥Î6ýPoe}+èa(ê{

    --
    "I've got more toys than Teruhisa Kitahara."
    1. Re:I Have an AMD CPU by zaguar · · Score: 5, Funny
      ðN9'óI]öR9ù¥Î6ýPoe}+èa(ê{

      Interesting Perl script.

      --
      "Sure there's porn and piracy on the Web but there's probably a downside too."
    2. Re:I Have an AMD CPU by Minwee · · Score: 4, Funny
      ðN9'óI]öR9ù¥Î6ýPoe}+èa(ê{

      Interesting Perl script.

      It's also rule number 26 in sendmail.cf.

    3. Re:I Have an AMD CPU by ozmanjusri · · Score: 2, Funny
      Don't you have someone to press the refresh button for you?

      Yeah, but unfortunately I hired a MCSE and it's turning out to be tougher than I thought training him.

      --
      "I've got more toys than Teruhisa Kitahara."
  4. Corruption by XanC · · Score: 2, Informative

    Corruption is the cardinal sin of a CPU. If it can't compute a result accurately, it should shut down rather than give a wrong answer.

    1. Re:Corruption by leendertv · · Score: 5, Insightful

      No CPU can guarantee to be free of corruption, the goal of the designer is just to minimize the likelihood of corruption. The design margins are usually such that proper operation is ensured, except for the statistical outliers. However, even CPUs with several error checking and correcting mechanisms can still corrupt data, it is just extremely unlikely. A CPU can never know for sure if it can compute a result accurately, or if an operation was performed correctly, just like no communications system can achieve bit error rates of 0.

      Data corruption in integrated circuits can come from several different sources. Cosmic rays are likely to alter memory values, especially so in DRAM cells. Typically, only ICs for space applications are actually radiation hardened. Much less likely, transistor device noise can corrupt data. Transistor device noise is usually more an issue in RF circuits. Finally, not all manufacturing defects can be found during manufacturing test, since most test sequences don't even achieve 100% fault coverage under currently used fault models, and this does not even consider how closely the models represent the actually circuit failure modes.

      Really, for most people this floating point data corruption is probably a non-issue. It is even more unlikely that errors in floating point data lead to exploits. It is more likely that some bits of your DRAM memory will get corrupted. On my system with ECC RAM that is a few years old, logs show that I get about 1 or 2 (correctable) errors per day...

    2. Re:Corruption by caspper69 · · Score: 2, Informative

      Granted, for most people this may well be a non-issue, and data corruption is a fact of life. However, when a CPU is KNOWN DEFECTIVE in a repeatable, data-corrupting way, it is the vendor's responsibility to replace/fix it.

      Similar to vehicle recalls. Most people would never be affected by many of the things vehicles are recalled for, but that doesn't mean that known *serious* defects are simply let go.

      I have actually studied this bug, and it is only observed when the fpu code is iterated in the MILLIONS of times without ever executing another instruction (only a tight FPU loop), in addition, the environmental temperature must also be high (think tropical). AMD has stated (1) that this problem has never been identified in actual production code (only a single benchmark in these environmental conditions); and (2) that they are identifying and replacing (for free) all affected CPUs. It is estimated that 2-3,000 chips have this particular defect (out of the millions shipped). Further, AMD has added an additional validation step to identify processors affected by this glitch, which will cause them to be pushed down to a lower speed grade (i.e. 2.8GHz affected CPUs will be sold as 2.6GHz parts), where this problem does not manifest itself.

      I for one am happy that this story broke 2 days ago, and 1 day ago AMD had already figured out which CPU batches could potentially be affected, and is offering free replacements (without the customer complaining first). Now today it's on Slashdot. At least this isn't the F00F bug which Intel didn't tell anyone about until the public discovered it and raised hell. Further, the likliehood of data corruption caused by this glitch, even in fpu-heavy code, is extremely unlikely as there would be other non-fpu instructions executed in between in nearly every case (except extreme benchmarking-- i.e. the reason AMD discovered the problem in the first place).

  5. An old problem by AndrewStephens · · Score: 4, Informative
    Something similar used to happen on very old processors, back in the day. If certain instructions were executed in tight loops, the chips would experience localised heating and eventually malfunction (sometimes with permanent damage).

    I'm too young to remember the details (I think it goes back to the early eighties at least), but perhaps some of the elder gods that lurk around here might be able to supply more details.

    --
    sheep.horse - does not contain information on sheep or horses.
    1. Re:An old problem by Alien+Being · · Score: 3, Funny

      I used to burn out a lot of abacus beads.

    2. Re:An old problem by Jerf · · Score: 4, Funny

      Do not meddle in the affairs of the Elder Gods, for you are crunchy, and good with ketchup.

    3. Re:An old problem by Dadoo · · Score: 2, Informative

      If certain instructions were executed in tight loops, the chips would experience localised heating and eventually malfunction (sometimes with permanent damage).

      You're thinking about magnetic cores.

      Whenever you reverse a core's magnetic field, its temperature rises a little. Keep reversing the field fast enough and for a long enough period of time, and the core (or maybe the wires running through it?) will melt, permanently damaging that bit.

      --
      Sit, Ubuntu, sit. Good dog.
    4. Re:An old problem by Mister+Transistor · · Score: 5, Informative

      You may be referring to the early MC6800 8-bit processors. The first ones had a major problem in that the internal registers were dynamic RAM style memory, and synchronized to the internal state clock. If you halted the processor for an extended period of time, the refresh clock to them ceased and the registers got hot, drew too much current and burned up!

      I'm pretty sure that gave rise to the joke "Halt and Catch Fire"...

      I always figured that if you were to burn out a register from overuse, it would be the carry bit ;)

      Anyway, as to the story at hand, it sounds like this would only ever occur a) to only 3000 processors total - MAYBE, and b) would only ever happen under such an artifically contrived laboratory stress-test/benchmark situation. Any CPU running in a real system would a) have to do other things like service hardware interrupts, and b) wouldn't do something useless like perform a looping calculation without checking to see if it was done periodically. It really sounds like this is a big non-issue in reality.

      --
      -- You are in a maze of little, twisty passages, all different... --
    5. Re:An old problem by AndrewStephens · · Score: 2, Insightful

      I agree with your comments on the current story. In reality, all modern processors have flaws that only occur in extrememly unlikely circumstances. This one is not any different.

      --
      sheep.horse - does not contain information on sheep or horses.
    6. Re:An old problem by Mister+Transistor · · Score: 4, Insightful

      I'll go you one better - I have formed my own personal postulate/theory/law that:

      No sufficiently complex system can ever be completely bug-free.

      and it's corollary:

      It is impossible to completely test a sufficiently complex system in every possible way to be certain that it's bug-free.

      In that vein, someone once said "Foolproof is impossible because fools are so ingenious", and "As soon as an idiot-proof system is devised, they go and invent a better idiot!"

      --
      -- You are in a maze of little, twisty passages, all different... --
    7. Re:An old problem by something_wicked_thi · · Score: 5, Informative

      RTFA. They are offering a free replacement. However, the FDIV bug was overblown. For most people, it didn't matter (few people were using software that required division precise enough to be affected). This bug is even less worrisome. Its effect is, at the moment, completely unobserved in the wild using real world applications. The FDIV bug was apparent to anyone with a calculator.

      I'm not saying AMD should be let off the hook completely, but the bug isn't a big problem, they are offering free replacements, and they publicized it. The FDIV bug was bigger (though still hardly catastrophic), refused (at first) to offer replacements, and they sat on it. The two scenarios are nowhere near similar. Maybe AMD just has more character than Intel, or maybe they were watching in 94/95 when the FDIV bug happened and they've actually learned from Intel's mistakes. Regardless, this whole story is more of a heads-up to concerned buyers than a criticism of AMD.

    8. Re:An old problem by Soul-Burn666 · · Score: 2, Interesting

      Actually hardware IS different. As complex as hardware is, it is much less complex than software and has much simpler logic to check. This allows for systems for "formal verification" which happen to work exceedingly well for hardware. For example IBM's "RuleBase" is a system that uses temporal logic to verify a certain piece of "code" (which will later be compiled to hardware) against a set of logical rules.
      When the system can be used, it helps clear out logic bugs very efficiently.

      That being said, today's microprocessors are huge and therefore have to be split to modules in order to test like this. Moreover, it only tests logic. Other systems have to be used to test issues of overheating, cross-talk and actual physical design.

      --
      ^_^
    9. Re:An old problem by myowntrueself · · Score: 2, Funny

      "Keep reversing the field fast enough and for a long enough period of time, and the core (or maybe the wires running through it?) will melt"

      Shit, so for once reversing the polarity does more harm than good?

      --
      In the free world the media isn't government run; the government is media run.
    10. Re:An old problem by AWhistler · · Score: 3, Insightful

      There is a HUGE difference between this AMD problem and the FDIV bug. The FDIV bug, once found, was one of those "1,2, BANG" bugs (do step 1, then step 2 and BANG, the bug is there). With this AMD bug, you have to do the same operation many times before you see the problem, and then the problem is random (only if it overheats enough). Another possible solution to this is to use better heat sinks. This AMD problem isn't 1,2,BANG. Bugs that are of this nature are orders of magnitude harder to find and characterize.

      But you're right, since Intel blundered so badly on their handling of he FDIV bug, everyone else learned from it.

    11. Re:An old problem by AWhistler · · Score: 2, Interesting

      Then you REALLY need to get new MBA textbooks, since the one you have been reading is too politically correct to be useful. Here is a link from the guy who discovered the bug which includes a timeline (I can't believe his FAQ is still online!)...

      http://www.trnicely.net/pentbug/pentbug.html/

      Pay close attention to questions 9, 10, and 11. It explains what REALLY happened, and the author's opinions on the matter, which to my memory are quite accurate. How do I know? At the time I owned a Gateway Pentium 90 that I could use the Windows calculator on to verify the bug. So once Intel announced the recall, I called to get mine replaced. The box they shipped the replacement in was about 12" x 12" x 8" and very overpadded...way larger than the boxes they ship processors in now for sale. I had to replace my Pentium, box it up and send it back to Intel, and I had to give them my credit card number so that they could bill me $500 ($600?) if I failed to return the old CPU.

    12. Re:An old problem by LurkerXXX · · Score: 2, Interesting
      What the hell kind of crappy MBA program did you go to? Intel did *NOT* handle it well. I had one of those CPUs. Intel tried to tell me (a scientific researcher) that my computations didn't need that level of FPU accuracy, and that they wouldn't replace it. It was only after we, the users, screamed bloody murder and brought lawsuits that they decided to back down and replace them all.

      The PR nightmare was *caused* specifically by the way Intel handled the discovery. They thought that they had the right to decide which users did or didn't 'need' accurate FPU computations.

      I have been an 'AMD fanboy' from that day forward, specifically because of Intel's totally botched handling of an engineering glitch.

  6. Fearmongering? by zaguar · · Score: 2, Interesting

    Is it reasonable to be afraid of this. To exploit this, in a way to allow running of arbitary code, you would need a buffer overflow - which is what this AMD weakness is purporting to allow. However, how many are affected? Only a few of the AMD chips, and AMD has only what, 30% of the market. So to code an exploit, you would be writing to a very limited audience, to a point where it is futile. Why not just exploit the latest create.Textrange of WMF exploit in IE/Windows? Much more money in that.

    --
    "Sure there's porn and piracy on the Web but there's probably a downside too."
    1. Re:Fearmongering? by Saven+Marek · · Score: 2, Insightful

      > Only a few of the AMD chips, and AMD has only what, 30% of the market.

      The intel fanboys have been too noisy lately! AMD has more than 50% of the market since this year already!

  7. Re:Kernel fix? by Umbral+Blot · · Score: 4, Insightful

    The big question is will someone write malware/virus to somehow take advantage of this flaw?

    I am curious how a virus could possibly exploit this. It would have to a) hog the resources so that it ran nearly exclusively, which would mean the virus already had control, and b) somehow cause a floating point error to result in a priviliages error. (priviliages and security routines rarely use floating point numbers). Also why would a kernel patch be released for this? It would hurt performance for the rest of us, customers with defective chips should simply return and replace them.

  8. Uh oh.. by BigZaphod · · Score: 5, Funny

    Wow! AMD has invented a way to crash an infinite loop! Awesome! Intel? I bet their solution will take twice as long to crash this loop:

    10 PRINT "HELLO WORLD"
    20 GOTO 10

    AMD is always innovating.

    1. Re:Uh oh.. by JollyFinn · · Score: 2, Interesting

      No that won't crash its FLOATING POINT memory fetch, addition and multiplication loop! Then we need to unroll the loop enough to hide the floatingpoint unit latency. So that it stays active.

      10 I = 10.1
      20 K = I
      21 K2 =I
      22 K3= I
      23 K4= I
      30 K = K + 2.1
      40 K = K * 2.1
      50 K2 = K2 + 2.1
      60 K2 = K2 * 2.1
      70 K3 = K3 + 2.1
      80 K3 = K3 * 2.1
      90 K4 = K4 + 2.1
      100 K4 = K4 * 2.1
      50 GOTO 20

      --
      Emacs is good operating system, but it has one flaw: Its text editor could be better.
  9. Deja Vu: Intel Processor's Bug in 1994 by reporter · · Score: 3, Insightful
    In 1994, Intel's Pentium processor suffered from a division error. Intel handled the problem by initially requiring customers to "prove" that the error caused a serious impact on the customers' lives before Intel would agree to replace the defective chips. Later, after much pressure and lost credibility, Intel agreed to replace all the defective chips without requiring the customer to "prove" his case.

    AMD has a unique opportunity to do the right thing: offering to replace all the defective chips. If AMD does the right thing, then it will only help AMD in its litigation against Intel and in various attempts to increase marketshare. After all, would you not prefer to buy from a reputable company instead of a dishonest, shifty company?

  10. Re:Deja Vu: Intel Processor's Bug in 1994 by Anonymous Coward · · Score: 4, Informative

    "The company is also working with OEMs to identify affected parts and contact customers who could be affected - if they are, they will be offered free replacements."

    forth paragraph in TFA.

  11. nice! by B3ryllium · · Score: 3, Interesting

    Wow, that was fast. FreeBSD already has a patch for this.

    Judging from the posting date, I *really* need to be updating my sources more often. :)

    20060419: p7 FreeBSD-SA-06:14.fpu
                    Correct a local information leakage bug affecting AMD FPUs.

    (could be an unrelated correction, I guess, it doesn't provide much more information in /usr/src/UPDATING)

    1. Re:nice! by larry+bagina · · Score: 3, Informative
      it is an unrelated correction:

      ...As a result of this discrepancy remaining unnoticed until now, the FreeBSD kernel does not restore the contents of the FOP, FIP, and FDP registers between context switches.

      source

      --
      Do you even lift?

      These aren't the 'roids you're looking for.

  12. Re:Kernel fix? by larry+bagina · · Score: 5, Informative
    I'm sure someone will have a kernel patch to prevent this from happening in linux in very short order.

    Not likely. This is valid user code that is being executed. On other CPUs, the same code wouldn't cause a problem. Something like the F00F bug is fixable in the kernel by mucking with exception handler. This is pure user-land code.

    --
    Do you even lift?

    These aren't the 'roids you're looking for.

  13. It's like you're overclocking when you're not by IvyMike · · Score: 4, Insightful

    This is different than the Intel bug; that was a logic flaw, where the chip computed a floating point quantity using an incorrect algorithm. This is an implementation error. In fact, the article mentions that they're going to re-spec the parts and they'll be fine. So if you've got a 2.8Ghz part, and you run this loop at 2.8Ghz (within the old spec), it's like you're "overclocking" (because you're actually outside of AMD's new spec). My guess is that if you over-bought your heatsink and got something better than the stock OEM cooling solution, you would be fine even if you ran this loop all day. Yay, arctic silver!

  14. Fearmongering? No, you misunderstand ... by AHumbleOpinion · · Score: 2, Informative

    Is it reasonable to be afraid of this. To exploit this, in a way to allow running of arbitary code, you would need a buffer overflow

    I think you are misunderstanding the nature of the problem. This is not data corruption as in buffer overflow, this is data corruption as in the calculation comes up with an incorrect answer. For some people that is not acceptible.

    1. Re:Fearmongering? No, you misunderstand ... by larry+bagina · · Score: 2, Funny

      yeah, 1.0 + 1.0 = 3.0, for sufficiently large values of 1.0

      --
      Do you even lift?

      These aren't the 'roids you're looking for.

  15. Could be worse by Khith · · Score: 2, Funny

    Just imagine if you had one of those Pinnacle chips and accidently pressed @[=g3,8d]\&fbb=-q]/hk%fg followed by delete..

  16. Prime95 as a detection tool? by Antiocheian · · Score: 2, Informative

    I have used Prime95 in the past to identify problematic configurations. It's a tool whose main goal is to find prime numbers, but it can be used as an excellent stress test for the processor and memory units.

    Could Prime95 be used to identify those AMD chips?

  17. CALL ESP by Myria · · Score: 3, Interesting

    Probably the easiest errata to come by is the instruction "CALL ESP" (or "CALL RSP"). On AMD CPUs, "CALL ESP" will jump to the address in ESP, *then* push the return address. However, on Intel CPUs, it will push the return address first, then jump to the value it just pushed. This is, of course, disasterous if you try to use it.

    According to Intel errata documents, this is a bug in the Pentium Pro that has been kept for several generations. The Pentium and below, except the 8086 and 8088, worked correctly with this instruction.

    If you want to differentiate Intel and AMD in your program and don't want to use CPUID, you can set up a test with CALL ESP.

    Melissa

    --
    "Screw Sun, cross-platform will never work. Let's move on and steal the Java language." - Visual J++ Product Manager
  18. Quality Control at AMD must be good. by MROD · · Score: 5, Interesting

    Having read a lot about this flaw it's actually amazing that AMD's quality control found the problem in the first place.

    The actions needed to cause the problem to arise are so extreme that they'd never happen in the field. i.e. Loop through tight floating-point only instructions without any comparisons for maybe hours before the error occurs.

    This would *NEVER* happen in the field. Firstly, in any modern OS the process would have been pre-empted long before any problem could occur (causing other instructions to run and hence stopping the overheating). Secondly, no real-world program would ever do this sort of thing as there would always be a comparison in the loop within the timeframe.

    This is a theoretical problem only in the real world, especially as it only affects about 3000 processors in total (it has been quoted). This is why AMD gave it such a low priority. We should just forget about it and move on.

    --

    Agrajag: "Oh no, not again!"
    1. Re:Quality Control at AMD must be good. by kinnell · · Score: 2, Informative
      Having read a lot about this flaw it's actually amazing that AMD's quality control found the problem in the first place.

      The actions needed to cause the problem to arise are so extreme that they'd never happen in the field.

      This kind of thing is standard practice. If you want to stress test a piece of hardware, you write specialised test code which will consume the maximum amount of power possible, not a real world program. You have to be sure that nobody will be able to write software which will drive the processor harder than your tests have. Its good that AMD found this fault, and even better that they owned up to it, but it's not remarkable.

      --
      If I seem short sighted, it is because I stand on the shoulders of midgets
  19. Re:Sounds familiar by smash · · Score: 4, Informative
    Hmm.... I doubt you'd need a few million cells though.

    Some of the tendering spreadsheets i've seen for a few companies i've worked for have had quite a lot of calculation going on in them - change a few cells that others depend on that have others depending on them, etc.... do that all day, it adds up quick.

    You only need 1 of those operations in that instance to screw up and you could be down a few million dollars, if it's not picked up.

    Even forgetting that it's just the moral thing to do...Risk vs replacement cost = no brainer. If only 3000 cpus are affected at say $300 each for amd to sell retail (i'm sure their cost is FAR less), they'd be mad not to just do it (maybe even offer a free speed bump) and reap the positive PR.

    All it needs is for ONE company to blame a budget blowout on them and it's well and truly paid for...

    smash.

    --
    I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
  20. This will not happen to you by Bloater · · Score: 4, Informative

    If you have any interrupts coming in, or your loop has a termination condition. I think you have to have your hardware set to send an interrupt many hours in the future then start an otherwise nonterminating loop.

    So under normal conditions on normal PC hardware, this simply won't happen.

  21. Re:What? by tomstdenis · · Score: 2, Interesting

    There are two parts to that. First off, the composition of the die is varied. Some parts are the ALU, FPU, cache, etc. So depending where the current is going changes the heat [no duh]. The FPU is particularly nasty as unlike the ALU it takes at least 2 EX cycles to do anything and most complicated instructions are at least 4 EX cycles. This means something in the FPU is running for 4 cycles at a time, cannot be interrupted, etc.

    So getting heat local to the FPU isn't too surprising. There are various things in place to mitigate that, for example, the heat spreader. But it can only absorb heat so fast. The lack of APIC interrupts (e.g. timers) makes this test rather artificial. If I recall correctly OSes send timer interrupts to processors to schedule tasks. So this would have to be something that is beyond an OSes control. Like you'd have to write your own mini-OS or something.

    The other part though is you have to keep in mind making processors is not an exact process. My two x85 series opterons probably have slightly different features (e.g. exact alignment) even though they're made from the same design. If I sliced them open and got "my first electron microscope" and looked at them I'd probably be able to measure slight differences. There are other controlled issues (quality of material, chemcials, etc). So that a batch of processors exhibit this problem is concerning but not impossible.

    I'll bet you they probably have another test on the QA line now :-)

    Tom

    --
    Someday, I'll have a real sig.
  22. Re:Haha by WilliamSChips · · Score: 2, Insightful

    When AMD has a problem, it only affects 3000 or so processors and causes minor corruption when a million-line-long piece of code is called without being stopped at any time. When Intel has a problem it affects millions of processors and crashes your computer when a single 32-bit command is called. I know whom I'll be buying from.

    --
    Please, for the good of Humanity, vote Obama.
  23. Surprising. AMD uses my `cpuburn` by redelm · · Score: 4, Informative
    About 7 years ago, I wrote a suite of open-source CPU stress-tests I called `cpuburn`. Little optimized assember pgms designed to stress different parts of the CPU. `burnK7` does precisely this FPU dot product.

    Of course, I expect AMD's production testing dept to have far better code, since they will devote more job hours to it and know proprietary chip details. Still, different parts of AMD as emailed me several times to thank me because they found the pgms useful. Great.

    But these guys know what they're doing. Heat transfer from the hot multipliers has to be carefully analysed [3D finite element heat transfer analysis]. I suspect something far more mundane, like someone reducing die or slug thickness, or a mfg problem with the die/slug gap or thermal goop.

  24. Re:Humanly reproducable :) by fimbulvetr · · Score: 2, Insightful

    I think someone's confusing user error/not enough troubleshooting with an almost not reproducable issue. TFA mentions a lot of instructions without enough pause of FPU code to cool down. This isn't your bug if you're playing WC3. WC3 uses TCP/IP. TCP/IP generates interrupts - lots of interrupts. So many interrupts that your FPU has plenty of time to cool down between calculations. There are many handy ways of troubleshooting this issue of yours, and I'd bet you're not going to identify the problem by some slashdot story submission.

  25. This flaw seems damned serious to me... by anubi · · Score: 2, Insightful
    ... because the multiply-add is the basic building block of digital signal processing.

    You are apt to be doing this extensively when processing audio or video streams.

    --
    "Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]