Some Core I7 5960X + X99 Motherboards Mysteriously Burning Up

← Back to Stories (view on slashdot.org)

Some Core I7 5960X + X99 Motherboards Mysteriously Burning Up

Posted by timothy on Saturday September 6, 2014 @10:08AM from the think-of-it-as-a-feature dept.

An anonymous reader writes "Intel's Haswell-E Eight-Core CPU and X99 motherboards just debuted but it looks like there may be some early adoption troubles leading to the new, ultra-expensive X99 motherboards and processors burning up. Phoronix first ran a story about their X99 motherboard having a small flame and smoke when powering up for the first time and then Legit Reviews also ran an article about their motherboard going up in smoke for reasons unknown. The RAM, X99 motherboards, and power supplies were different in these two cases. Manufacturers are now investigating and in at least the case of LR their Core i7-5960X also fried in the process."

102 comments

Min score:

Reason:

Sort:

HCF by Anonymous Coward · 2014-09-06 10:15 · Score: 5, Funny

Seriously don't execute the halt and catch fire instruction.
1. Re:HCF by infolation · 2014-09-06 10:52 · Score: 4, Funny
  
  As the late Murray Walker said...
  
  "...there's nothing wrong with the motherboard, except that it's on fire"
2. Re:HCF by geoskd · 2014-09-06 11:03 · Score: 1
  
  Seriously don't execute the halt and catch fire instruction.
  At least he didn't accidentally execute an LNM instruction. This looks like it might also have been an attempt at SRSD with an SSD...
  
  --
  I wish I had a good sig, but all the good ones are copyrighted
3. Re:HCF by Anonymous Coward · 2014-09-06 12:20 · Score: 1
  
  ...murray walker isnt dead
4. Re: HCF by Selivanow · 2014-09-06 12:47 · Score: 4, Funny
  
  Maybe they need to include NOSMOKE.SYS in their CONFIG.SYS file.
  
  --
  -- ...trying to make digital files uncopyable is like trying to make water not wet. -Bruce Schneier
5. Re:HCF by Anonymous Coward · 2014-09-06 14:24 · Score: 1
  
  I'm Murray Walker and so's my wife
6. Re:HCF by FatdogHaiku · 2014-09-06 15:34 · Score: 2
  
  Seriously don't execute the halt and catch fire instruction.
  I think it can be simply counteracted with a quick Ctrl+Alt+ohfuck...
  But it does make one long for the days of the three fingered salute.
  
  --
  You have the right to remain sentient. If you give up the right to remain sentient, you will be elected to public office
7. Re:HCF by gweihir · 2014-09-06 16:03 · Score: 2
  
  Is that another of the NSA instructions added with RDRAND? Seriously, would not really surprise me, the NSA is after sabotaging anything these days.
  I gather however that this is plain incompetence (Dunning-Kruger-Type) with regards to the voltage regulators. Switching voltage regulation is really hard to do right unless you over-engineer seriously. You can get all sorts of bizarre effects, including a puff of smoke.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
8. Re:HCF by CaptQuark · 2014-09-06 18:14 · Score: 4, Insightful
  
  I gather however that this is plain incompetence (Dunning-Kruger-Type) with regards to the voltage regulators. Switching voltage regulation is really hard to do right unless you over-engineer seriously. You can get all sorts of bizarre effects, including a puff of smoke.
  I appreciate the irony of you mentioning the Dunning-Kruger syndrome with your statement. Switching voltage regulation has been around for over 30 years and isn't much of a mystery. Since the early motherboards started reducing voltages from 5v down to 3.3v (and below), every motherboard has had on-board voltage regulation. It's hard to believe that something as fundamental as a switching regulator would suddenly exceed the engineering skill of the motherboard designers.
  
  ~~
9. Re:HCF by rhyder128k · 2014-09-06 20:05 · Score: 1
  
  "AMD Really Is On Fire This Generation!" Put that on the cover of the mag.
  
  --
  Michael Reed, freelance tech writer.
10. Re:HCF by richy+freeway · 2014-09-06 21:30 · Score: 1
  
  He came in my shop once and I gave him a pirate copy of Window 98.
11. Re: HCF by Anonymous Coward · 2014-09-07 01:01 · Score: 0
  
  That was a fail.
12. Re:HCF by Anonymous Coward · 2014-09-07 03:30 · Score: 0
  
  Oh.. I really like that one..
13. Re:HCF by Agripa · 2014-09-07 06:37 · Score: 1
  
  I gather however that this is plain incompetence (Dunning-Kruger-Type) with regards to the voltage regulators. Switching voltage regulation is really hard to do right unless you over-engineer seriously. You can get all sorts of bizarre effects, including a puff of smoke.
  I appreciate the irony of you mentioning the Dunning-Kruger syndrome with your statement. Switching voltage regulation has been around for over 30 years and isn't much of a mystery. Since the early motherboards started reducing voltages from 5v down to 3.3v (and below), every motherboard has had on-board voltage regulation. It's hard to believe that something as fundamental as a switching regulator would suddenly exceed the engineering skill of the motherboard designers.
  Switching voltage regulation has been around longer than that but point-of-load switching voltage regulation is more recent and the impedance, power levels, and regulation requirements of these POL switching regulators are cutting edge for a consumer level product.
14. Re:HCF by gweihir · 2014-09-08 10:50 · Score: 1
  
  Really, you know nothing about this issue, the irony is entirely imagined on your side.
  Switching power regulator design is among the hardest things in power electronics. It has gotten easier, but even well-known designs based on manufacturer application notes (and that is the extent to which most designers have "mastered" anything here) can literally blow up in your face if you get something wrong or the situation is not quite as expected. There is noting "fundamental" here. You need a specialized design for most situations unless you do serious over-engineering, with the increased per-item cost that brings.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
15. Re:HCF by PipsqueakOnAP133 · 2014-09-11 12:10 · Score: 1
  
  An engineer for Intel mentioned once that the CPU power regulators now, although low voltage, pass enough amps to do arc welding. Just think about that for a second.
Early adopters by Anonymous Coward · 2014-09-06 10:23 · Score: 1

Adopt a fire extinguisher
1. Re: Early adopters by DigiShaman · 2014-09-06 10:43 · Score: 2
  
  It's called water cooling. Just loosen this clamp over here and...
  
  --
  Life is not for the lazy.
2. Re: Early adopters by Anonymous Coward · 2014-09-06 11:39 · Score: 0
  
  water + electrical fire = fun
Same thing happened to me with AM2 by oic0 · 2014-09-06 10:25 · Score: 4, Interesting

I bought one of the first AM boards and they said it was rated for high watts. I made the power regulator shoot flames 10 minutes after I had it together. They lowered the rated power handling and refunded me but lame Newegg made me pay return shipping...
1. Re:Same thing happened to me with AM2 by Anonymous Coward · 2014-09-07 07:28 · Score: 0
  
  Yep, NewEgg has a bottom line to protect.
Flame and Smoke by Anonymous Coward · 2014-09-06 10:30 · Score: 0

The lifeblood of modern electronics.
Buck feta by KiloByte · 2014-09-06 10:36 · Score: 1, Offtopic

Why in the blazes the inter-Slashdot link leads to the Beta travesty? If I, due to some mental malady, would prefer Beta, I'd set it as my default.
Why does even Dice keep Beta afloat after it failed? I seriously hope the plans to make it the main -- and only -- interface are gone. Oh well, there's still Soylent...

--
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
1. Re:Buck feta by DexterIsADog · 2014-09-06 11:40 · Score: 1, Troll
  
  Get over your butt hurt, or move on. It's really pretty simple.
2. Re:Buck feta by Anonymous Coward · 2014-09-06 20:03 · Score: 0
  
  Get over your butt hurt, or move on. It's really pretty simple.
  How about.. No. Bitch.
3. Re:Buck feta by Anonymous Coward · 2014-09-06 22:25 · Score: 0
  
  AAAaaaaand FUCK YOU TOO!
  Seriously though, beta fucking sucks. If it ever takes over i wont be coming back.
Looks like... by Kyrubas · 2014-09-06 10:36 · Score: 4, Informative

...a failure to contain the magic smoke.
Easy to repair by ArcadeMan · 2014-09-06 10:40 · Score: 5, Informative

All you need is this little kit.

--
Get free satoshi (Bitcoin) and Dogecoins
expectable by Anonymous Coward · 2014-09-06 10:45 · Score: 0

there is a reason these are engineering samples
1. Re:expectable by Anonymous Coward · 2014-09-06 11:10 · Score: 0
  
  When I read "engineering sample" I think of a vial full of caffeinated piss.
NSA by qrwe · 2014-09-06 10:47 · Score: 2

The remotely destructible chip is finally here. I feel sorry for the guinea pigs, though...

--
There are 2 types of people in the world - those who understand decimal and those who don't.
1. Re:NSA by sumdumass · 2014-09-06 11:27 · Score: 1
  
  Finally, it's been around for a while
  Well, maybe not a single chip but the concept anyways.
2. Re:NSA by gweihir · 2014-09-06 16:09 · Score: 1
  
  Funny as that sounds at first glance, it actually makes a lot of sense. In an ordinary mainboard, the CPU cannot directly influence the voltage regulators, but in these CPUs, it can, and hence self-destruct has become possible. After the NSA transparently sabotaged the RDRAND design (the design is insecure, individual CPUs may or may not be sabotaged, but the design basically makes it impossible to tell), it would not surprise me one bit if they actually had Intel add a self-destruct as well. We really need to stop those that want to sabotage critical infrastructure. Nothing good can come of it.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
3. Re:NSA by gweihir · 2014-09-06 16:14 · Score: 1
  
  Ah, sorry. These are not the CPUs with integrated voltage regulators. Still possible but harder to add this type of self-destruct.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Not just one mobo by gman003 · 2014-09-06 10:47 · Score: 4, Informative

Since nobody reads TFA, Phoronix killed an MSI X99S, and LR lost an Asus X99 Deluxe. It was also different RAM (Corsair vs G.Skill). However, both reported the burn was near the VRMs (Phoronix also reported a second event near the northbridge). The two mobos might be using identical parts for that, but I was unable to find out for sure.
1. Re:Not just one mobo by Forever+Wondering · 2014-09-06 11:32 · Score: 4, Interesting
  
  Does Intel have a reference design board for this? Also, how close are the VRMs to the chips they're regulating?
  I once worked at a company that had a reference board with 3 FPGAs with 3 VRMs near the FPGAs. When designing their own board, the company reduced this to one VRM for all 3 FPGAs and put the VRM on the opposite side of the board. It took nine months to realize that this caused the FPGAs to reset during heavy logic switching because the single VRM + the greater length of the traces meant that the VRM couldn't keep up with the demand.
  
  --
  Like a good neighbor, fsck is there ...
2. Re:Not just one mobo by Charliemopps · 2014-09-06 11:32 · Score: 2
  
  Since nobody reads TFA, Phoronix killed an MSI X99S, and LR lost an Asus X99 Deluxe. It was also different RAM (Corsair vs G.Skill).
  However, both reported the burn was near the VRMs (Phoronix also reported a second event near the northbridge). The two mobos might be using identical parts for that, but I was unable to find out for sure.
  I've had 7 Asus motherboards burn up in the past 4yrs. 2 actually caught fire. So that's no suprise to me, Asus is on my banned list.
  MSI, however, has been nothing but good to me. They don't generally have the fastest or most feature rich boards available, but reliabilities been their strong suit over the years.
3. Re:Not just one mobo by ShanghaiBill · 2014-09-06 12:05 · Score: 5, Informative
  
  It took nine months to realize that this caused the FPGAs to reset during heavy logic switching because the single VRM + the greater length of the traces meant that the VRM couldn't keep up with the demand.
  FPGAs use synchronous logic, so they pull power in spikes as the logic switches. If it took 9 months to realize there was a problem, you can probably make some small modifications to get it working reliably. Make sure the leads from the VRM are as fat as possible, preferably have it feed into full ground and power layers, and make sure no other traces are splitting those planes. Clock all three FPGAs from the same xtal, and use a delay gate or tune the length of the traces so the signal is skewed enough that the power spikes from each FPGA are not hitting simultaneously. Add plenty of decoupling caps on every power and ground pin. Make sure the caps have leads that are fat and short. It is better to have a physically small cap (0201 or 01005) in close than a bigger one further out. Good luck.
4. Re:Not just one mobo by Anonymous Coward · 2014-09-06 12:08 · Score: 0
  
  That's simple stuff to diagnose... 9 months? Is there an engineer somewhere on staff or are you all just designers?
5. Re:Not just one mobo by technical_maven · 2014-09-06 12:30 · Score: 1
  
  I have yet to ever have an Asus mobo fail on me after many years and many boards
6. Re: Not just one mobo by Anonymous Coward · 2014-09-06 12:43 · Score: 0
  
  I had an Asus mobo fail the smoke test back in '06. Flame and smoke on powering the board. Took it back, but the shop wouldn't replace it, claiming it worked fine. On closer examination, it was one of the ten or so USB controllers. So I put the computer together and called it Sparky. The original HDD has since crashed, the vid card eventually cooked itself, but the damn thing still runs.
7. Re:Not just one mobo by Charliemopps · 2014-09-06 13:29 · Score: 2
  
  I kind of do my own non-profit buisness of building computers for everyone I know or am related to. So I've got a small business account with newegg and do about $25k in computers a year. Asus was my board of choice for years, but about 3yrs ago they just went to shit. I've no idea why but suddenly I had massive failures, massive compatibility issues, etc... When a computer I build actually catches fire, that worries me. Asus was decent about the RMAs... which actually worried me more. A MB manufacturer will rarely take a return with scorch marks on it unless they know there's an issue. When the RMA boards I got back from them started blowing caps as well, I knew something was terribly wrong.
  Also on my banned list:
  Gigabyte - I had several Gigabyte MB and Gigabyte Video cards. They would not work with each other and Gigabyte claimed it was a capability issue and not their problem, despite having put their names on both the card and the board! This was purely a customer service issue, they should have shipped me a different card to make things right.
  Zotac - For 2yrs I shipped the same video card back to them over and over again. They just kept replacing it with defective cards. Some came to me dirty, or with blown components. You can't just dig around in the RMA'd parts bin and ship some other broken piece of crap back to me. I'm currently awaiting about the 4th RMA on that card and my warranty will run out. At least they're paying for the shipping.
  Anyways, I'm done building computers for people. Components are just too unreliable now. I don't need to be spending half my life in the UPS shipping office.
8. Re:Not just one mobo by PopeRatzo · 2014-09-06 13:40 · Score: 4, Insightful
  
  "FPGAs use synchronous logic, so they pull power in spikes as the logic switches. If it took 9 months to realize there was a problem, you can probably make some small modifications to get it working reliably. Make sure the leads from the VRM are as fat as possible, preferably have it feed into full ground and power layers, and make sure no other traces are splitting those planes. Clock all three FPGAs from the same xtal, and use a delay gate or tune the length of the traces so the signal is skewed enough that the power spikes from each FPGA are not hitting simultaneously. Add plenty of decoupling caps on every power and ground pin. Make sure the caps have leads that are fat and short. It is better to have a physically small cap (0201 or 01005) in close than a bigger one further out. Good luck."
  It's like Penthouse Forum for nerds.
  
  --
  You are welcome on my lawn.
9. Re:Not just one mobo by Man+On+Pink+Corner · 2014-09-06 13:42 · Score: 2
  
  The real problem would have been inadequate bypassing at the FPGA. From the point of view of high-speed logic, power comes from capacitors, not voltage regulators.
10. Re:Not just one mobo by PopeRatzo · 2014-09-06 13:51 · Score: 3, Funny
  
  Also on my banned list:
  OK, no Asus or Gigabyte. I'm gonna build a new game rig. Which companies should I use? I've had good luck with Asus motherboards, but I only make a new computer every 3-4 years or so.
  Personally, I'm surprised every system I assemble doesn't burst into flames, but that's only because I'm not really expert at these things. I hold my breath whenever I have to plug a CPU into a motherboard or slop that silver goop on top of one when I'm attaching a cooler. Once many years ago, I attached a motherboard without putting in those little round standoffs onto the case and it just sort of went "zzzt!" and then smelled like a vacuum cleaner when the belt burns. I took it back to MicroCenter and wept and moaned and they actually gave me a new one. Since then, I make sure to keep a fire extinguisher and a pint of vodka on hand when I build a system. The vodka is to keep my hands from shaking.
  I know I should just go with one of the outfits on the internet that assembles gaming PCs, but I'll probably end up doing the next one myself.
  
  --
  You are welcome on my lawn.
11. Re: Not just one mobo by DigiShaman · 2014-09-06 15:09 · Score: 4, Insightful
  
  Never -ever- cheap out on the PSU. You'll have to read the reviews, but make damn sure you get one with quality Japanese components.
  If motherboards are using shit components these days, at least rely on clean and stable power to it.
  
  --
  Life is not for the lazy.
12. Re:Not just one mobo by fizzer06 · 2014-09-06 15:22 · Score: 2
  
  You can't just dig around in the RMA'd parts bin and ship some other broken piece of crap back to me.
  Well, they obviously can so. Companies like that need to go bankrupt.
13. Re:Not just one mobo by Forever+Wondering · 2014-09-06 15:36 · Score: 1
  
  You're quite right about all that you mentioned. My example was from a 2004 timeframe.
  I can't recall if the faulty design cut back on bypassing/deglitching as well as the VRMs, but my guess would be yes.
  Afterwards the engineer [that caused the problem in the first place] added back all bypassing/deglitching, shorter leads, VRMs, etc. that got elided from the [then] new design. The engineer was definitely in the dog house for this: for creating the problem and taking so long to diagnose it.
  Personally, based on experiences over the years, it should have taken about two months. The extra time took a great deal of momentum out of the product launch.
  To add insult to injury, the same type problem happened two more times. The second time was caused by the same engineer. The third time this engineer got it right, but the PCB layout specialist changed things without telling the engineer. Fortunately, the first engineer, having been burned twice, rechecked the design immediately upon receiving the first PCB sample.
  
  --
  Like a good neighbor, fsck is there ...
14. Re:Not just one mobo by Forever+Wondering · 2014-09-06 15:37 · Score: 1
  
  That's simple stuff to diagnose... 9 months? Is there an engineer somewhere on staff or are you all just designers?
  That question was posed at the time. And not calmly or so politely ;-)
  
  --
  Like a good neighbor, fsck is there ...
15. Re:Not just one mobo by Anonymous Coward · 2014-09-06 15:47 · Score: 0
  
  MSI is on the permanent shitlist fort stuff like this:
  http://www.tomshardware.com/news/MSI-GTX-660-670-overvolting-PowerEdition,18013.html (This is 2013)
  http://discuss.pcmag.com/forums/thread/256921001.aspx (Note this is 2003)
  A similar claim can be leveled at ASUS
  http://www.geek.com/chips/asus-accused-of-cheating-on-motherboard-benchmarks-552624/ (2003)
  I used Gigabyte, but I've had their boards flash themselves to death, and ASRock seems to have a similar issue (where I've had to boot from the backup BIOS every time I update the bios, because the firmware update fails if not done from inside the BIOS itself.)
  I'm running out of options for reliable motherboards. Call me paranoid, but I'm not willing to buy highest-end parts from anyone because of "catching fire" types of reports.
16. Re:Not just one mobo by Anonymous Coward · 2014-09-06 16:04 · Score: 0
  
  and you waited 9 months? And didn't put your resume on the street? Is that place even still in business?
17. Re:Not just one mobo by Anonymous Coward · 2014-09-06 17:12 · Score: 0
  
  Since nobody reads TFA, Phoronix killed an MSI X99S, and LR lost an Asus X99 Deluxe. It was also different RAM (Corsair vs G.Skill).
  However, both reported the burn was near the VRMs (Phoronix also reported a second event near the northbridge). The two mobos might be using identical parts for that, but I was unable to find out for sure.
  I've had 7 Asus motherboards burn up in the past 4yrs. 2 actually caught fire. So that's no suprise to me, Asus is on my banned list.
  MSI, however, has been nothing but good to me. They don't generally have the fastest or most feature rich boards available, but reliabilities been their strong suit over the years.
  Only problem I've ever had with ASUS boards, is 2 boards purchased about 2.5 years apart both had a bad NIC from the factory. (ROM blank). So frustrating >. But I've worked with probably 100 ASUS boards of varying model number/generation, and I prefer them far over any others.
18. Re:Not just one mobo by Anonymous Coward · 2014-09-06 17:46 · Score: 0
  
  Not at all. The real problem is often that the regulator sense pin is not connected at the load. This is because schematic capture software is remarkably stupid and PCB designers are often not aware of things like resistive losses.
  So while capacitors are nice for instantaneous current, when the average current draw goes up, the DC voltage at the FPGA's power pins goes below minimum spec.
  Seen it myself. The regulator's sense pin was drawn connected to the supply. This is correct. But the sense line needs to be physically connected close to the load, ie at the FPGA ball grid, this wasn't done on the PCB.
19. Re:Not just one mobo by tlhIngan · 2014-09-06 18:28 · Score: 1
  
  The problem is not the fix - once you know the problem is power, it's trivial to fix.
  The problem is identifying the root cause. Power problems are highly subtle - and usually very intermittent. The FPGAs may crash under heavy load, but it's one of the "phase of moon" bugs because you can feed in the same test patterns that crash it and it'll work the next time around.
  And bugs that are impossible to replicate are the hardest ones to fix - especially if it's a new board that requires a new change to the RTL so you're not exactly sure if it's a hardware or software problem. Or even a compiler problem (since half the issues can easily be caused by bugs in the compiler).
  So you have a problem, and the problem can be in the hardware, or the software (both the stuff you wrote, and the vendor's stuff), and even on the software running on top of the FPGAs. And you can't replicate it, either, so yeah, it's going to take a LOT of work to find the root cause.
  If you're lucky, you can isolate matters by running the current design on the previous hardware (provided that's possible) which narrows down the list of potential causes greatly. But because of the lead time in hardware, that's generally impossible - you just can't run the current software on the old hardware because too many things changed, just as you can't run the old software on the new hardware.
20. Re:Not just one mobo by ShanghaiBill · 2014-09-06 19:59 · Score: 1
  
  The problem is identifying the root cause. Power problems are highly subtle - and usually very intermittent.
  Power problems are also very common. If you have intermittent failures, it should be the first place you look. They are also easy to diagnose: If you have a failure once a week, then remove some decoupling caps. Now it fails every hour or so. Remove a few more caps, and now it fails in minutes or seconds. Once you are sure it is a power problem, it is straightforward to remedy. Add more capacitance. Check your ground and power layers. etc.
21. Re:Not just one mobo by Anonymous Coward · 2014-09-06 20:59 · Score: 0
  
  Or you need better warranty laws: the time they need to replace it doesn't count, each replacement (attempt) increases warranty by 6 months, if they fail 3 times they have to refund the money. That's e.g. (more or less) the rules in Germany.
22. Re:Not just one mobo by Forever+Wondering · 2014-09-06 21:14 · Score: 1
  
  Thanks for what you just said. Dead on and I couldn't agree more.
  In our case, we were a small [startup] company, so we didn't have the resources to be second guessing each other. I was doing the device drivers, but I'm also 50% EE. When we found out what had happened, we were struck speechless that the first thing to check [IMO, yours, and the opinion of some of the other engineers] wasn't checked. Sigh.
  
  --
  Like a good neighbor, fsck is there ...
23. Re:Not just one mobo by Forever+Wondering · 2014-09-06 22:12 · Score: 1
  
  The problem is not the fix - once you know the problem is power, it's trivial to fix.
  ShanghaiBill is correct. Power is the first thing to check/suspect. In our case, the other engineering team members assumed the lead engineer had checked this--because it is so fundamental. He hadn't. He was almost fired for this.
  
  The problem is identifying the root cause. Power problems are highly subtle - and usually very intermittent. The FPGAs may crash under heavy load, but it's one of the "phase of moon" bugs because you can feed in the same test patterns that crash it and it'll work the next time around.
  We had no problem generating test vectors that caused the problem to occur once per hour.
  
  And bugs that are impossible to replicate are the hardest ones to fix - especially if it's a new board that requires a new change to the RTL so you're not exactly sure if it's a hardware or software problem. Or even a compiler problem (since half the issues can easily be caused by bugs in the compiler).
  We were quite confident that it was a hardware problem because both boards were 100% compatible software [device driver]-wise. My device drivers also would log all access to the board in realtime, so this would be dumped post-mortem and jointly analyzed by myself and the given logic designer for correctness against the hardware spec.
  If this didn't match up, either my driver was wrong or the compiler had a bug. In this particular case, if we weren't tracking an intermittant, a hard fail error usually resulted in the logic designer fixing his logic. That is, before involving the designer, I prechecked the post-mortem dump and corrected the driver logic errors [if any] first.
  
  So you have a problem, and the problem can be in the hardware, or the software (both the stuff you wrote, and the vendor's stuff), and even on the software running on top of the FPGAs. And you can't replicate it, either, so yeah, it's going to take a LOT of work to find the root cause.
  One of my specialties is finding a way to add a stress mode to a driver or the driving application (e.g. diagnostic program) that reduces a "once a month" intermittant to "once an hour" or, in many cases, force the same fail in 10 seconds.
  
  If you're lucky, you can isolate matters by running the current design on the previous hardware (provided that's possible) which narrows down the list of potential causes greatly. But because of the lead time in hardware, that's generally impossible - you just can't run the current software on the old hardware because too many things changed, just as you can't run the old software on the new hardware.
  I've never had a problem writing device drivers that autodetect and work on any and all current and previous generations of hardware. If you had a driver routine that was "perform_xop" and what that had to do changes from v1 to v2 of the hardware, you just create a vtable and call vtable->perform_xop where this is populated with "v1_perform_xop" or "v2_perform_xop". Really, it's not that magical.
  In our particular case, we had a hardware "capabilities" register [which I had the logic designers add]. So, our driver would also do things like:
  if (capreg & CAP_HAS_FEATURE_A) ...
  
  --
  Like a good neighbor, fsck is there ...
24. Re: Not just one mobo by PopeRatzo · 2014-09-06 23:27 · Score: 1
  
  Never -ever- cheap out on the PSU.
  
  Yes, I've learned that the hard way, too. When I built my current system, I spent almost as much money (and time reading reviews) on the power supply as on the video card. It's more than I need but it has a longer warranty than a new car.
  
  --
  You are welcome on my lawn.
25. Re:Not just one mobo by Anonymous Coward · 2014-09-07 02:05 · Score: 0
  
  Don't worry about this guy. I assemble PCs with ASUS and Gigabyte motherboards every single week of the year with ZERO issues. I had no problems with Gigabyte GPUs either. As is the case for many other brands for that matter.
  Zotac on the other hand? Yes, they're the worst of the worst. Their RMA department just sends back your broken motherboard, or offers to replace it with something you'd never want of after making you wait over a month.
26. Re:Not just one mobo by Midnight+Thunder · 2014-09-07 03:12 · Score: 1
  
  Or you need better warranty laws: the time they need to replace it doesn't count, each replacement (attempt) increases warranty by 6 months, if they fail 3 times they have to refund the money. That's e.g. (more or less) the rules in Germany.
  One company I know will attempt to repair three times. If it still fails they replace, but the replacement part starts with a new warranty. I think that is the way it should be. The warranty should be on the part, so any replacement would effectively reset the effective purchase date.
  That may get expensive for some companies, but maybe they should be rethinking their business model?
  
  --
  Jumpstart the tartan drive.
27. Re:Not just one mobo by the_skywise · 2014-09-07 05:26 · Score: 2
  
  It's like Penthouse Forum for nerds.
  Not quite - It didn't start with "I never believed that this would happen to me..."
28. Re:Not just one mobo by K10W · 2014-09-07 13:33 · Score: 1
  
  I kind of do my own non-profit buisness of building computers for everyone I know or am related to. So I've got a small business account with newegg and do about $25k in computers a year. Asus was my board of choice for years, but about 3yrs ago they just went to shit. I've no idea why but suddenly I had massive failures, massive compatibility issues, etc... When a computer I build actually catches fire, that worries me. Asus was decent about the RMAs... which actually worried me more. A MB manufacturer will rarely take a return with scorch marks on it unless they know there's an issue. When the RMA boards I got back from them started blowing caps as well, I knew something was terribly wrong.
  Also on my banned list: Gigabyte - I had several Gigabyte MB and Gigabyte Video cards. They would not work with each other and Gigabyte claimed it was a capability issue and not their problem, despite having put their names on both the card and the board! This was purely a customer service issue, they should have shipped me a different card to make things right.
  Zotac - For 2yrs I shipped the same video card back to them over and over again. They just kept replacing it with defective cards. Some came to me dirty, or with blown components. You can't just dig around in the RMA'd parts bin and ship some other broken piece of crap back to me. I'm currently awaiting about the 4th RMA on that card and my warranty will run out. At least they're paying for the shipping.
  Anyways, I'm done building computers for people. Components are just too unreliable now. I don't need to be spending half my life in the UPS shipping office.
  I've found msi make the list too, lot of others find the same. Seems common across wide range of experiences of rig builders and no guaranteed reliable manufacturers now IMO. I just use handful of UK stores with great support whop replace no questions asked for free and honour the warranties for products; the 3 main places I use cover everything without specifics for 12month but do 3 to 5 year on certain products. I always always use psu's that are reliable and tested by good sources. Takes a week of reading around before purchase and always pay more than a crap one but it is worth it IMO but even with more positive than negative experiences I'd be foolish to trust in something completely. Consumer products just aren't made to be failproof, redundant failsafes and seriously over-engineered are things that do not mix with commercial products with sane pricetags for items with limited life before they become obsolete. Failure will always be part of the chain if you make enough machines for you/others. All you can do is insulate yourself from how likely it is by picking tested combos and good warranty scheme for if it did come to crunch
  
  Even free shipping back as places have various policies from courier to pick up defective one when they deliver your replacement, email prepay post label to print out or post prepay box to ship back in (failure to do so gets you billed to stop scams). My advice to friends/family is only use someone with good returns policy who wont palm the problem off on the oem as none seem to give a toss with all products unless it is major known issue.
  
  FWIW the last 2 rigs I made for me (msi mobo with i3 2120 xfx gtx460 and asus w/ i7 4790k gigabyte gtx770 both with corsair ram (I have less issues with gskill tbh)) are fine. Shockingly the msi mobo one going strong for 4 years nearly despite how many rma's in the first week of life on the mobo alone. I received numerous new boards (due to way it was returned I know it was not the same one shipped back out) and all had issues with DOA/, faulty or refusing to play nice with other components despite listed on msi and component manufacturers sites as tested confirmed compatible when it wasn't. Never had bad luck like that on single part in building rigs since mid 90's for myself/ family/ friends/ their friends etc etc. I'd tried other new components and tested working older parts t
29. Re:Not just one mobo by toddestan · 2014-09-07 15:35 · Score: 1
  
  That's not my experience. I've always wondered why Asus has been held in such high regard when I've found their stuff be to be pretty much crap, dating back to at least the Socket A days. Not just motherboards too, as their video cards are just as flaky and die just as quickly, and don't buy their laptops either unless you need a paperweight. Heck, I'd buy ECS before buying Asus. The quality may not be any better but I'd at least save myself some money.
30. Re:Not just one mobo by Anonymous Coward · 2014-09-07 23:11 · Score: 0
  
  With X99, intel has changed from having multiple lines to it all being controlled within the processor, and only having a VCCIN. Could it simply be an overload?
31. Re:Not just one mobo by Larry_Dillon · 2014-09-08 02:46 · Score: 2
  
  > They just kept replacing it with defective cards.
  I've seen a few companies do this over the years. They just keep sending defective parts until you give up in frustration or they go out of business.
  
  --
  Competition Good, Monopoly Bad.
32. Re:Not just one mobo by Anonymous Coward · 2014-09-10 11:50 · Score: 0
  
  With X99, intel has changed from having multiple lines to it all being controlled within the processor, and only having a VCCIN. Could it simply be an overload?
  The previous "multiple lines" scheme wasn't multiple rails supplying about the same amount of power each. Instead, probably ~90% of the power came in on a single ~1.0 volt rail which supplied all of the CPU core logic power. The other rails were separated because they supplied things which needed other voltages, e.g. ~1.5V for DDR3 I/O power, 1.8V PLLs (clock generation circuitry), and so forth. These "misc" supply rails needed at most a few watts each.
  This means there's not much more load on the big motherboard point-of-load regulator than before. Also, its output voltage is ~1.8V instead of ~1.0V, and current is therefore down (according to P=I*V). All else equal, a regulator generating higher voltage at less current is easier to design. And since it's now just an intermediate conversion step, it shouldn't have to maintain insanely tight output regulation tolerances the way the Vcore regulator on older boards did.
  All that said, an overload's certainly possible. You can always screw a design up. But based on general engineering principles FIVR processors should result in easier motherboard power supply design, not harder.
Re: by kurkosdr · 2014-09-06 10:48 · Score: 2

Just a failure of the magic smoke sealant.
Houston ... by Forever+Wondering · 2014-09-06 10:50 · Score: 4, Funny

... We've had a main B bus undervolt

--
Like a good neighbor, fsck is there ...
1. Re:Houston ... by distilate · 2014-09-06 11:27 · Score: 4, Funny
  
  It looks like we are venting somthing
2. Re:Houston ... by Anonymous Coward · 2014-09-06 11:59 · Score: 1
  
  Roger. We copy your venting.
  
  Dramatic Music Intensifies
3. Re:Houston ... by Ed_1024 · 2014-09-07 00:37 · Score: 4, Insightful
  
  I was interested by the actions of the user of the motherboard on one of the linked articles. Initially this happened:
  
  "The system came up, hung for a very short time and then powered off with a audible click of the Corsair AX860i power supply. If you have ever heard the loud click of the Over Current Protection (OCP) shutting down the PSU you know exactly what click I heard. Now when I press power button on the motherboard the system clicks after being on for a split second. I unplugged all the cables on the power supply and did the built-in self-check and it passed with flying colors. I still swapped out the PSU with a backup Corsair AX860i and the same click was to be heard. and it is doing the same thing (Corsair AX860i). After clearing the CMOS, removing the memory, SSD and video card the system still would not post. At that point in time I switched to a non-digital power supply (Corsair AX1200) and it did the same thing although this time the OCP took a little longer to kick in. There was some audible crackling noises, followed by some smoke near the CPU VRM heatsink. So, the heart shattering smell of burnt electronics filled the room..."
  
  10/10 for investigative journalism but putting more and more juice into something that is continually tripping out the power supply is not going to have a happy ending. Maybe some of the $1,400-worth of motherboard and processor may have been salvageable if he had stopped at the first warning?
  
  If the circuit breaker pops twice on a ring main at home, do you a) replace the circuit breaker with a bigger one, b) hold it in until smoke appears from behind the wall or c) do some serious investigation and/or call an electrician before putting the power back on?
Sample size of two by JoeyRox · 2014-09-06 10:54 · Score: 1

Kind of a small sample size to categorize these failures as "some boards", with the implication that "some" has on perception. Admittedly two mobos failing in spectacular fashion out of the relatively few that have shipped into customers' hands is a bit troubling.
It's not into customer hands... by Anonymous Coward · 2014-09-06 11:11 · Score: 1

It's *REVIEW* Boards. Even assuming the reviewers bought them off the shelves, having two fail spectacularly that were different brands and memory, but the same CPU/Chipset raises some eyebrows.
Assuming the failures were similiar and the non-discrete components along the failure paths were not from the same manufacturer, it would sound like either a design flaw in the reference implementation, or manufacturing defect in either the cpu or chipset.
1. Re:It's not into customer hands... by gweihir · 2014-09-06 16:17 · Score: 1
  
  Actually, the chip-set is pretty irrelevant. The damage observed would indicate that the CPU draws a lot more vurrent, maybe in short spikes, than what the voltage regulators can handle. It may also be that the CPU causes instabilities. Switching power regulators can be tricky, and they certainly are at the voltages (very low) and currents (very high) we are talking about here.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
2. Re:It's not into customer hands... by CaptQuark · 2014-09-06 18:21 · Score: 1
  
  Switching power regulators can be tricky, and they certainly are at the voltages (very low) and currents (very high) we are talking about here.
  Citation?
  
  ~~
3. Re:It's not into customer hands... by Anonymous Coward · 2014-09-06 18:22 · Score: 0
  
  Every datasheet for regulators.
4. Re:It's not into customer hands... by gweihir · 2014-09-08 10:41 · Score: 1
  
  Experience cannot be gotten from citations. Experience is something you have to acquire yourself. For this case here, even an introductory text will warn you, so stop being lazy.
  
  --
  Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Just oil by AndyKron · 2014-09-06 11:24 · Score: 1

It's not burning up, it's just burning off the machine oil.
1. Re: Just oil by Anonymous Coward · 2014-09-06 12:14 · Score: 0
  
  This is only the beginning for Intel's new non-profit motherboard marinade.
did they take the fan off? by Joe_Dragon · 2014-09-06 12:04 · Score: 0

https://www.youtube.com/watch?...
Voltage regulator? by dbc · 2014-09-06 12:10 · Score: 2

From the photos and the write-ups, it looks like a voltage regulator is failing. So, maybe a spec in the data sheet is wrong (for reasons from typo to ooops, we didn't compute that rating correctly...) or maybe a parts vendor for that regulator had a bad-batch day. It happens. Years ago I was involved in one of the latter... "Which date codes do you want us to pull from the parts crib again? I think we have about $2 million of the bad ones...." -- at least that time I was on the customer side, which has much less impact on your sleep schedule.
The Hottest, New Things by Anonymous Coward · 2014-09-06 12:39 · Score: 0

When you're living on the edge, prepare for the best; the hottest of the hot, the best of the best. It's that burning feeling, wowow burning feeling. Oops, it did it again.
expectable by Anonymous Coward · 2014-09-06 12:49 · Score: 0

the board in the Phoronix article wasn't an engineering sample as mentioned multiple times about the author buying the board from NewEgg...
It's not into customer hands... by Anonymous Coward · 2014-09-06 12:52 · Score: 1

the board in the Phoronix article wasn't an engineering sample / review board but the author mentioned about buying the board from NewEgg...
solar flares or ... by Anonymous Coward · 2014-09-06 13:24 · Score: 0

fukunuke fallout made it into the high tech compputer component factory?
also check ur excel table results?
Before transistors... by NewtonsLaw · 2014-09-06 13:46 · Score: 1

In the old days, before computers went solid state, smoking on startup was often put down to worn valve-guides.
Damn... I think Americans call them "tubes" -- in which case the joke doesn't work :-(
So much for the smoke test ... by UnknownSoldier · 2014-09-06 15:46 · Score: 1

.. looks like consumers are on the bleeding edge.
1. Re:So much for the smoke test ... by Anonymous Coward · 2014-09-06 17:12 · Score: 0
  
  I take my hat off to the early adopters, the ones on the bleeding edge of anything new that comes out.
  But, over the years I've learned that if anyone is going to get hurt with the next new thing, it is the early adopters.
  Me, I wait a while.
  But I still thank the early adopters that take the risk the rest of us are too gutless to join in.
  Thank you, all.
2. Re:So much for the smoke test ... by dotwhynot · 2014-09-06 22:19 · Score: 1
  
  I take my hat off to the early adopters, the ones on the bleeding edge of anything new that comes out. But, over the years I've learned that if anyone is going to get hurt with the next new thing, it is the early adopters. Me, I wait a while. But I still thank the early adopters that take the risk the rest of us are too gutless to join in.
  Thank you, all.
  I'm itching to buy a X99 PC build but waiting for exactly this reason. Anybody happen to have any insights on "normal" timing for revised motherboards (rev A/B/C etc.) -- how long it usually takes after launch of a new platform like this before the first minor/major revisions of the motherboards are out?
3. Re:So much for the smoke test ... by UnknownSoldier · 2014-09-07 03:32 · Score: 1
  
  > the rest of us are too smart to join in.
  FTFY. :-)
4. Re:So much for the smoke test ... by messymerry · 2014-09-07 08:53 · Score: 1
  
  > the rest of us are too poor to join in. FTFY. :-)
  
  --
  Dear Microlimp: I give you 2 valid product keys for win7 and you reject both of them. Piss off you wankers!!!
5. Re:So much for the smoke test ... by UnknownSoldier · 2014-09-07 14:27 · Score: 1
  
  That too !
Linux...really by Anonymous Coward · 2014-09-06 17:07 · Score: 0

This hardware has only been on the market for 2 hours with a brand new memory standard. Let's see how well it runs linux.
Moral of the story by Anonymous Coward · 2014-09-06 20:59 · Score: 0

Never be an early adopter of new tech and gadgets.
Let the idiots with too much money to burn leap in and do the beta testing for you.
Manufacturers always make revisions to their motherboards. Firmware upgrades or hardware refinements; you'll get Rev A/B/C or Rev 00/01/02.
New thermal insulation? by skovnymfe · 2014-09-06 21:05 · Score: 1

Aren't these chips covered in Intels new thermal insulation paste? Maybe it's just insulating a bit too well?
Perhaps by Lussarn · 2014-09-06 21:09 · Score: 2

It was a bad motivator?
AMD VRMs have problems too by Anonymous Coward · 2014-09-07 04:52 · Score: 1

Anyone who has a 6 or 8 core AMD FX chip will know the troubles with motherboard makers and VRM quality. If you plan to really use those chips then you better have a board with quality VRMs and proper cooling. If you use water cooling, then no airflow is going over the VRM heatsink. If you use a side to side air cooler, the situation is the same.
Overclock.net has had people complain about this very issue for years.
http://www.overclock.net/t/943109/about-vrms-mosfets-motherboard-safety-with-125w-tdp-processors
1. Re:AMD VRMs have problems too by Anonymous Coward · 2014-09-09 19:17 · Score: 0
  
  Not the same problem as the topic said I think.
SystemD caused it.... by Anonymous Coward · 2014-09-07 05:42 · Score: 0

If they were running Slackware it wouldn't have happpened.....!
for reasons unknown by Anonymous Coward · 2014-09-07 07:45 · Score: 0

Probably someone tried to load an Android emulator and build an app at the same time.