Taiwanese Capacitors Leaking, Exploding

ackthpt writes "A few astute slashdot readers were on to something back when this article was published. After a tip (at e-insight.net) on failing caps over at amdmb I did a little looking around and found this article by Dennis Zogbi on TTI Inc.'s site, which goes into more detail. In a nutshell, many motherboards are now failing due to electolytic capacitors made with an inferior water-based electolyte. Within days or a few months these capacitors build up hydrogen gas and blow the rubber bung out the end of the capacitor, leaking electolyte and causing havoc. The problem may be widespread, as many consumer electronics made with these capacitors may also fail prematurely. Gary Headlee specializes in Abit motherboards, but as his FAQ states, he will work on other makes and the FAQ has more info on capacitor problems."

Comments

Recalls?

[Anonymous Coward]

Good god...how many of these things could be lurking about in automotive airbags, ABS systems, or in any sort of medical device?

Re:Recalls?

[shivianzealot]

Not many, I should hope. Just as we have "server grade" components in the tech industry, other standards exist in different industries.

Re:Recalls?

[victim]

Good god...how many of these things could be lurking about in automotive airbags...

Well I would hope none. The only thing in your air bag should be the sodium azide and an igniter. The last thing you need in an accident is a bunch of loose capacitors and crap being blown into your face.

Re:Recalls?

[zsazsa]

Good god...how many of these things could be lurking about in automotive airbags, ABS systems, or in any sort of medical device?

(Slightly OT)

While not quite as bad, this is becoming a problem in older automotive ECUs (engine control units). The problem is especially rampant in DSM cars (Mitsubishi Eclipse, Eagle Talon, Plymouth Laser) manufactured between 1989 and 1994. I know because it happened to me. All the big electrolytic capacitors leaked all over the place, pretty much destroying the board. One of the symptoms is a 'rotten seafood smell' coming from behind the console. :)

Mitsubishi wanted around $750 for a replacement. Luckily since this is such a problem, refurbished ECUs are available for cheap.

Re:Recalls?

[Anonymous Coward]

Good god...how many of these things could be lurking about in automotive airbags, ABS systems, or in any sort of medical device?

For the most part, none.

Electrolytic capacitors have a fixed lifetime and are by nature unreliable. They don't meet MILSPEC for electronic components. An F16 fighter for example, won't contain a single electrolytic capacitor.

Where reliability is critical, Tantalum capacitors are used, but they're physically larger and more expensive. Any -critical- system manufacturer(automotive safety systems, medical equipment, etc) that uses electrolytic capacitors should be shot.

Want to see some fun stuff? Do a search for "capacitor" and "Bose" on Google. Bose had a TON of problems with bad capacitors in their car stereo systems(installed on millions of vehicles from at least a dozen manufacturers), and it took years for the car manufacturers to fess up to the problem and offer recalls. Bose knew about the problem for years, so one presumes the car companies did. The Ford Pinto incident apparently never taught companies anything; they still don't issue recalls until enough people die/get hurt that it becomes in the company's financial interests to announce a recall.

Some never did announce a recall, until recently. Audi used Bose speakers in the 1991 200 Quattro 20v turbo(car I happen to drive), and the orientation of the circuit boards for the rear speakers were such that the (conductive) electrolytic fluid often caused serious shorts- the Audi enthusiast community knows of at least a few 200q20v's that met their untimely ends due to speaker FIRES. Loud thumping, smoking, popping, etc are more common.

It was pretty much exclusive to the one model, of which only about 2000-3000 at most were imported. The number of reported problems were low(I'd say maybe half a dozen fires), but keep in mind- this wasn't a Ford Taurus will millions on the road.

Audi continuously lied to us- we have proof that they knew from about 1992 onwards that the speakers were defective. A year or two ago we banded together and got a LOT of owners who had experienced speaker problems to call in and report the problem.
When we called Audi, each owner was separately told "we have no knowledge of any problems with your model vehicle"(or something to that effect.) I guess they didn't think we would talk to each other, eh?

Only after several owners submitted paperwork to NHSTA did we see any action; early spring of this year, Audi -announced- the problem and said there would be a recall. It took months to get the replacement circuit boards in and for the recall procedure to get out to dealerships.

Still, guess what? If you report a problem with your car to NHSTA, you can't actually follow up with anyone at NHSTA. The ONLY people you can talk to are a bunch of lazy government call center workers who can ONLY mail you a form or take information on a new case over the phone. You could have evidence of over 50 vehicles that have had said safety problem, and guess what? The call center couldn't care less, they just want to mail you a @#$! form.

Ask any Audi owner and they'll tell you- they love their car. My 200q20v is well over a decade old, but(thanks to a $500 ECU modification) does 0-60 in under 6 seconds, has all wheel drive for incredible traction, 5-speed, stealthy looks(looks like a 5000, basically) and huge amounths of interior and trunk space. Galvanized panels(standard in most audis since 1985 or so) means that there is barely a spot of rust anywhere on the car despite living its entire life in the Northeast US.) Audis are the most utilitarian of german luxury cars, and you usually get more for your money(in terms of features and interior quality) compared to a BMW or Mercedes...and Audi's all wheel drive system is still the best(despite what Subaru would like you to believe, Audi has been doing AWD since 1980, dominating the rallying world at the time. Subaru is about 15 years behind the game.)

But, ask any Audi owner what they think of the company, and the answer will probably change dramatically. It is a terrible shame when such a great product is hampered by piss-poor support.

Re:Recalls?

[Rich0]

Just as we have "server grade" components in the tech industry, other standards exist in different industries.

Hmm - I take it the server grade components only blow up after the server is linked on slashdot?

Re:Recalls?

[Rich0]

The only thing in your air bag should be the sodium azide and an igniter. The last thing you need in an accident is a bunch of loose capacitors and crap being blown into your face.

Actually, I would rather be hit in the face with a bunch of capacitors than aerosolized sodium azide, which is highly toxic by US definition, and is about as healthy as sodium cyanide powder. It is commonly used as a laboratory preservative since it can kill just about anything...

The NaN3 and ignitor are not actually in the air bag - they are in an inflater, with a filter so they don't end up in the air bag.

Re:Recalls?

[sigwinch]

I think that's overly harsh on electrolytics. Like everything else, what you get depends on what you buy. You can buy from respected companies that have been making good caps for 20 years, or you can buy from whatever random Chinese company was cheapest this week. You can settle for any specs you can get, or you can insist on caps that are rated for 5000 hours of operation at 105 degrees Celcius (hotter than boiling water!).

There are also system design issues. You can push the caps to the very limit of their rated ripple current, or you can use more caps and share the current around.

Good god...how many of these things could be lurking about in automotive airbags, ABS systems, or in any sort of medical device?

For the most part, none.

Medical stuff routinely uses electrolytics. It doesn't have to be perfect, it just has to fail a lot less often than doctors and nurses.

Electrolytic capacitors have a fixed lifetime and are by nature unreliable.

They do not. The lifetime depends on the grade selected by the engineer, and how hard the design pushes the cap. A good cap used properly can last for many years of continuous service. That's good enough for many applications, even in safety-critical systems.

Where reliability is critical, Tantalum capacitors are used, but they're physically larger and more expensive.

You can't be serious! Tantalums are notoriusly flaky. Not only that, the usual failure mode is that the cap vanishes in a spectacular flash of purple fire. Every capacitor failure I've ever seen in computing equipment has been a tantalum. An engineer who used to work at Motorola told me that tantalums were banned from pager designs. At the time, Motorola would rather pay the premium for ceramic caps than risk tantalums.

Any -critical- system manufacturer(automotive safety systems, medical equipment, etc) that uses electrolytic capacitors should be shot.

It depends entirely on the service life that is needed, and the degree of redundancy you can afford. Satellites and airbags have to remain in service for decades without repair, so electrolytics are probably unacceptable. Medical equipment generally doesn't need such high reliability, and frequently uses electrolytics. (Seriously. Med equipment is regularly replaced, there's no point in making it more than a couple of orders of magnitude more reliable than physicians, and the critical stuff has spares sitting on shelves.) Telecom equipment can afford redundancy in almost everything, and so it's full of electrolytics.

Re:Recalls?

[plover]

While you are correct in your assertion that tantalum capacitors are (or at least should be) used where reliability is paramount, there are severe environmental issues with much of the tantalum mining going on today.

80% of the world's supply of coltan, the ore from which tantalum is extracted, is found in the Congo (formerly Zaire). Illegal mining has caused large sections of Kahuzi-Biega National Park and the Okapi Wildlife Reserve forests to be cleared. The U.N. has determined that 90% of the gorillas in these parks have been killed in the last five years, primarily by the illicit miners, leaving only about 3000 remaining. The miners kill the gorillas as a threat to their safety, and sell the meat for extra income.

It's not an easy problem to solve: coltan sold for upwards of $600/kg in the late 1990s (although it's down to about $100/kg today.) A team of miners can produce about one kg per day, and each miner can earn about $200 US/month in a region where the average income is $10 US/month. Technology's demand for tantalum has driven the poorer residents of these nations to take the easy money where possible. And according to the U.N., both sides in neighboring Rwanda have been funding their civil war in large part by sales of this illegally mined tantalum.

What can be done? It is reported that deposits of coltan have been found in dormant volcanos in Greenland. And there are legitimate mines elsewhere in the world. MoBo manufacturers can and should agree to purchase and use only "gorilla safe" or "non-Congo" tantalum caps (or they can continue to use better quality electrolytic caps.) But this will only occur if the demand for gorilla-safe tantalum crosses some magical political threshhold. I wish I knew what it would take.

Re:Recalls?

[chriso11]

I use tantalums instead of electrolytics as a rule. Electrolytic caps have a higher ESR, and the dielectric fluid is prone to leakage. The lifetime of an electrolytic is shorter than that of a tantalum - they have a finite lifetime, as the liquid electrolyte evaporates.
While it is true about tantalumns having a particularly impressive failure mechanism, once you remember not to reverse the polarity you don't have problems. Electrolytics also behave poorly when reversed. In addition, electrolytics have a minimum operating voltage and a maximum operation voltage. And how's this:
"Conventional aluminum electrolytic capacitors which have gone 6 months or more without voltage applied may have to be reformed."
Electrolytics are also physically larger than Tantalums, not smaller. They are indeed more expensive, but worth it.

Yes, I use ceramics whenever possible (esp. NP0/C0G, none of the x7r or worse grades). Ceramics are the best general purpose - no polarity issues, small physical size allows them to be extremely close to the DUT (for bypassing), and they have pretty good SRF. The only problems - the max capacitance you can get isn't too good. Polyprop/polystyrene are better for high fidelity audio type signals (earthquake detection anybody?).

Re:Recalls?

[Tangurena]

There were an enormous number of electrolytic capacitors that went boom in Car audio systems. Surprisingly, it was only the capacitors made by Nichicon and almost always the 7mm high capacitors. Pity, nichicon was the low bidder on almost everything, and they were the only supplier in the world of 7mm high axial electrolytics.

A previous employer of mine made somewhere near $10,000,000 in repair costs (both in-warranty and out of warranty) because of these capacitors. Radio went poof? $250 repair please. Bose Amp squeals like a siren and pops like a canon? $200 please.

Ford could not believe that our repair shop needed as many capacitors as we were using, and sent out auditors and engineers to get a grip on what they were convinced was out of line repair expenses (or maybe outright fraud). When we showed them radio after radio, and Bose amplifier after another with exactly the same failure mode, they started waking up. We even gave them boxes of ruined circuit boards for them to analyse. The real kicker was the Bose amp used in Chevy Caprices: the board is mounted so that the capacitors are suspended from the board, the electrolyte boils and spurts out of the base of the caps so hard that it splatters all over the board. Once they saw these, and learned how they are mounted in the vehicle, they went after nichicon.

Because of the size of the part, and that nichicon has a stranglehold on the market, we had to order parts directly from them. When you need 2,000 to 4,000 per month, you use them far faster than the US car makers ordered for replacement parts. However, instead of ordering the 65C rated parts, like the OEMs used, we ordered the 105C parts. Still took 12-20 weeks to send the boxes from Japan to Florida. That is real fun committing your employer to buying stuff for a year at a time and having to wait months for each delivery.

Because of heat and humidity issues, the south florida climate accelerates the aging process for these parts. What fails in 2-3 years down here, may take take much longer for you folks who live with frost. Heck, car batteries only last 2 years before they need to be replaced.

so...

[iamthemoog]

there's magic water in capacitors as well as magic smoke ?

My MSI board failed.

[dsb3]

My MSI board failed a couple of months ago, and we didn't have a dog to blame the smell on.

I noticed many of the caps around the memory banks appeared blown - there was a lot of brown residue around the top. The smell occured a week or so (perhaps?) before final failure.

For my money, even though the original board cost around $120, I just bought a $50 replacement from ECS. It took most of the original memory (2 DIMM slots only, compared to the 3 slots in the original), and otherwise did what was needed without spending repair money on what's now an old-tech product.

The machine has an Athlon 900 T-bird, now has a 1/2G of ram (did have 3/4) and doesn't really do a great deal other than email, web, games, photoshop. Sure, the extra 1/4G of ram would have been nice to keep but for the money of even thinking about the repair I'd be better off just recycling and buying new with a DDR333 system.

Once again, technology is cheaper to replace/upgrade than it is to repair.

Cheap capacitors

[HotNeedleOfInquiry]

Nothing new about the annoucement. Cheap electrolytic capacitors have been around and been a problem for years. There are other failure modes. i've fixed several old Mac's where the cap has pissed it's electrolyte all over the motherboard. Usually removing the cap, scrubbing the board and installing a new cap fixes the problem. Even worse is when the electrolyte is lost gradually. The product that it's in gets flakey over time and the problem is very hard to find. These problems are all made worse by exposing your gear to high temperatures. Never leave your electronics in the passenger compartment of your car in the summer.

Not the only problem

[Bobulusman]

It seems that motherboards in general are being made more poorly lately. Last April I bought a Soyo Dragon Plus motherboard. It has been give me and others problems. Apparently, they screwed something up because the board is not technically PCI compliant on the top two pci slots. So basically, if you use the AGP slot and either of those slots with anything more taxing than a modem, you will be riddled with reboots and the like.

Not to mention that there is something else screwed up with the board because the MadOnion benchmark always identifies it as having twice as much ram as it does (I have 512 mb on two 256 mb's. It thinks I have two 512 mb's) and it can not seem to complete the PCMark test without rebooting during the ram tests. This has happened to other Soyo Dragon Plus users, so it's not like it's just the software.

And don't even get me started on how they ripped me off by not bothering to tell me that they would not give me the accessories needed to make various functions work. Had to by them seperately....

Same case with the motherboard I bought before that.

Anyway, my point is that it just seems that MB manufacturers are cutting a lot of corners, so it doesn't surprise me that they are using cheap capacitors.

Your motherboard is next

[jmcwork]

many motherboards are now failing due to electolytic capacitors made with an inferior water-based electolyte.

Early indications of capacitor/motherboard demise include failure of spell checking software.

Isn't Anyone Doing Qualification Testing?

[Anonymous Coward]

Yes these are obviously bad components but I am curious. Do consumer electronic manufacturers do any type of development validation or component qualification testing?

In the automotive world, this would have been caught way before production started, unless of course, the component supplier changed the electrolyte type without notifying its customers after start of production.

The amount of testing that occurs on automotive electronics is sometimes thought of as gross overkill. When I hear stories like this, it reminds me of why.

uh oh

[tps12]

If you're trying to boot up, and your rubber bung breaks and leaks electrolyte, then I'd recommend getting tested as soon as possible, especially if you were trying to boot from a strange floppy.

We got hit....

[IpSo_]

The company my parents own bought 30 machines a while back which apparently all had these bad caps on the mainboards. When the first few mainboards started failing we tried to send them back on warranty, but our vendor wasn't cooperating, and shipping them all back to ABIT was resulting in too much down time. (shipping time, etc...)

So we went to the nearest electronic wholesaler in town and bought a box of the equivilent caps and soldered them on ourselves. It doesn't take more than 5 minutes and the caps themselves are very inexpensive.

Of the 30 machines we bought I think almost 25 have failed, just a matter of time before the rest fail I'm sure.

blame the gangs!

[misterhaan]

they're always threatening to "pop a cap," and now this starts happening! coincidence? i think not!

How to check before you buy

[Jeffrey Baker]

It is easy enough to check this before you buy. Go up to the display case at your local computer parts retailer and ask to see XYZ motherboard that you are thinking of buying. Jot down all the markings on the electrolytic caps. Now go home and look up the datasheet for those caps. A good computer grade capacitor will have longevity of 2000 to 3000 hours or more at maximum ripple current and a temperature of 105 or 125C. Reputable brands are Panasonic HA or NHG, Rubycon, etc.

Forget case mods, maybe we need to start modding our mainboards with better caps.

Re:How to check before you buy

[Jeffrey Baker]

3000 hours is actually a very good rating for an electrolytic capacitor. This rating means that the capacitor will operating within specs after being subjected to its maximum ripple current at maximum working voltage and maximum operating temperature. Electrolytic capacitor lifetime is most directly related to temperature. Panasonic TS-HA types for example will last 3000 hours at 105C but will last 200000 hours at 45C. So, keep the case temperatures down for high reliability.

This is why you should always buy quality.

[Prince_Ali]

It is always important to buy quality components. I for example bought a *shuffle shuffle* Abit... BH6. Dang. Excuse me I have to go check something.

My experience with failing Abit motherboards.

[Lethyos]

I can finally vent about my misery! This experience ought to be useful to anyone who is currently in the market for a motherboard. Simply put, don't buy Abit.

About 1.5 years ago, I purchased two motherboards from Abit. This one for an 800MHz Athlon system, and this board for a dual, 733MHz Coppermine system. Last semeter, my KA7 failed slowly over time. At first, I thought it was the power supply because it seemed all the capasitors around the power regulator were fried (they were encrusted with the carbon of some substance that appeared to boil out of them and burn). I replaced my power supply and motherboard. A few weeks ago, I started having interrupt failures on my VP6 (APIC errors on both CPUs). I replaced the motherboard with a Gigabyte GA-6VTXD (sorry for the shameless plug, but Gigabyte denies deep linking, and this is where I got the board - a great buy). Turns out the VP6 also had fried capasitors and I *know* the PS in my that dual proc box is solid (a well tested Antec). The only two Abit mobos I've ever purchased burned out their capasitors. The moral of this story? Don't buy Abit. While this problem is wide spread, Abit seems to have a particular affliction.

Magic Smoke

[Hollinger]

I suppose they're letting the magic smoke out prematurely.

(Lifted from the Jargon File)

Magic Smoke - n. A substance trapped inside IC packages that enables them to function (also called `blue smoke'; this is similar to the archaic `phlogiston' hypothesis about combustion). Its existence is demonstrated by what happens when a chip burns up -- the magic smoke gets let out, so it doesn't work any more. See smoke test, let the smoke out.

Usenetter Jay Maynard tells the following story: "Once, while hacking on a dedicated Z80 system, I was testing code by blowing EPROMs and plugging them in the system, then seeing what happened. One time, I plugged one in backwards. I only discovered that *after* I realized that Intel didn't put power-on lights under the quartz windows on the tops of their EPROMs -- the die was glowing white-hot. Amazingly, the EPROM worked fine after I erased it, filled it full of zeros, then erased it again. For all I know, it's still in service. Of course, this is because the magic smoke didn't get let out." Compare the original phrasing of Murphy's Law.

Re:Product of globalization

[Usquebaugh]

Rubbish. What are you xenophobic?

Unless you buy a known grade of electronics you have no hope of getting stable, reliable kit. American or otherwise. Quality costs period.

If you've used a cheap board for a mission critcal server then who is at fault? It ain't the supplier.

My mouse exploded.

[mypalmike]

I had a Logitech Cordless Mouseman Optical which only lasted about a month. Then, at one point, I was just mousing along, and suddenly my mouse made a loud bang, which I heard, felt, and saw. Man did that scare the crap out of me. This was about a year ago, before the whole D.C. sniper news, but my first thought was seriously that a sniper had shot the mouse out of my hand.

Anyhow, I ended up deciding it must have been a buildup of gas leaking from the batteries. However, now I'd bet my money on a capacitor exploding, since it still kind of worked after that, but mouse control would be spastic, possibly indicating failure in voltage regulating circuits.

-_-_-

Tanatlum shortage

[huie]

Do a Google search on "tanatlum shortage" and you'll see that there was a flury of articles about a year and a half ago. This prompted development of other electrolytic capacitors, one of which is the aluminum electrolyte that seems to be having problems.

I assume that it's only taken this long to find the problem due to the development time and time to qualify (ha!) and integrate these new caps onto boards. Needless to say, I guess they needed to develop the caps better, but they may have rushed to market since there was little else available (at a decent price).

Re:Tanatlum shortage

[Flakeloaf]

It wasn't what I'd call a shortage exactly.... the suppliers definitely *had* the Tantalum, they'd just wave it in front of my nose & refuse to sell it.

Computer Grade

[Detritus]

In the good old days, when mainframes ruled the earth, you could order "computer grade" electrolytic capacitors from electronics parts companies. I never saw a definition of what exactly "computer grade" was, but they were noticeably more expensive than generic electrolytic capacitors.

Part of the problem may be that the engineers are underspecing the capacitors in an effort to cut costs. A friend of mine used to have a job evaluating component reliability. He had lots of graphs that showed reliability as a function of how hard the component was driven in the circuit, for example dissipating 5W in a 5W transistor instead of using a beefier transistor.

Real Culprit is Chamelionic Mycrobe

[bstadil]

Remember Holly from Red Dwarf?

[A plain starfield. Narrative text draws across the screen:]

"Lone escape pod from SS Hermes - Survivors one.

Ship destroyed by Chameleonic Microbe."

[A pause, then the words 'by Chameleonic Microbe.' are deleted, and replaced with:]

"by Chamelionic Mycrobe."

[A second pause, then 'by Chamelionic Mycrobe.' is deleted, and replaced with the much simpler:]

"by shape changing weird space thing.

Non essential electrics all down, including spell checker.

Massage ends."

Always Suspect Electrolytics

[ONOIML8]

I've worked as a professional electronics technician (radio communications) for 13 years, and grew up around the industry. The one thing I learned early on is to always suspect the electrolytics.

Any electrolytic will change value with age, they simply dry out. Change it enough and the circuit either quits or is way out of spec. But I'm talking about caps that are 20+ years old. It seems like caps made back then could hold up for that length of time.

I've noticed in newer equipment that the caps just don't hold up. This seems to be a trend in the last 10 years or so. Everything else like diodes, resistors, transistors, etc. holds up just fine as long as you don't exceed engineered values in the circuit. But caps, anymore you just cant rely on an electrolytic to stay within spec for more than a year or two.

All this time I thought it was just me and my bad luck. Guess not.

Note that I'm not talking about just computer equipment here. Most of my experience is with land mobile radio, power supplies, and telephone equipment.

If your switching power supply in your computer has gone on to the afterlife, and the fan still worked (they won't take heat buildup)......I'll lay odds it was a cap that croaked.

Re:Product of globalization

[Velox_SwiftFox]

What a comfortably short memory you must possess.

The reason electronic parts manufacturers in the United States lost market share to foreign makers in the first place was the shoddy workmanship of the US companies' products. Like the US Auto companies, they exaggerated the importance of their own prestige and assumed that this guaranteed competition did not matter, so they inflated their profits by shipping bad parts. The machine control manufacturer I worked for in the 1980s, when trying to purchase parts, would receive shipments that fit into two classes: The ones where the 10-15% non-operating parts were scattered throughout the shipment, and the other variety where the manufacturer had tested the parts, then placed what they already knew were the bad ones in the bottom of the cartons in the hope that they would thus slip by incoming inspection.

It was not until foreign companies began to supply the parts as well, usually with failure rates so low the incoming inspections were no longer necessary, that the US companies realized they could no longer get away with this crap, and began to get their own act together.

Worst case scenario: protectionism placing non-US manufacturers under a handicap with regards to US electronic parts makers - inevitable result would be the domestic suppliers slacking off on their quality again.