Slashdot Mirror
Zinc Whiskers Cripple Colorado's Computers
Mr. Christmas Lights writes "While zinc whiskers, small metallic fibers which grow on surfaces that have been electroplated with zinc, aren't a problem for Christmas lights, they can cause serious problems for computers. The Denver Post reports how they caused computer outages for the last three weeks in the Colorado secretary of state's office. This basically halted business and elections document filings. Zinc whiskers are becoming more of a problem as computers electronics get smaller. NASA has a good reference site which includes a interesting PDF summary paper complete with pictures. /.'ers with computer rooms might want to check this out."
(rolls eyes)
Where I used to work, we had this issue - in our case they grew from the cheap computer floor panels in the room. The case was so bad, you could see them in direct sunlight, and the flowed in the breeze like grass.
We had no choice but to go through cleaning, as the underfloor was about to be used for blowing air to new systems, without it, the zinc whiskers would blow free and cause hell on all our systems. As it was, three systems failed in the week after the clean. We don't want to think what would have happened if we didn't clean it.
It's not bullshit. Get over it. Interestingly, there are very few people who know of this issue, but knowledge is spreading.
Robert Anton Wilson
We are in the final streach for replacing the entire datacenter floor where I work because of zinc Whiskers. We had the underfloor area cleaned for the new A/C that blows from the floor up. We promptly lost 11 power supplies and the total count is up over 20 now. We did verify the presents of zinc whiskers in the dead powersupplies thanks to the onsite electron microscope.
Thankfully the high temperature never got above 85 degrees so the old A/C was able to keep up.
Yup.
And the metal that the whiskers come from, have been used for computer room floor tiles, racks, and even (shudder) PSU cases.
Now that's scarey.
Robert Anton Wilson
And are a problem with the new lead free processes.. especially as lead spacing decreases, and the euro lead-free requirement kicks in.
Agere wrote a good article in Analog Zone, available at http://www.analogzone.com/grnt0216.pdf. It has a good micrograph showing the problem.
"'Tis great confidence in a friend to tell him your faults, greater to tell him his." --Poor Richard's Almanac
No kidding... I mean, if they would have said that they were shorting out the CPU's data bus, or something along those lines it might be believable.
But the POWER SUPPLIES?!?
A microscopic fiber of zinc (a metal with a _very_ low melting point) would not short out ANYTHING with more power than a hearing-aid battery!
As another pointed out already, one of the people interviewed is trying to sell their services cleaning it up!
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
We have 3 rooms where I work that house our servers. We are migrating some stuff over to an HP-UX system running oracle. When we had some HP representatives come over to our area to check out the place we intended to house the HP servers, they insisted that zinc whiskers could potentially be very damaging and that we make sure our rooms were clean and free of zinc whiskers (not that our rooms were particularly dirty). I always wondered how legitimate their claims were.
Dusting, god no. The last thing you want to do is get these fuckers airborne, that's when they start to cause problems. They get sucked into a tower or rack by the air convection, and wind up settling into places they shouldn't. Next thing you know, you have equipment shorting out.
Forget the duster. Cleaning will probably involve mineral oil, or some other decently viscous but harmless liquid, being sloughed across the floor, sponged up, then washed away using conventional cleaners. The oil would weigh down the zinc whiskers to the point that they couldn't get into the air and cause problems.
My A-100's chorus/vibrato likes to go on summer vacation when it gets humid in the house, but works fine during the winter. Just thought I'd throw that in - I think the stuff is zinc-plated...
I so agree with you.
back in college I paid my way as a maintaince guy at a foundry. we had rackmount Pc's that would have almost 1/4 inch of metal/sand dust on the motherboards and the computers were STILL working.
Cince then I have worked as a freelance consultant and specalist for many different companies that certianly do not have their computers in a "clean room" (machine shops for the best example) and they never EVER have these problems and they are exposed to nasty air + metal. The worst was a water filtration plant I worked at for 7 years where a workstation for monitoring the vats of hydro-flouro-sisicilic acid ( what they put in your water to add flouride) a product that is so corrosive that it eat's through the rubber lined fiberglass tanks within 3 years had, just by the amount released in the air during tank inspections, had eaten almost all the legs off the surface mount chips on the motherboard and it was STILL operating.
Maybe some really REALLY old mainfraime computers might see the problem in a 20-30 year lifespan that the article suggests, but even the PDP-11 I saw back in college that was retired in the basement but still maintained operating by students did not have any problems like this.
Do not look at laser with remaining good eye.
...small compact computer designs get these whiskers easier. I don't doubt it happens, but where are all the millions of laptops shorted out then, or the mini itx machines,game machines, etc?
Is there something else here causing whiskers to grow some places and not in others, even though both have zinc?
Wohoo first post ever on slashdot!
Anyway, I work at an IBM data center with many computers and sensitive equipment around. We have zinc ducts here(well they look like zinc so I assume they are), and in the 15+ years this place has been running no problems have become of it as far as I know. That is no problem that can be traced to it. I suppose it is possible some problem that was blamed on something else could of been caused by this, but I've never heard of it.
Just some info.
That is a myth. "Magnetic fields and magnetic storage media do mix, as long as the field isn't really amazingly strong. The field has to be really strong, because it has to exceed the coercivity of the magnetic coating on the storage device. Hard drive platters have a coercivity of a few thousand Oersteds, which means a field of the same number of Gauss is needed to demagnetise them. The ferrite magnet on a computer's PC speaker, assuming it's not shielded, will have a surface field strength of only about a thousand Gauss, so it won't endanger hard drive data even if the drive's right next to it. Even 10,000 Gauss rare earth magnets can't wipe a hard drive if they're not sitting on top of it."
Are you a VF grad? Check out the VFMA Alumni Forums VFMA Alumni Forum
If these things are so small as to require an electron microscope to see, why arn't they simply vaporized/melted as soon as they find themselves shoved between two (relativly) massive wires?
You'd think that a microscopic piece of zinc would go before a macroscopic fuse/chunk of copper/etc. And since it's alot of single pieces blown around, it's not like several million are all going to do it at exactly the same time.
I've melted zinc, and it's pretty snappy, (Pennies after 1982 are mostly Zinc. When you melt them, you get a cool copper-skin effect going on.), but with the same torch ($15 propane torch) I was unable to even visibly affect the copper.
...green dots are an audio solution. This smacks of freakish pseudoscience masquerading as real problems. Kind of like the gold plated power cords for stereos that cost audiophiles $180 each. The supposed reason that they improve sound is that gold has less resistance and therefore your amp will perform better if the electricity gets to it more readily. That's bullshit just like the zinc whisker issue.
Un-news
I'd be a lot more inclined to believe the "whiskers" are coming from movement of the tiles when people walk across them(they do shift, as does the frame slightly) and not some "growing whiskers" BS.
Furthermore, the problem is easily solved via any/combination of these:
Furthermore, if the little buggers are metallic, why don't you just install a few small but powerful magnets in various ducts? A metal grate made up with a set of magnetic rods would probably work like a swell charm, and only require periodic cleaning...
Please help metamoderate.
No, I'm not worried about zapping floppies. Your ideas are, for the most part very good ones, but zinc is paramagnetic, if I recall right, and is not attracted to the magnets. Same for tinfoil, BTW. The magnets would be effective in catching iron filings and related ferromagnetic particles, though, but that didn't seem to be the reported problem in this case.
Your friction hypothesis has merit, too, but growth of whiskers, more scientifically known as dendrites is actually quite common, especially where electric fields exist between conductors. I ran into that in a flexible touch keyboard we had designed using a silver alloy that was screened on as the conductors. Durn things would develop shorts after a while in the field, literally since it was on agricultural equipment. You couldn't see the shorts, but examination under a microscope revealed those nasty little whiskers. A metalurgist was consulted and provided a different alloy that solved the problem.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
1. I first like to thank simoniker for adding the "small metallic fibers which grow on surfces that have been electroplated with zinc" to the article - made it more understandable/readable.
2. The NASA URL is one-level deep (a mistake on my part) - here is the top-level.
3. Related to #2, I would STRONGLY recommend /.'ers actually READ what that says. The Denver Post article was written by a reporter - would you expect that to be technically accurate/broad/etc? Again, take a look at the NASA site which DOES present a compelling case that this is a REAL issue and not FUD. The original study with the medical equipment makes for facinating reading.
4. Some Anonymous Coward seems to have a problem with my nickname. Did you actually click on the "Mr. Christmas Lights" and see what is there - tell me that isn't appropriate (it's been used before BTW).
5. The same AC made a smart-ass comment about the Nigritude Ultramarine SEO contest - while I'm aware of that contest (#4 above is a hint for 'ya!), I'm currently ranking #199 for the keyphrase with less than a week to do, so I'm not a contendor ... although I do rank #1 for the phrase Nigritude Ultramarine Hulk! ;-) ... and I actually did submit a wrapup article a few days ago about this, but it got rejected - good news is the contest is over July 7th, so all those N-U links will go away - they are a bit annoying.
6. I haven't seen anyone comment on a business (verus technical) aspect of the Denver Post article (but this is /.) where some state mucky-muck basically says this is a reason to bring all state websites under one authority and talks about $7.5 million in funding. One wonders if some empire building going on and/or play for more money!
7. There have been several Denver Post articles about the failure of these computer systems. I didn't mention that fact in my submission because I thought it would be too lengthy, but apparently the inability to electronically check/file business/elections stuff has been a real big deal - good example of our dependancy on computers.
'Nuff random late night rambling!
Hulk SMASH Celiac Disease
Metal whiskers aren't usually a problem on the mobo itself (everything is conformal coated) but on exposed metal -- surface mount devices for example -- and especially in power supplies. And even then, it's only a problem if you can't take (or blow through) a 50-300 ohm short every now and then. Newer equipment is a lot more sensitive -- denser boards, less slop in timing and signals, etc ... and of course manufacturers cut costs wherever they can, even on networking and server equipment. Unless you can afford NEBS or industrial grades, "server grade" rackmounts aren't necessarily any more rugged these days than consumer grade crap.
But it has been a problem outside of data centers, especially where you're looking at small (analog) signals with high input impedence. Examples include medical monitoring equipment and scientific research equipment, and it's why you don't see established manufacturers of either using uncoated, unalloyed zinc electroplating, especially in humid environments.
we had rackmount Pc's that would have almost 1/4 inch of metal/sand dust on the motherboards and the computers were STILL working.
Contacts that are exposed to a "harsh" environment generally form a very thin non-conductive film due to oxidation and/or corosion. This would also apply to any conductive surfaces. Depending on exactly what gets laid down, what you describe seems completely reasonable. However, something in a "clean" environment can easily be killed by something your self-protected PCs wouldn't even notice.
I live within a mile of the ocean, and we have a lot of fixtures in our house which must be made of zinc, because they grow these whiskers just like were described. We have a chandelier in particular which looks like it's brass, but it's always covered with fuzz. Then I have a chin-up bar in a doorway and the same thing happens to it.
I wonder if it has to do with some kind of electrochemical reaction, where maybe there have to be different unlike metals with varying electronegativity, and enough humidity to get a low grade current flowing between them. I never saw this problem, when I lived elsewhere. But if my computers had zinc in them I'm sure it would grow whiskers just like the rest of my house.
Extraordinary claims require extraordinary evidence.
Risk = probability * loss. Since loss is obvious and fairly constant (a flaky or inoperative computer), if you want to hype this risk, besides just showing clear evidence that zinc whiskers are increasing the probability that a computer will die, you have to show to what degree that probability is increased - that is, that it merits more concern than other obscure things that can cause a computer to die.
Well you better start doing your homework.
I worked for many years on a replacement Air Traffic Control System for Canada and as the project matured, our stage - lab (containing litterally hundreds of machines, a complete lab recreation of the coast-to-coast ATC system) started to experience an MTBF on the power supplies in the equipment that was over an order of magnitude smaller than spec'd by the manufacturer (Hewlett-Packard).
Since this was a long-term contract that included commitments to deliver over an extended period of time (25 years), the material cost of this problem was VERY significant to the equipment vendor, not the customer. (In other words there was no financial motivation to fail to find fault, quite the opposite; the fault was costing them money.)
In the spirit of "old HP" they sent us some senior hardware design guys to look at our lab and our environmentals (humidity, temp, pressures, cycles and power suppy spectra) to see what was causing the problem.
Being about 6 years ago -- I hadn't heard of the Zinc problem yet, and neither had the guys from HP. They took everything back to their labs, including about 6 failed supplies and a couple 'still good ones', some from reserve stock and some from working machines.
A few weeks later they came back; there was a big meeting -- this was an issue with potentially enormous cost -- including the ultimate customer's representatives.
I can remember the Project Manager practically spitting his coffee when informed the underlying cause. The 'special ESD safe A/C'd lab' was part of the problem. Thank fully, the final deployed environment had different flooring, so we didn't have to change the sites, just some modifications to the lab.
This is far from BS -- it's a problem that has cost millions and will likely cost millions more before it's over. But the SEM photos of the failed devices we cool to see.
Switching power supplies can be surprisingly fragile. I've killed a couple working on TVs (that's basically what a flyback circuit is) and you can do it in one or two cycles (of your AC, not CPU cycles). And no, you don't want to know how much those power transistors cost. And I've killed computer PS by shorting across IC pins. There's not a lot of current going through these, and a 50-200 ohm short will definitely do the job. Remember, it only has to conduct long enough to nuke the chip.
Or maybe you'd prefer to ask the Cisco power engineer about it. Naah, he probably doesn't know what he's talking about.
I love this kind of stuff. Clueless moderators mod the parent insightful because none of them happen to have any personal experience with the subject. Lack of personal evidence does not constitute a conspiracy. Really, it doesn't.
/. need for anecdotal evidence, yet you groundlessly claim that the articles anecdote MUST have some other explanation. Why? I am all for being a little skeptical, but there is such little reason for it in this case. Would you begrudge them the opportunity to clean out their under-tile areas? Most of us know how bad those areas can get regardless of the whisker issue.
I have personally known about whiskers for over five years. It was becoming a potential problem in an old datacenter at work. It is a serious condition that datacenters with critical machines (or contractual obligations) take into account in datacenter design and maintenance. With 5 9's required for a lot of machines (Hitachi, IBM, Unisys, etc) there is little room for allowing electrically conductive dust particles to flow across every board on your machines.
The other posters have given examples to satisfy the typical
_damnit_
It's my job to freeze you. -- Logan's Run
The preferred Zinc compound for mineral supplements is usually Zinc Picolinate, which is the form most readily metabolized by humans. Zinc Oxide is frequently the form used in mixed multi-vitamin and mineral tablets, but some nutritionists argue that the manufacturers are cutting corners to save a few cents there.
Zinc Oxide is also absorbed very, very slowly through the skin (as when used in sunblock cremes, so that most of the sunblock's Zinc is washed off instead of entering the body). Pure Zinc metal is not nearly as reactive as sodium or calcium, so it really isn't very toxic compared to them. People eat pennies frequently (4 year old people that is), with few harmful effects, although pennies are mostly zinc with a copper surface.
For comparison, a significant amount of pure sodium won't just give you chemical poisoning, it will oxidize so rapidly with contact with air or water, and liberate so much heat in the process, that the burns you would get from handling or swallowing it would probably kill you as fast as the toxicity, or even quicker.
Yes, Zinc metal fiber in a hospital environment isn't nearly what I would call perfectly safe. Pennies usually pass before too much dissolves, and getting a lot of metal fibers inside a surgical incision before closing is not such a benign environment. Still, there are a lot worse possibilities for substances to get in a wound.
Who is John Cabal?
The real reason for gold plating has more to do with oxidation than the resistance of the different metals. Dending on where you live, those connections can develop problems fairly quickly. I also had someone point out that they paid a LOT of money for their equipment and don't want any dissimlar metal issues. While I'm not sure about the odds of that being an issue, who am I to quibble with him over a 100 dollar cable to hook up 3000 dollar rack equipment?
_damnit_
It's my job to freeze you. -- Logan's Run
problem happens in copper wiring small copper "whiskers" grow right through the insulation. if two wire are close when this happens they can short out and cause fires. usually only happens in really old wiring
You'd think that, wouldn't you....
In one of my jobs I got to set up an $800,000 prototyping shop with nice CNC equipment and all that. One of the toys I bought was a 350Amp Synchrowave welder. It was hard wired into the 460V main service we built into the building, all brand new freshly installed from the 10kv transformer to the disconnects. 600Amp 3 phase 460V service just to my little shop.
First night there I figured I'd try the welder out, flipped the disconnect, hit the start button on the welder and poop, out the lights went go. I turned the disconnect off, checked the fuses - they were fine. I went out to figure out what had happened. The 600A service breaker was popped.
I figured the welder had been wired wrong, opened the case. It was wired to the disconnect with, well, welding wire - stranded copper about an inch around. One - ONE - little tiny strand had unraveled and shorted across two phases. I bent it out of the way, buttoned up and it worked fine for the next 3 years. The main breaker never tripped again, not even when I was using the welder to blow holes through aluminum plate just for fun.
Now there's no way that little tiny wire could take 600A at 460V, and I can't think of a plausible narrative to explain why the main breaker tripped - but it wasn't just floating - the CNC had been running and it had a 10HP spindle.
A whisker doesn't have to survive conducting enough current to let the smoke out of the power supply - in it's incandescent passing, if the ghost of it's exisistance is a sliver of plasma, a very substaintial, if evanescent, conductor is created, literally, out of thin air.
Hi, It just so happens that I work with this guy and he his correct in his statement. In fact my mom is the electron microscopist on site. Please note this is a VERY OLD electron microsope. around 8 - 10 years old. It doesn't have the ability to look at the lattice structure of a molecule because it simply isn't powerful enough. In our case the zinc whiskers where in the hundreds of microns in size. There is an attachment to the machine that can give you a breakdown of metals in the sample you are analyzing. In our case it was very obviously zinc. We have had problems with our new Sun equipment popping power supplies for several months nows (V480, 420R, etc.) although none of the Ultra 2's have had a problem at this point. We thought it was heat and power (thus the new AC unit for cooling) and the power checked out ok. The sun rep said we are the only company we service that was having supplies pop (in fact the data center across the hall wasn't having any issues but they have different tiles than us) when we had the cleaning crew in to clean under the tiles (prep for the under floor AC unit...note that we are using ceiling air now) In the span of 3 hours during the cleaning we lost 13 supplies all on new sun equipment. Once they stopped running vacs and pulling tiles it dropped off. Sun confirmed (with a sample of five dead supplies) that they were all caused by a short. Our tiles were wood core with hot dip galvanizing that had been recycled from an old computer room. These tiles had 20 years plus of use.
Given it's a recently understood phenomenon (in data centers), I doubt anyone can quantify the risk. I certainly can't. I'm not trying to hype a risk, I'm trying to dispute repeated claims that this is bogus or exceptionally rare. I certainly don't have any financial interest in this, I've got better things to do with my life than vacuum wood-core subfloor panels.
Oh, and metallurgy isn't my current field, but despite that I've run across this personally. So I started reading, and was amazed to find out how well known it was given I'd never heard of it. Turns out several engineers I knew (and one military data center guy) were familiar with it too.
Keep in mind what the parent of my post said: You simply cannot convince me that this is a real problem that we need to worry about. Not, that this is a common problem, or a problem everyone should worry about. His or her claim was much stronger than that: it it's either unreal, or we don't have to worry about it (i.e., infinitessimal or zero risk).
Furthermore, I found it telling that s/he said "you cannot convince me" instead of "you haven't convinced me". Not the best indicator of logical thought in my experience, but maybe it was just a poor choice of words.
Like I said, believe what you want. Just let me know if you're running a data center in case I ever need to colo.
Actually, in addition to have the potential to kill your computer, these fibers can do damage to your lungs to, just like asbestos, according to the PDF
A whisker doesn't have to survive conducting enough current to let the smoke out of the power supply - in it's incandescent passing, if the ghost of it's exisistance is a sliver of plasma, a very substaintial, if evanescent, conductor is created, literally, out of thin air.
Thanks for this, gessel. In the aerospace industry, where I work, tin whiskers are the problem. In a vacuum (test or on-orbit), a shorting whisker can result in a plasma condition that will arc, allowing literally hundreds of amperes of current to flow for durations on the order of tens of seconds. This has been the cause of loss of many units and even entire systems. Of course, this is in a vacuum; in a non-vacuum situation, though, it is still possible to create localized low-pressure areas that might enable this phenomenon.
Whether or not this relates to the computer room problems of this article is not strictly relevant to the point I want to make. My point is: Many things happen in ways you wouldn't expect; intuition is not always your best guide to determining cause-effect.
Anything that adds cost and isn't absolutely essential doesn't go into a product with merciless cost pressure. Consumer-grade PCs will last for years in home use without the conformal coating, so they don't have it.
Anything that increases production time and isn't essential doesn't go into a high volume product. Applying and curing the coating takes additional time, space, and equipment. A fume hood or spray booth is needed to control fumes during application, and a UV curing "oven" is desirable for high volume products (air drying takes much longer).
Conformal coating is widely used in military products, and also is in some industrial products that must survive severe environments. But they are rarely used in office equipment, and I've never seen it in a mass market PC.