Slashdot Mirror
Zinc Whiskers Cripple Colorado's Computers
Mr. Christmas Lights writes "While zinc whiskers, small metallic fibers which grow on surfaces that have been electroplated with zinc, aren't a problem for Christmas lights, they can cause serious problems for computers. The Denver Post reports how they caused computer outages for the last three weeks in the Colorado secretary of state's office. This basically halted business and elections document filings. Zinc whiskers are becoming more of a problem as computers electronics get smaller. NASA has a good reference site which includes a interesting PDF summary paper complete with pictures. /.'ers with computer rooms might want to check this out."
Shocking! /sorry, couldn't resist.
Etiquette is etiquette. He kills his mother but he can't wear grey trousers.
...with a magnet!
I'm probably at the karma cap. Mod up a funny troll instead, it lightens the mood
Zinccccccc Zinnnccccc! Oh my god what have I done!
I don't need no instructions to know how to rock!!!!
I guess we have to shave all zinc cats before we let them into the building.
Never attribute to stupidity what can be construed as a monopoly preservation tactic.
(rolls eyes)
Where I used to work, we had this issue - in our case they grew from the cheap computer floor panels in the room. The case was so bad, you could see them in direct sunlight, and the flowed in the breeze like grass.
We had no choice but to go through cleaning, as the underfloor was about to be used for blowing air to new systems, without it, the zinc whiskers would blow free and cause hell on all our systems. As it was, three systems failed in the week after the clean. We don't want to think what would have happened if we didn't clean it.
It's not bullshit. Get over it. Interestingly, there are very few people who know of this issue, but knowledge is spreading.
Robert Anton Wilson
The metal frames for the raised floor are where the zinc whiskers are coming from. They get sucked into the power supplies and short them out.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
We are in the final streach for replacing the entire datacenter floor where I work because of zinc Whiskers. We had the underfloor area cleaned for the new A/C that blows from the floor up. We promptly lost 11 power supplies and the total count is up over 20 now. We did verify the presents of zinc whiskers in the dead powersupplies thanks to the onsite electron microscope.
Thankfully the high temperature never got above 85 degrees so the old A/C was able to keep up.
... amazinc.
Here's a yummy little google cache for y'all.
(-(friend^2))^(1/2)
Incoming mod-bombing for having a different viewpoint, 2 o'clock! Heads up!
And are a problem with the new lead free processes.. especially as lead spacing decreases, and the euro lead-free requirement kicks in.
Agere wrote a good article in Analog Zone, available at http://www.analogzone.com/grnt0216.pdf. It has a good micrograph showing the problem.
"'Tis great confidence in a friend to tell him your faults, greater to tell him his." --Poor Richard's Almanac
I work in a large midwest hospital, and we've got a constant issue with Zinc whiskers in our network jacks in operating rooms. Supposedly the origin is the cleaning solutions that they use for the floor.
#931654: "Sorry, our computers seem to be growing metallic whiskers. What did you say your username was? bwahahahahaha"
bash: rtfm: command not found
I so agree with you.
back in college I paid my way as a maintaince guy at a foundry. we had rackmount Pc's that would have almost 1/4 inch of metal/sand dust on the motherboards and the computers were STILL working.
Cince then I have worked as a freelance consultant and specalist for many different companies that certianly do not have their computers in a "clean room" (machine shops for the best example) and they never EVER have these problems and they are exposed to nasty air + metal. The worst was a water filtration plant I worked at for 7 years where a workstation for monitoring the vats of hydro-flouro-sisicilic acid ( what they put in your water to add flouride) a product that is so corrosive that it eat's through the rubber lined fiberglass tanks within 3 years had, just by the amount released in the air during tank inspections, had eaten almost all the legs off the surface mount chips on the motherboard and it was STILL operating.
Maybe some really REALLY old mainfraime computers might see the problem in a 20-30 year lifespan that the article suggests, but even the PDP-11 I saw back in college that was retired in the basement but still maintained operating by students did not have any problems like this.
Do not look at laser with remaining good eye.
Zinc whiskers are quite real, and they can pose a hazard. The reason you don't see data centers around the world having this problem has many aspects, some of which I can think of include:
-Too new. It takes years for these whiskers to grow to any length (1mm/year), and it may take years before it even starts to happen.
-Only happens to electroplated surfaces. Dip galvanized objects (Like electrical boxes and bolts and such) don't seem prone to this effect.
-Newer machines are more vulnerable with their more delicate circuits and smaller, tighter tracings and pins. A data center with older machines might be all but immune to it simply because the equipment is old and robust enough.
High taffic areas they are likely to get stomped on/eroded away long before they pose a problem. So you would have to have a situation where you have a spot like under a table, where you have objects electroplated with zinc sitting undisturbed for a long time, then get disturbed. Then you would have to be unlucky enough to disturb them and get it into the air.
Ironically, there are wood based floors used in data ceneters with steel reinforcing on the back of the tiles that are zinc-electroplated (thus being essentially undisturbed for years). So if a fairly old data center, that happens to have the right (wrong?) type of flooring, undergoes an upgrade or reorganization... well that might do it.
Now if these guys ARE just using it as an excuse, that's another story. But that doesn't make the problem any less real.
=Smidge=
and a narative
Access Floors
I know I speak against my own name, here. But:
Living by the sea, especially with the high humidity that comes with it...and the salt...can give your more than just a few zinc whiskers on the PC board.
Salt oxidation--depending on how long the windows are open--can really eat a PC in two years or less; never mind the quality of the MB.
Sea salt is hydrophilic. If it accumulates on something, episodes of high humidity will attract moisture from the air, and add the basis for typical corrosive effects. I have had containers with dry sea salt, which have pulled moisture out of the air on their own.
(Most acids need water...so does salt to release it's own ions, which can have a corrsive effect similar to an acid on metallic equipment...usually involving the non-metal in the salt. Sea salt has lots of chlorine, a very strong oxidizing agent.)
You can conformal-coat boards yourself, using Fine-L-Kote spray. We use this stuff on the Overbot.
It's a flammable, toxic chemical mixture until it dries; you need gloves, goggles, a respirator mask, and proper flammable liquid storage. Cover connectors with masking tape before spraying. It's a clear coat, but glows in UV, so you can check for missed spots.
If these things are so small as to require an electron microscope to see, why arn't they simply vaporized/melted as soon as they find themselves shoved between two (relativly) massive wires?
You'd think that a microscopic piece of zinc would go before a macroscopic fuse/chunk of copper/etc. And since it's alot of single pieces blown around, it's not like several million are all going to do it at exactly the same time.
I've melted zinc, and it's pretty snappy, (Pennies after 1982 are mostly Zinc. When you melt them, you get a cool copper-skin effect going on.), but with the same torch ($15 propane torch) I was unable to even visibly affect the copper.
I'd be a lot more inclined to believe the "whiskers" are coming from movement of the tiles when people walk across them(they do shift, as does the frame slightly) and not some "growing whiskers" BS.
Furthermore, the problem is easily solved via any/combination of these:
Furthermore, if the little buggers are metallic, why don't you just install a few small but powerful magnets in various ducts? A metal grate made up with a set of magnetic rods would probably work like a swell charm, and only require periodic cleaning...
Please help metamoderate.
I work tech support. I can't wait to start using this as an explanation of all sorts of things:
Customer: When I go on mah Innernet, I get this error 'bout id not bein' displayed.
Me: Sir, it looks like you've fallen victom to Zink Wiskers. No sir, you don't need to get rid of your cats.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
and therein lies the problem: you just asserted that this is an item of faith for you, not reason; facts be damned, you cannot be convinced.
Never mind that there are several companies who do raised subfloors who've been addressing this problem for some time. They're all peddling snake oil, and NASA is helping them do it.
Never mind that Zn whiskers grow slowly, Zn-electroplated subfloor panels in data centers aren't that old, and PC board density has been increasing. Or that they only occur on electroplated zinc, and only grow long enough to be problematic in very low traffic areas. Nope, must be bogus.
Never mind that hospitals are affected as well and take this seriously. Or that the condition is well known among electroplaters and materials engineers, and was discussed at least ten years ago in the literature. Or that it's been involved in at least one product liability case. Or that Bell Labs has known about it for over fifty years (since 1948).
And never mind you could have found everything I mentioned above within the first 30 google results for "Zinc Whiskers". Nope. It must all be a myth, because there's no such thing as newly discovered age-related problems.
(Oh, and I hear that automobile corrosion is a myth too ... I went to the new car lot and looked around and didn't see any, so it must not exist)
No, I'm not worried about zapping floppies. Your ideas are, for the most part very good ones, but zinc is paramagnetic, if I recall right, and is not attracted to the magnets. Same for tinfoil, BTW. The magnets would be effective in catching iron filings and related ferromagnetic particles, though, but that didn't seem to be the reported problem in this case.
Your friction hypothesis has merit, too, but growth of whiskers, more scientifically known as dendrites is actually quite common, especially where electric fields exist between conductors. I ran into that in a flexible touch keyboard we had designed using a silver alloy that was screened on as the conductors. Durn things would develop shorts after a while in the field, literally since it was on agricultural equipment. You couldn't see the shorts, but examination under a microscope revealed those nasty little whiskers. A metalurgist was consulted and provided a different alloy that solved the problem.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
It's also worth noting that, at the finishing.com link I gave above, you'll find Cisco and well-known companies have known about this problem since at least 1996. Or perhaps finishing.com is involved in the conspiracy, or Cisco power engineers are idiots.
1. I first like to thank simoniker for adding the "small metallic fibers which grow on surfces that have been electroplated with zinc" to the article - made it more understandable/readable.
2. The NASA URL is one-level deep (a mistake on my part) - here is the top-level.
3. Related to #2, I would STRONGLY recommend /.'ers actually READ what that says. The Denver Post article was written by a reporter - would you expect that to be technically accurate/broad/etc? Again, take a look at the NASA site which DOES present a compelling case that this is a REAL issue and not FUD. The original study with the medical equipment makes for facinating reading.
4. Some Anonymous Coward seems to have a problem with my nickname. Did you actually click on the "Mr. Christmas Lights" and see what is there - tell me that isn't appropriate (it's been used before BTW).
5. The same AC made a smart-ass comment about the Nigritude Ultramarine SEO contest - while I'm aware of that contest (#4 above is a hint for 'ya!), I'm currently ranking #199 for the keyphrase with less than a week to do, so I'm not a contendor ... although I do rank #1 for the phrase Nigritude Ultramarine Hulk! ;-) ... and I actually did submit a wrapup article a few days ago about this, but it got rejected - good news is the contest is over July 7th, so all those N-U links will go away - they are a bit annoying.
6. I haven't seen anyone comment on a business (verus technical) aspect of the Denver Post article (but this is /.) where some state mucky-muck basically says this is a reason to bring all state websites under one authority and talks about $7.5 million in funding. One wonders if some empire building going on and/or play for more money!
7. There have been several Denver Post articles about the failure of these computer systems. I didn't mention that fact in my submission because I thought it would be too lengthy, but apparently the inability to electronically check/file business/elections stuff has been a real big deal - good example of our dependancy on computers.
'Nuff random late night rambling!
Hulk SMASH Celiac Disease
Metal whiskers aren't usually a problem on the mobo itself (everything is conformal coated) but on exposed metal -- surface mount devices for example -- and especially in power supplies. And even then, it's only a problem if you can't take (or blow through) a 50-300 ohm short every now and then. Newer equipment is a lot more sensitive -- denser boards, less slop in timing and signals, etc ... and of course manufacturers cut costs wherever they can, even on networking and server equipment. Unless you can afford NEBS or industrial grades, "server grade" rackmounts aren't necessarily any more rugged these days than consumer grade crap.
But it has been a problem outside of data centers, especially where you're looking at small (analog) signals with high input impedence. Examples include medical monitoring equipment and scientific research equipment, and it's why you don't see established manufacturers of either using uncoated, unalloyed zinc electroplating, especially in humid environments.
we had rackmount Pc's that would have almost 1/4 inch of metal/sand dust on the motherboards and the computers were STILL working.
Contacts that are exposed to a "harsh" environment generally form a very thin non-conductive film due to oxidation and/or corosion. This would also apply to any conductive surfaces. Depending on exactly what gets laid down, what you describe seems completely reasonable. However, something in a "clean" environment can easily be killed by something your self-protected PCs wouldn't even notice.
Extraordinary claims require extraordinary evidence.
Risk = probability * loss. Since loss is obvious and fairly constant (a flaky or inoperative computer), if you want to hype this risk, besides just showing clear evidence that zinc whiskers are increasing the probability that a computer will die, you have to show to what degree that probability is increased - that is, that it merits more concern than other obscure things that can cause a computer to die.
Well you better start doing your homework.
I worked for many years on a replacement Air Traffic Control System for Canada and as the project matured, our stage - lab (containing litterally hundreds of machines, a complete lab recreation of the coast-to-coast ATC system) started to experience an MTBF on the power supplies in the equipment that was over an order of magnitude smaller than spec'd by the manufacturer (Hewlett-Packard).
Since this was a long-term contract that included commitments to deliver over an extended period of time (25 years), the material cost of this problem was VERY significant to the equipment vendor, not the customer. (In other words there was no financial motivation to fail to find fault, quite the opposite; the fault was costing them money.)
In the spirit of "old HP" they sent us some senior hardware design guys to look at our lab and our environmentals (humidity, temp, pressures, cycles and power suppy spectra) to see what was causing the problem.
Being about 6 years ago -- I hadn't heard of the Zinc problem yet, and neither had the guys from HP. They took everything back to their labs, including about 6 failed supplies and a couple 'still good ones', some from reserve stock and some from working machines.
A few weeks later they came back; there was a big meeting -- this was an issue with potentially enormous cost -- including the ultimate customer's representatives.
I can remember the Project Manager practically spitting his coffee when informed the underlying cause. The 'special ESD safe A/C'd lab' was part of the problem. Thank fully, the final deployed environment had different flooring, so we didn't have to change the sites, just some modifications to the lab.
This is far from BS -- it's a problem that has cost millions and will likely cost millions more before it's over. But the SEM photos of the failed devices we cool to see.
Switching power supplies can be surprisingly fragile. I've killed a couple working on TVs (that's basically what a flyback circuit is) and you can do it in one or two cycles (of your AC, not CPU cycles). And no, you don't want to know how much those power transistors cost. And I've killed computer PS by shorting across IC pins. There's not a lot of current going through these, and a 50-200 ohm short will definitely do the job. Remember, it only has to conduct long enough to nuke the chip.
Or maybe you'd prefer to ask the Cisco power engineer about it. Naah, he probably doesn't know what he's talking about.
I love this kind of stuff. Clueless moderators mod the parent insightful because none of them happen to have any personal experience with the subject. Lack of personal evidence does not constitute a conspiracy. Really, it doesn't.
/. need for anecdotal evidence, yet you groundlessly claim that the articles anecdote MUST have some other explanation. Why? I am all for being a little skeptical, but there is such little reason for it in this case. Would you begrudge them the opportunity to clean out their under-tile areas? Most of us know how bad those areas can get regardless of the whisker issue.
I have personally known about whiskers for over five years. It was becoming a potential problem in an old datacenter at work. It is a serious condition that datacenters with critical machines (or contractual obligations) take into account in datacenter design and maintenance. With 5 9's required for a lot of machines (Hitachi, IBM, Unisys, etc) there is little room for allowing electrically conductive dust particles to flow across every board on your machines.
The other posters have given examples to satisfy the typical
_damnit_
It's my job to freeze you. -- Logan's Run
Actually, in addition to have the potential to kill your computer, these fibers can do damage to your lungs to, just like asbestos, according to the PDF
"It's not my code, it's the zinc whiskers"
As I recall, old computer used to suffer from mice chewing on the wires and they used to keep a cat to keep the programs running smoothly. Then it was bugs and I am not sure if spiders helped or added to the problem.
I, for one, welcome our new zinc-eating nanobots.
Anything that adds cost and isn't absolutely essential doesn't go into a product with merciless cost pressure. Consumer-grade PCs will last for years in home use without the conformal coating, so they don't have it.
Anything that increases production time and isn't essential doesn't go into a high volume product. Applying and curing the coating takes additional time, space, and equipment. A fume hood or spray booth is needed to control fumes during application, and a UV curing "oven" is desirable for high volume products (air drying takes much longer).
Conformal coating is widely used in military products, and also is in some industrial products that must survive severe environments. But they are rarely used in office equipment, and I've never seen it in a mass market PC.