Slashdot Mirror
Zinc Whiskers Cripple Colorado's Computers
Mr. Christmas Lights writes "While zinc whiskers, small metallic fibers which grow on surfaces that have been electroplated with zinc, aren't a problem for Christmas lights, they can cause serious problems for computers. The Denver Post reports how they caused computer outages for the last three weeks in the Colorado secretary of state's office. This basically halted business and elections document filings. Zinc whiskers are becoming more of a problem as computers electronics get smaller. NASA has a good reference site which includes a interesting PDF summary paper complete with pictures. /.'ers with computer rooms might want to check this out."
Shocking! /sorry, couldn't resist.
Etiquette is etiquette. He kills his mother but he can't wear grey trousers.
...with a magnet!
I'm probably at the karma cap. Mod up a funny troll instead, it lightens the mood
Zinccccccc Zinnnccccc! Oh my god what have I done!
I don't need no instructions to know how to rock!!!!
I am wondering if they will have some sort of sheilding system for this in the future, if it becomes more of a problem. That could be as simple as a small layer of some nonconductive resin on the surface of the circuits. But will it ever be economically feasible?
I suppose in the mean time we'll have to do our own safeguarding if we are in a risky area.
The Cheese Stands Alone.
I guess we have to shave all zinc cats before we let them into the building.
Never attribute to stupidity what can be construed as a monopoly preservation tactic.
(rolls eyes)
Where I used to work, we had this issue - in our case they grew from the cheap computer floor panels in the room. The case was so bad, you could see them in direct sunlight, and the flowed in the breeze like grass.
We had no choice but to go through cleaning, as the underfloor was about to be used for blowing air to new systems, without it, the zinc whiskers would blow free and cause hell on all our systems. As it was, three systems failed in the week after the clean. We don't want to think what would have happened if we didn't clean it.
It's not bullshit. Get over it. Interestingly, there are very few people who know of this issue, but knowledge is spreading.
Robert Anton Wilson
The metal frames for the raised floor are where the zinc whiskers are coming from. They get sucked into the power supplies and short them out.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
Anybody ever see, much less be affected by, "zinc whiskers"? In the one or two times I've seen anything resembling this, I always thought it was metal shavings from the assembly process. That, or clever dust :-)
We are in the final streach for replacing the entire datacenter floor where I work because of zinc Whiskers. We had the underfloor area cleaned for the new A/C that blows from the floor up. We promptly lost 11 power supplies and the total count is up over 20 now. We did verify the presents of zinc whiskers in the dead powersupplies thanks to the onsite electron microscope.
Thankfully the high temperature never got above 85 degrees so the old A/C was able to keep up.
... amazinc.
Here's a yummy little google cache for y'all.
(-(friend^2))^(1/2)
Incoming mod-bombing for having a different viewpoint, 2 o'clock! Heads up!
Yup.
And the metal that the whiskers come from, have been used for computer room floor tiles, racks, and even (shudder) PSU cases.
Now that's scarey.
Robert Anton Wilson
I thought almost all quality electronic devices have a conformal coating (non-conducting polymer) applied to PCBs? Colorado must be buying el-cheapo crap. Remember, you get what you pay for.
"I'd rather be a lightning rod than a seismometer." -Ken Kesey
And are a problem with the new lead free processes.. especially as lead spacing decreases, and the euro lead-free requirement kicks in.
Agere wrote a good article in Analog Zone, available at http://www.analogzone.com/grnt0216.pdf. It has a good micrograph showing the problem.
"'Tis great confidence in a friend to tell him your faults, greater to tell him his." --Poor Richard's Almanac
No kidding... I mean, if they would have said that they were shorting out the CPU's data bus, or something along those lines it might be believable.
But the POWER SUPPLIES?!?
A microscopic fiber of zinc (a metal with a _very_ low melting point) would not short out ANYTHING with more power than a hearing-aid battery!
As another pointed out already, one of the people interviewed is trying to sell their services cleaning it up!
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
I work in a large midwest hospital, and we've got a constant issue with Zinc whiskers in our network jacks in operating rooms. Supposedly the origin is the cleaning solutions that they use for the floor.
But I could google on and on. Hey, poster (NigritudeUltramarine). Care to explain your nickname? Was it intended to offend? I suspect it's just a trash
We have 3 rooms where I work that house our servers. We are migrating some stuff over to an HP-UX system running oracle. When we had some HP representatives come over to our area to check out the place we intended to house the HP servers, they insisted that zinc whiskers could potentially be very damaging and that we make sure our rooms were clean and free of zinc whiskers (not that our rooms were particularly dirty). I always wondered how legitimate their claims were.
Power supplies? Makes me wonder if the issue goes back to the old capacitor problems with the bulging and exploding and stolen electrolyte, etc..
I know I've seen more then just motherboards with bad capacitors and done a bit of soldering now. Heck, I have a linksys 10/100 switch with a single buldging capacitor in it, not to mention the Antec power supply with several bad ones as well.
Morphing Software
Couldn't manufacturers spray some sort of nonconductive film onto new boards?
With regard to the NASA reference, could you please go into more detail? I though growing crystals were slow to enlarge and fragile. I'm not doubting the veracity, I just don't understand how zinc crystals could grow *through* an inch of epoxy.
Zinc Whiskers? Hey, they stole my username!
Ok, no they didn't.
In each, often hundreds (or even thousands) of computers are consolidated in one room.
You simply cannot convince me that this is a real problem that we need to worry about. Yes, OBVIOUSLY you don't start pouring out bags of metal filings into your ventilation system. We already KNEW that. But not buying anything made from zinc? No way. We'd have data centers shutting down all over the place, if as Rich Hill says, "Metal on floor panels and even in computer cases can secrete zinc crystals over time."
The state said, "The outages
Dusting, god no. The last thing you want to do is get these fuckers airborne, that's when they start to cause problems. They get sucked into a tower or rack by the air convection, and wind up settling into places they shouldn't. Next thing you know, you have equipment shorting out.
Forget the duster. Cleaning will probably involve mineral oil, or some other decently viscous but harmless liquid, being sloughed across the floor, sponged up, then washed away using conventional cleaners. The oil would weigh down the zinc whiskers to the point that they couldn't get into the air and cause problems.
Read the various papers. They are not talking about one whisker tripping a power supply, or even a dozen. Its hundreds, gradually building up enough of a path for a short (or drain on the power supply) to cause problems.
someone (I can't recall who) offered a prize to the site that could generate the highest google pagerank for the nonsense phrase "Nigritude Ultramarine". I'm sure if you google you can find some more information.
#931654: "Sorry, our computers seem to be growing metallic whiskers. What did you say your username was? bwahahahahaha"
bash: rtfm: command not found
My A-100's chorus/vibrato likes to go on summer vacation when it gets humid in the house, but works fine during the winter. Just thought I'd throw that in - I think the stuff is zinc-plated...
With all the RoHS crap going on (the "lead free" european requirements), you'll be seeing a lot more failures do to tin wiskers. It's already looking like it will affect the life of most future electronic products.
First Computer Viruses,DoS, and now this ! Can't we see that humans and technology don't go along very well?? let's all go back to the dark ages *ZINC* .... Oh no I was just kidding!
What is best in life? To crush your enemies, to see them driven before you and to hear the lamentations of their women.
I so agree with you.
back in college I paid my way as a maintaince guy at a foundry. we had rackmount Pc's that would have almost 1/4 inch of metal/sand dust on the motherboards and the computers were STILL working.
Cince then I have worked as a freelance consultant and specalist for many different companies that certianly do not have their computers in a "clean room" (machine shops for the best example) and they never EVER have these problems and they are exposed to nasty air + metal. The worst was a water filtration plant I worked at for 7 years where a workstation for monitoring the vats of hydro-flouro-sisicilic acid ( what they put in your water to add flouride) a product that is so corrosive that it eat's through the rubber lined fiberglass tanks within 3 years had, just by the amount released in the air during tank inspections, had eaten almost all the legs off the surface mount chips on the motherboard and it was STILL operating.
Maybe some really REALLY old mainfraime computers might see the problem in a 20-30 year lifespan that the article suggests, but even the PDP-11 I saw back in college that was retired in the basement but still maintained operating by students did not have any problems like this.
Do not look at laser with remaining good eye.
Zinc whiskers are quite real, and they can pose a hazard. The reason you don't see data centers around the world having this problem has many aspects, some of which I can think of include:
-Too new. It takes years for these whiskers to grow to any length (1mm/year), and it may take years before it even starts to happen.
-Only happens to electroplated surfaces. Dip galvanized objects (Like electrical boxes and bolts and such) don't seem prone to this effect.
-Newer machines are more vulnerable with their more delicate circuits and smaller, tighter tracings and pins. A data center with older machines might be all but immune to it simply because the equipment is old and robust enough.
High taffic areas they are likely to get stomped on/eroded away long before they pose a problem. So you would have to have a situation where you have a spot like under a table, where you have objects electroplated with zinc sitting undisturbed for a long time, then get disturbed. Then you would have to be unlucky enough to disturb them and get it into the air.
Ironically, there are wood based floors used in data ceneters with steel reinforcing on the back of the tiles that are zinc-electroplated (thus being essentially undisturbed for years). So if a fairly old data center, that happens to have the right (wrong?) type of flooring, undergoes an upgrade or reorganization... well that might do it.
Now if these guys ARE just using it as an excuse, that's another story. But that doesn't make the problem any less real.
=Smidge=
...small compact computer designs get these whiskers easier. I don't doubt it happens, but where are all the millions of laptops shorted out then, or the mini itx machines,game machines, etc?
Is there something else here causing whiskers to grow some places and not in others, even though both have zinc?
I normally don't reply to ACs, but what you're describing can only happen to a power supply that is OFF for extremely extended periods of time, with some other fan causing airflow into it.
Not gonna happen.
Each whisker that hits it will be vaporized long before your hundreds can build up.
Think about how thick a 3A fuse is, the fibers they're describing would have to pile in the THOUSANDS for this to happen, and all at once at that!
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
and a narative
Access Floors
Wohoo first post ever on slashdot!
Anyway, I work at an IBM data center with many computers and sensitive equipment around. We have zinc ducts here(well they look like zinc so I assume they are), and in the 15+ years this place has been running no problems have become of it as far as I know. That is no problem that can be traced to it. I suppose it is possible some problem that was blamed on something else could of been caused by this, but I've never heard of it.
Just some info.
Capacitors (electrolytics, anyways) go bad over time, that's a given.
I just replaced a pair of 680uF 180V caps in the power supply of my Deskjet 1600CM, as they failed catastrophically, but it had nothing to do with zinc whiskers.
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
technology rears its cold head, and the zinc-whiskered men and women of the airplane and car rise up against their fathers...
I know I speak against my own name, here. But:
Living by the sea, especially with the high humidity that comes with it...and the salt...can give your more than just a few zinc whiskers on the PC board.
Salt oxidation--depending on how long the windows are open--can really eat a PC in two years or less; never mind the quality of the MB.
Sea salt is hydrophilic. If it accumulates on something, episodes of high humidity will attract moisture from the air, and add the basis for typical corrosive effects. I have had containers with dry sea salt, which have pulled moisture out of the air on their own.
(Most acids need water...so does salt to release it's own ions, which can have a corrsive effect similar to an acid on metallic equipment...usually involving the non-metal in the salt. Sea salt has lots of chlorine, a very strong oxidizing agent.)
It's not BS. Although it's not Zinc, growth of copper or iron wiskers inside high-tension transformers (as in the ones used by the electric utilities) is one of the reason they fail.
and where does that vaporised metal go? Sure some of it is converted into zinc oxide but some of it will be redeposited on the system. Further, what about the situation where the connection isn't made until after the depositing of numerous fibres. Its not as simple or as complex as either side is making it out to be. (I personally haven't seen enough either way personally to be swung to one side or the other in this debate.) Remember, dust can carry enough of a static charge to destroy circuitry over time.
Bad Panda! No Bamboo for you! In matters of importance ACs will not be responded to. Want to say something critical,OK
So the days of calling program errors 'bugs' is finally over, huh? Now we need to call them 'whiskers'! Time to update BugZilla to WhiskerZilla I guess..
If these things are so small as to require an electron microscope to see, why arn't they simply vaporized/melted as soon as they find themselves shoved between two (relativly) massive wires?
You'd think that a microscopic piece of zinc would go before a macroscopic fuse/chunk of copper/etc. And since it's alot of single pieces blown around, it's not like several million are all going to do it at exactly the same time.
I've melted zinc, and it's pretty snappy, (Pennies after 1982 are mostly Zinc. When you melt them, you get a cool copper-skin effect going on.), but with the same torch ($15 propane torch) I was unable to even visibly affect the copper.
in my life God comes first.... but Linux is pretty high after that
Francis Smit
We had to replace all the raised floor panels in one of our old data centers because of zinc whiskers. It took us a while to figure out what was going on.
If you had taken time to check into it, you could have avoided looking foolish.
I'm not completely sure, but isn't this one of the purposes of silk screening anyway? Which happens to be done on every single motherboard I've ever seen....
I'm not sure about this, though, I could be wrong.
-ReK
md5sum -c reality.md5
reality: FAILED
md5sum: WARNING: 1 of 1 computed checksum did NOT match
...green dots are an audio solution. This smacks of freakish pseudoscience masquerading as real problems. Kind of like the gold plated power cords for stereos that cost audiophiles $180 each. The supposed reason that they improve sound is that gold has less resistance and therefore your amp will perform better if the electricity gets to it more readily. That's bullshit just like the zinc whisker issue.
Un-news
Our whole terminology is going to change. What was debugging is going to be called shaving, and instead of running DDT I'll be running Gilette to trace through and eliminate whiskers from my code. I guess it's finally time to learn FORTRAN++.
Tag lost or not installed.
Which effectively means that it's a component of dust... so if you have dust buildup in your room, you're likely to find this among other problems forming.
But take a look on newer hardware, If the chips aren't mounted in a BPGA (Ball Pin Grid Array), then chances are the legs on the chips are very very small. So the less metal you have, the shorter lifespan they will have in such harsh conditions as the one you've stated.
Life is not for the lazy.
The only thing I can think of is the MST3k short where a man wishes there were no springs, and a satanic spring elf named "Springy" laughs manically as everything with springs stops working.
Not a Twitter sockpuppet... but I wish I was.
I'd be a lot more inclined to believe the "whiskers" are coming from movement of the tiles when people walk across them(they do shift, as does the frame slightly) and not some "growing whiskers" BS.
Furthermore, the problem is easily solved via any/combination of these:
Furthermore, if the little buggers are metallic, why don't you just install a few small but powerful magnets in various ducts? A metal grate made up with a set of magnetic rods would probably work like a swell charm, and only require periodic cleaning...
Please help metamoderate.
I work tech support. I can't wait to start using this as an explanation of all sorts of things:
Customer: When I go on mah Innernet, I get this error 'bout id not bein' displayed.
Me: Sir, it looks like you've fallen victom to Zink Wiskers. No sir, you don't need to get rid of your cats.
Hi! I make Firefox Plug-ins. Check 'em out @ https://addons.mozilla.org/en-US/firefox/addon/youtube-mp3-podcaster/
This is really just a subset of a larger problem... dust.
Any particle that's floating in the air that isn't a gas simply doesn't belong in a computer room. However, any time you've got humans in the room dust is sure to form. Dust can cause a computer to overheat, and these guys are just pointing out that if you ever get small charged particles flying in your room you've got real problems.
Computers most certainly can be run in a dirty room without immediate failures... but the dirtier a room happens, the more likely a phantom failure that is scientifically valid by some far-out reason like this but seems like a ghost in the machine crops in. We've just accepted unexplainable computer crashes on Windows machines... but this one's an OS-independent way to have problems.
Well done AC.
Have you ever touched a 9V battery to a pile of steel-wool?
In case you haven't... the steel wool (which is many thousands of times thicker than the micro-fibers we're talking about here) instantly ignites and burns, even though the 9V battery has a tiny fraction of the current capability of a computer's power-supply. The amount of current that is tapped is miniscule, as the fibers that are actually MAKING the contact are instantly destroyed.
The remains are non-conductive, as they are completely oxidized.
Any metal that oxidizes will, when in small enough form (ie: dust or fibers) becomes extremely flamable. Witness the powdered aluminum and magnesium used in fireworks for a good example.
BTW: if anyone decides to try the steel-wool thing, it is very cool looking, but don't do it over carpeting or anything flamable! Also, it won't work if it's coated with soap (brillo/SOS pads). The finer the wool, the more dramatic the effect... try some 00 from a hobby shop.
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
and therein lies the problem: you just asserted that this is an item of faith for you, not reason; facts be damned, you cannot be convinced.
Never mind that there are several companies who do raised subfloors who've been addressing this problem for some time. They're all peddling snake oil, and NASA is helping them do it.
Never mind that Zn whiskers grow slowly, Zn-electroplated subfloor panels in data centers aren't that old, and PC board density has been increasing. Or that they only occur on electroplated zinc, and only grow long enough to be problematic in very low traffic areas. Nope, must be bogus.
Never mind that hospitals are affected as well and take this seriously. Or that the condition is well known among electroplaters and materials engineers, and was discussed at least ten years ago in the literature. Or that it's been involved in at least one product liability case. Or that Bell Labs has known about it for over fifty years (since 1948).
And never mind you could have found everything I mentioned above within the first 30 google results for "Zinc Whiskers". Nope. It must all be a myth, because there's no such thing as newly discovered age-related problems.
(Oh, and I hear that automobile corrosion is a myth too ... I went to the new car lot and looked around and didn't see any, so it must not exist)
If you want, I'll send you a copy on Wednesday when I get back to work.
No, I'm not worried about zapping floppies. Your ideas are, for the most part very good ones, but zinc is paramagnetic, if I recall right, and is not attracted to the magnets. Same for tinfoil, BTW. The magnets would be effective in catching iron filings and related ferromagnetic particles, though, but that didn't seem to be the reported problem in this case.
Your friction hypothesis has merit, too, but growth of whiskers, more scientifically known as dendrites is actually quite common, especially where electric fields exist between conductors. I ran into that in a flexible touch keyboard we had designed using a silver alloy that was screened on as the conductors. Durn things would develop shorts after a while in the field, literally since it was on agricultural equipment. You couldn't see the shorts, but examination under a microscope revealed those nasty little whiskers. A metalurgist was consulted and provided a different alloy that solved the problem.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
It's also worth noting that, at the finishing.com link I gave above, you'll find Cisco and well-known companies have known about this problem since at least 1996. Or perhaps finishing.com is involved in the conspiracy, or Cisco power engineers are idiots.
1. I first like to thank simoniker for adding the "small metallic fibers which grow on surfces that have been electroplated with zinc" to the article - made it more understandable/readable.
2. The NASA URL is one-level deep (a mistake on my part) - here is the top-level.
3. Related to #2, I would STRONGLY recommend /.'ers actually READ what that says. The Denver Post article was written by a reporter - would you expect that to be technically accurate/broad/etc? Again, take a look at the NASA site which DOES present a compelling case that this is a REAL issue and not FUD. The original study with the medical equipment makes for facinating reading.
4. Some Anonymous Coward seems to have a problem with my nickname. Did you actually click on the "Mr. Christmas Lights" and see what is there - tell me that isn't appropriate (it's been used before BTW).
5. The same AC made a smart-ass comment about the Nigritude Ultramarine SEO contest - while I'm aware of that contest (#4 above is a hint for 'ya!), I'm currently ranking #199 for the keyphrase with less than a week to do, so I'm not a contendor ... although I do rank #1 for the phrase Nigritude Ultramarine Hulk! ;-) ... and I actually did submit a wrapup article a few days ago about this, but it got rejected - good news is the contest is over July 7th, so all those N-U links will go away - they are a bit annoying.
6. I haven't seen anyone comment on a business (verus technical) aspect of the Denver Post article (but this is /.) where some state mucky-muck basically says this is a reason to bring all state websites under one authority and talks about $7.5 million in funding. One wonders if some empire building going on and/or play for more money!
7. There have been several Denver Post articles about the failure of these computer systems. I didn't mention that fact in my submission because I thought it would be too lengthy, but apparently the inability to electronically check/file business/elections stuff has been a real big deal - good example of our dependancy on computers.
'Nuff random late night rambling!
Hulk SMASH Celiac Disease
and if you look closely you will see bits of metal welded to the terminals. Further there is a big difference between zinc and iron as for oxidation rates. Which btw is why you see ferric oxide aka rust all over the place and you will rarely see zinc oxide except on the beach.
Bad Panda! No Bamboo for you! In matters of importance ACs will not be responded to. Want to say something critical,OK
For a minute, I could've SWORN I heard the BOFH reading this paragraph aloud, then advising some poor sap to pour gasoline all over their floor, because the hydrocarbons will interact with the zinc... ...and then lighting the floor on fire, killing the unsuspecting luser...
'If you're flammable and have legs, you are never blocking a fire exit.'
They'll do anything to keep Nader off the ballot in a swing state. :-)
dinner: it's what's for beer
Metal whiskers aren't usually a problem on the mobo itself (everything is conformal coated) but on exposed metal -- surface mount devices for example -- and especially in power supplies. And even then, it's only a problem if you can't take (or blow through) a 50-300 ohm short every now and then. Newer equipment is a lot more sensitive -- denser boards, less slop in timing and signals, etc ... and of course manufacturers cut costs wherever they can, even on networking and server equipment. Unless you can afford NEBS or industrial grades, "server grade" rackmounts aren't necessarily any more rugged these days than consumer grade crap.
But it has been a problem outside of data centers, especially where you're looking at small (analog) signals with high input impedence. Examples include medical monitoring equipment and scientific research equipment, and it's why you don't see established manufacturers of either using uncoated, unalloyed zinc electroplating, especially in humid environments.
we had rackmount Pc's that would have almost 1/4 inch of metal/sand dust on the motherboards and the computers were STILL working.
Contacts that are exposed to a "harsh" environment generally form a very thin non-conductive film due to oxidation and/or corosion. This would also apply to any conductive surfaces. Depending on exactly what gets laid down, what you describe seems completely reasonable. However, something in a "clean" environment can easily be killed by something your self-protected PCs wouldn't even notice.
There is a value to gold connectors, they don't tarnish, ergo, the connection doesn't degrade.
Is it worth the extra money? Not usually, but there is a benefit.
However, I do love when I hear people talk about improved 'sound quality'.
I always ask "So, the wires are gold as well?"
Usually the light downs.
I wil admit I have been out of the audio[hile loop for about 20 years. It seemed the widespread use popularity of CD's brought with it a large number of idiots who thought they where 'ausiophiles'.
Us a market around the edge of a CD to get better sound quality, indeed. bah
The Kruger Dunning explains most post on
I live within a mile of the ocean, and we have a lot of fixtures in our house which must be made of zinc, because they grow these whiskers just like were described. We have a chandelier in particular which looks like it's brass, but it's always covered with fuzz. Then I have a chin-up bar in a doorway and the same thing happens to it.
I wonder if it has to do with some kind of electrochemical reaction, where maybe there have to be different unlike metals with varying electronegativity, and enough humidity to get a low grade current flowing between them. I never saw this problem, when I lived elsewhere. But if my computers had zinc in them I'm sure it would grow whiskers just like the rest of my house.
Extraordinary claims require extraordinary evidence.
Risk = probability * loss. Since loss is obvious and fairly constant (a flaky or inoperative computer), if you want to hype this risk, besides just showing clear evidence that zinc whiskers are increasing the probability that a computer will die, you have to show to what degree that probability is increased - that is, that it merits more concern than other obscure things that can cause a computer to die.
Empty the computer room of all computers, and circulate the air through a filter that has somewhat dilute hydrochloric acid (Or whatever cheap stong acid) absorbed into it. The zinc will react to form ZnCl2 which will dissolve in the water. Shouldn't be to costly. Have one IT staffer and some grunts do it.
I hate grammar Nazi's.
Well you better start doing your homework.
I worked for many years on a replacement Air Traffic Control System for Canada and as the project matured, our stage - lab (containing litterally hundreds of machines, a complete lab recreation of the coast-to-coast ATC system) started to experience an MTBF on the power supplies in the equipment that was over an order of magnitude smaller than spec'd by the manufacturer (Hewlett-Packard).
Since this was a long-term contract that included commitments to deliver over an extended period of time (25 years), the material cost of this problem was VERY significant to the equipment vendor, not the customer. (In other words there was no financial motivation to fail to find fault, quite the opposite; the fault was costing them money.)
In the spirit of "old HP" they sent us some senior hardware design guys to look at our lab and our environmentals (humidity, temp, pressures, cycles and power suppy spectra) to see what was causing the problem.
Being about 6 years ago -- I hadn't heard of the Zinc problem yet, and neither had the guys from HP. They took everything back to their labs, including about 6 failed supplies and a couple 'still good ones', some from reserve stock and some from working machines.
A few weeks later they came back; there was a big meeting -- this was an issue with potentially enormous cost -- including the ultimate customer's representatives.
I can remember the Project Manager practically spitting his coffee when informed the underlying cause. The 'special ESD safe A/C'd lab' was part of the problem. Thank fully, the final deployed environment had different flooring, so we didn't have to change the sites, just some modifications to the lab.
This is far from BS -- it's a problem that has cost millions and will likely cost millions more before it's over. But the SEM photos of the failed devices we cool to see.
Switching power supplies can be surprisingly fragile. I've killed a couple working on TVs (that's basically what a flyback circuit is) and you can do it in one or two cycles (of your AC, not CPU cycles). And no, you don't want to know how much those power transistors cost. And I've killed computer PS by shorting across IC pins. There's not a lot of current going through these, and a 50-200 ohm short will definitely do the job. Remember, it only has to conduct long enough to nuke the chip.
Or maybe you'd prefer to ask the Cisco power engineer about it. Naah, he probably doesn't know what he's talking about.
there isn't even a link for zin whisker in wikipedia, but there is silver whisker and don't forget goatse is in wikipedia
steal this sig
First, I've done the 9V battery to steel-wool trick more times than I can count, and I've _NEVER_ seen any metal welded to the terminals.
Zinc != aluminum.
Aluminum oxide insulates it from further oxidation.
Zinc, on the other hand, is used as a sacrificial anode for electrolytic corrosion.
To do this, it must oxidize MORE readily than iron.
I've seen plenty of carbuerators that were pitted to the point of being useless that have never been near any beach. (the vast majority of carbs are cast from zinc as it's a very cheap metal and very easy to cast... low thermal expansion rate makes it stable in molds)
Either way you look at it, zinc DOES oxidize and zinc does have a low melting and low boiling point... these ultra-fine fibers would not be able to short a power-supply, ask anyone who's ever studied electronics!
IT CAN'T HAPPEN.
Q: You know what you call a super-fine fiber connected to a high-current source?
A: A FUSE.
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
I love this kind of stuff. Clueless moderators mod the parent insightful because none of them happen to have any personal experience with the subject. Lack of personal evidence does not constitute a conspiracy. Really, it doesn't.
/. need for anecdotal evidence, yet you groundlessly claim that the articles anecdote MUST have some other explanation. Why? I am all for being a little skeptical, but there is such little reason for it in this case. Would you begrudge them the opportunity to clean out their under-tile areas? Most of us know how bad those areas can get regardless of the whisker issue.
I have personally known about whiskers for over five years. It was becoming a potential problem in an old datacenter at work. It is a serious condition that datacenters with critical machines (or contractual obligations) take into account in datacenter design and maintenance. With 5 9's required for a lot of machines (Hitachi, IBM, Unisys, etc) there is little room for allowing electrically conductive dust particles to flow across every board on your machines.
The other posters have given examples to satisfy the typical
_damnit_
It's my job to freeze you. -- Logan's Run
Look zinc oxide ... is what people use in super density sunscreen .. hence the beach reference. I never said anything about aluminium and I did not, but should have mentioned electronegativity (why zincs are use as sacrificial metals not because of oxidation) Now go read a decent p-chem text and learn something.
Bad Panda! No Bamboo for you! In matters of importance ACs will not be responded to. Want to say something critical,OK
There is a value to gold connectors, they don't tarnish, ergo, the connection doesn't degrade.
Is it worth the extra money? Not usually, but there is a benefit.
Tin and lead tarnish yet it's still very common place. Silver also tarnishes yet is, or at least was common place on larger ships. Aluminium tarnishes practicly the moment you expose it to air. The wisdom to what you say is the fact that a copper tarnish aka copper oxide is an insolator.
I've been upgrading to gold connectors simply because i've been finding them for $3.99 per RCA pair. I don't claim they are the apex of audio cables, but they are decent quality and better then your mega-mart varity.
There is no sanctuary. There is no sanctuary. SHUT UP! There is no shut up. There is no shut up.
The real reason for gold plating has more to do with oxidation than the resistance of the different metals. Dending on where you live, those connections can develop problems fairly quickly. I also had someone point out that they paid a LOT of money for their equipment and don't want any dissimlar metal issues. While I'm not sure about the odds of that being an issue, who am I to quibble with him over a 100 dollar cable to hook up 3000 dollar rack equipment?
_damnit_
It's my job to freeze you. -- Logan's Run
problem happens in copper wiring small copper "whiskers" grow right through the insulation. if two wire are close when this happens they can short out and cause fires. usually only happens in really old wiring
No, _I_ mentioned aluminum, as aluminum has the properties you describe, not zinc.
Zinc turns to powder when weathered.
Aluminum does not (unless heavily exposed to salt-water).
Zinc will melt in the heat of a small gasoline fire (my cousin is a klutz and melted his bike's carb into a puddle).
Aluminum requires DRASTICALLY more heat to melt.
My point is, zinc DOES oxidize readily, especially when the surface area is great (what I've said repeatedly).
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
I'm curious as to what preparation process you used to preserve the fragile 3-D structure of the Zn whiskers. How did you differentiate the between the Zn whiskers from something like fibreglas insulation and other atmospheric floaties in your sample collection, or did you just plate everything?
Also, what kind of controls did you use?
(Curious because this may have an application outside of old floortiles.)
There are 1.1... kinds of people.
True, while this is a slight possibility, it is _very_ slight (as in life-on-mars slight), the likelyhood of this being a sudden all-at-once problem for an ENTIRE SITE, is... miniscule (as in life-on-the-moon).
I've worked in many server rooms with raised floors and zinc-plated support structures... many of which were so old they still had the (grandfathered) HALON systems in place.
NEVER have I seen a problem of this sort.
And most of these places have been fairly close to the beach in South Florida (read: humid and salty).
As far as killing a power-supply by shorting across one of its ICs... I'm quite sure you didn't accidentally short it with a resistor, it was a screwdriver, wasn't it?
I'm tired of typing this: The fibers they are talking about are TOO SMALL TO BE SEEN WITH THE NAKED EYE, that is the definition of "microscopic".
The amount of current required to vaporize a filament of this dimention (made of a poor conductor such as zinc) is miniscule!
As I said before, if they said it was causing bit-errors by contacting the data-bus, it'd be believable, but even the control ICs in power-supplies can sink more than a few tenths of a micro-amp!
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
You'd think that, wouldn't you....
In one of my jobs I got to set up an $800,000 prototyping shop with nice CNC equipment and all that. One of the toys I bought was a 350Amp Synchrowave welder. It was hard wired into the 460V main service we built into the building, all brand new freshly installed from the 10kv transformer to the disconnects. 600Amp 3 phase 460V service just to my little shop.
First night there I figured I'd try the welder out, flipped the disconnect, hit the start button on the welder and poop, out the lights went go. I turned the disconnect off, checked the fuses - they were fine. I went out to figure out what had happened. The 600A service breaker was popped.
I figured the welder had been wired wrong, opened the case. It was wired to the disconnect with, well, welding wire - stranded copper about an inch around. One - ONE - little tiny strand had unraveled and shorted across two phases. I bent it out of the way, buttoned up and it worked fine for the next 3 years. The main breaker never tripped again, not even when I was using the welder to blow holes through aluminum plate just for fun.
Now there's no way that little tiny wire could take 600A at 460V, and I can't think of a plausible narrative to explain why the main breaker tripped - but it wasn't just floating - the CNC had been running and it had a 10HP spindle.
A whisker doesn't have to survive conducting enough current to let the smoke out of the power supply - in it's incandescent passing, if the ghost of it's exisistance is a sliver of plasma, a very substaintial, if evanescent, conductor is created, literally, out of thin air.
I think that the flyback converter in a TV is rather different to a switching power supply in a PC. A TV needs high voltage to power the tube, while a PC needs, what? 12V at most! I don't think PC power supplies use flyback converters.
Now, with high voltage circuits, it is much easier to blow things up!
OTOH, I have seen several capacitors blow in power supplies. Most likeley explanation: faulty capacitor. These were electrolytics, which are notorious.
As it happens, I had 2 motherboards smoke on me today. Given that I used the same power supply, I'm guessing that the power supply has an over-voltage and that's what smoked the MBs.
Zink whiskers may be real, but I don't think they caused the problems I saw today and earlier when those capacitors blew up noisily!
The real "Libtards" are the Libertarians!
Come back zinc! Come back!
Zinc? Zinc? It was just a dream, zut my good computer still doesn't work. Damn you ZINC!!!
No, actually. Microscopes (in the traditional sense of the word) are optical devices. As hard as it is to believe in this day and age, most prisms and lenses don't require electricity.
Random and weird software I've written.
But then I thought to myself "too scary to be true" (especially the claims of asbestos-like hazard, in addition to the short inducing), and so I scrolled back to the cover page to doublecheck the date...
Hi, It just so happens that I work with this guy and he his correct in his statement. In fact my mom is the electron microscopist on site. Please note this is a VERY OLD electron microsope. around 8 - 10 years old. It doesn't have the ability to look at the lattice structure of a molecule because it simply isn't powerful enough. In our case the zinc whiskers where in the hundreds of microns in size. There is an attachment to the machine that can give you a breakdown of metals in the sample you are analyzing. In our case it was very obviously zinc. We have had problems with our new Sun equipment popping power supplies for several months nows (V480, 420R, etc.) although none of the Ultra 2's have had a problem at this point. We thought it was heat and power (thus the new AC unit for cooling) and the power checked out ok. The sun rep said we are the only company we service that was having supplies pop (in fact the data center across the hall wasn't having any issues but they have different tiles than us) when we had the cleaning crew in to clean under the tiles (prep for the under floor AC unit...note that we are using ceiling air now) In the span of 3 hours during the cleaning we lost 13 supplies all on new sun equipment. Once they stopped running vacs and pulling tiles it dropped off. Sun confirmed (with a sample of five dead supplies) that they were all caused by a short. Our tiles were wood core with hot dip galvanizing that had been recycled from an old computer room. These tiles had 20 years plus of use.
Given it's a recently understood phenomenon (in data centers), I doubt anyone can quantify the risk. I certainly can't. I'm not trying to hype a risk, I'm trying to dispute repeated claims that this is bogus or exceptionally rare. I certainly don't have any financial interest in this, I've got better things to do with my life than vacuum wood-core subfloor panels.
Oh, and metallurgy isn't my current field, but despite that I've run across this personally. So I started reading, and was amazed to find out how well known it was given I'd never heard of it. Turns out several engineers I knew (and one military data center guy) were familiar with it too.
Keep in mind what the parent of my post said: You simply cannot convince me that this is a real problem that we need to worry about. Not, that this is a common problem, or a problem everyone should worry about. His or her claim was much stronger than that: it it's either unreal, or we don't have to worry about it (i.e., infinitessimal or zero risk).
Furthermore, I found it telling that s/he said "you cannot convince me" instead of "you haven't convinced me". Not the best indicator of logical thought in my experience, but maybe it was just a poor choice of words.
Like I said, believe what you want. Just let me know if you're running a data center in case I ever need to colo.
Actually, in addition to have the potential to kill your computer, these fibers can do damage to your lungs to, just like asbestos, according to the PDF
Hi, Another note about how the electron microscope came into play. My mom took sticky sample pads and stuck them onto the bottoms of the floor tiles and then put the sample pad in the scope. In addition, we took the same sample pads and ran them over the input fan shrouds of the dead supplies. In the case of the floor tile samples the whole thing is pretty much just zinc whiskers. In the power supply samples it is a mix of dust and zinc.
"It's not my code, it's the zinc whiskers"
As I recall, old computer used to suffer from mice chewing on the wires and they used to keep a cat to keep the programs running smoothly. Then it was bugs and I am not sure if spiders helped or added to the problem.
I, for one, welcome our new zinc-eating nanobots.
If a /.'er falls in the computer room, does it make a noise?
No, this is real. Here's why: a tarnished connector can become a non-linear junction. Tarnish such as a thin film of a sulfide, in contact with a 'pure' metal, can be a rectifier. If you put one on a curve tracer, you could actually see this. So there really is merit in a gold, i.e., non-corrodable, connector.
it may delay whisker production, but it won't prevent it.
A whisker doesn't have to survive conducting enough current to let the smoke out of the power supply - in it's incandescent passing, if the ghost of it's exisistance is a sliver of plasma, a very substaintial, if evanescent, conductor is created, literally, out of thin air.
Thanks for this, gessel. In the aerospace industry, where I work, tin whiskers are the problem. In a vacuum (test or on-orbit), a shorting whisker can result in a plasma condition that will arc, allowing literally hundreds of amperes of current to flow for durations on the order of tens of seconds. This has been the cause of loss of many units and even entire systems. Of course, this is in a vacuum; in a non-vacuum situation, though, it is still possible to create localized low-pressure areas that might enable this phenomenon.
Whether or not this relates to the computer room problems of this article is not strictly relevant to the point I want to make. My point is: Many things happen in ways you wouldn't expect; intuition is not always your best guide to determining cause-effect.
Well, if those guys are involved, you just know it's a hoax.
No, the noise will be drowned out by all the wirring fans and hard disks...
Zinc is not attracted by magnets. The only ferromagnetic materials are iron, nickel and cobald.
Magnets will have no effect whatsoever.
What person will donate an airborne act of love?
Not every circuit in a power supply is high-current. The SMPS controller (eg TL494 and equivalents) are low-powered devices and have several terminals that will be operating with a few milliamps of current draw or less. For instance, shorting out the comparator sense input to ground would cause the output voltage to rise to way above operating values, thus blowing fuses or damaging other components downstream of the output.
It's plausible enough that on Monday I will be removing and inspecting *all* the 30-or-so floorbox lids in our comms room. It's too much of a risk to take to leave unchecked.
-- Sig Sig Sputnik
For a while I worked with heavily galvanized radio tower parts. I noticed that I would occasionally get splinters from these parts or that the surfaces of galvanized steel could sometimes feel crunchy the first time it was touched. I knew that I could get splinters from freshly cut sheet metal but I couldn't figure out how galvanized steel could have metal splinters, now I know.
Sweet. Lusers were getting wise to our PEBCAK and solar radiation excuses anyways.
"Extraordinary claims require extraordinary evidence."
I call BS. Extraordinary claims require exactly the same solid, persuasive evidence as all other claims. And once the evidence is provided, the claim is shown to be not extraordinary at all, but rather, factual.
DAILY ROTATION
Thermite is the stuff dreams are made of.
I don't read or respond to AC posts
As cut/pasted from my browser
HTTP 403.9 - Access Forbidden: Too many users are connected
Internet Information Services
GO NASA, GO!!!!!!!!!!!!!!! *WWoooooooooo!!!*
Life is like a box of chocolates, you never know when your gonna get food poisoning.
The EXACT mechanism that creates the metal (tin, zinc, etc.) whisker shaped crystals is unknown but LOTS is known:
What are the Commonly Reported Characteristics of Tin Whiskers?
The vast disparity in the observations reported by different experimenters is evidence of the complications associated with understanding and controlling tin whiskers. The following list is intended to provide a very basic overview of some of the observed characteristics of tin whiskers.
1.
Shapes: Whiskers may be straight, kinked, hooked or forked. Their outer surfaces are often grooved. Some growths may form as nodules or pyramidal structures.
2.
Incubation (Dormancy) Period: Experimenters report the incubation period may range from days to years. This attribute of whisker growth is particularly concerning because meaningful experiments to determine the propensity for a particular process to form whiskers may need to span very long periods of time.
3.
Growth Rate: Growth rates from 0.03 to 0.9 mm/yr have been reported. Growth is highly variable and is likely to be determined by a complex relationship of factors including plating chemistry, plating thickness, substrate materials, grain structure and environmental storage conditions.
4.
Whisker Length: Whiskers as long as a few millimeters are not uncommon with some experimenters observing whiskers as long as10 mm (400 mils) in length.
5.
Whisker Diameter: Typical diameters are a few microns with some reports as large as 10 um
6.
Environmental Factors: There is a great deal of contradictory information regarding environmental factors that might affect whisker formation. Several organizations are attempting to devise accelerated test methods to determine a particular plating process's propensity to form tin whiskers. However, to date, there are no accepted test methods for evaluating whisker propensity. Indeed, much of the experimental data compiled to date has produced somewhat contradictory findings regarding which factors accelerate (or retard) whisker growth.
Temperature: Some experimenters report that ambient temperatures of approximately 50C are optimal for whisker formation, while others observe that room temperatures (22C to 25C) grow whiskers faster. Reportedly, whisker growth ceases at temperatures above 150C
Pressure: Whiskers will grow in vacuum as well as earth based atmospheric pressure.
Moisture: Some observe that whiskers form more readily in high humidity (85% RH) whereas others report moisture is not a contributing factor
Thermal Cycling: Some experimenters report that thermal cycling increases the growth rate of whiskers, but others report no effect due to thermal cycling.
Electric Field: Whiskers grow spontaneously without requiring an applied electric field to encourage their growth. Some recent observations of tin whisker induced field problems in the commercial sector seem to suggest that an electric field could stimulate whisker growth, but more analysis is required to confirm these effects (if any). GSFC has demonstrated that whiskers can bend due to the forces of electrostatic attraction thus increasing the likelihood of tin whisker shorts (ref. to GSFC experiment #4).
7.
Whisker Prone Processes: There is tremendous debate in the industry regarding which plating processes are prone to whisker formation. Most of the literature agrees that "pure tin" electroplated surfaces (especially those that employ brighteners in the plating process) are the most susceptible to whisker formation. There are also reports that tin-lead plating can also grow whiskers; however, such whiskers are generally reported to be less than 50um long.
In their Best Practices Guide for the E10000 (written back in 2000) metions zink whiskers in 2.3.0.
Weird... So if Zinc Whiskers can cause system failures, what does breathing in 1mm/2mm strands of zinc do your your lungs? Silver Lung Disease? SysOps Disease? Breathing zinc can't be much better for you than breathing asbestos.
Maybe the reason why this phenomenon is starting to cause problems now is because the equipment fans *don't have filters*.
Thirty years ago industrial equipment usually had fans with nice tool-free pop-off covers so you could easily replace the filters. Often, you could replace the fans easily too.
Datacenter equipment used to be designed similarly.
Now even so-called "enterprise grade" models have a random selection of fans in hard-to-access places, and 99% have no filters at all. I don't know what the spec on such filters is, but I'll bet it wouldn't be hard to stop zinc whiskers.
Progress, shmogress.
Yikes, I'd be more worried about getting these things in my lungs...that can't be healthy.
What was debugging is going to be called shaving
You deserve some points for this, but as I've already posted in this thread, I'll have to give you an honorary +1 Funny. It's like an honorary degree, but not as good.
And before anyone points out that they're the same thing, they're not. Galvanizing involves dipping steel into molten zinc. Despite the name, it has nothing to do with galvanic action, electroplating or Mr Galvani. Galvanized panels don't suffer from zinc whiskers, anyway.
When I am king, you will be first against the wall.
Solid and persuasive evidence is, it seems, quite rare and extraordinary. For example in this case we have a breadth of anecdotal evidence but no quantifications - no statistics.
The phrase was popularized by Carl Sagan. Read a paper on it.
You actually claim that old problems are learned from and the mistakes are never repeated....
It might be true that new quality routines rule out the possibility of the same thing happening again, but quality routines don't communicate the actual cause and can't prevent similar things from happening out of the same couse.
Nowadays it has been reported that whiskers can form inside the computers themselves, because of the new lead-free circuits that are to replace every mainboard in a few years time and no filters can do anything about them there.
Feel free to bet away.
..
All I can talk about is my *actual* experiences, rather than your opinions.
Not all zinc products do this. I dont remember the exact details, but apparently it's in the cooling process when applying the zinc. A quick cooling cycle allows for higher levels of production - but also cause the metal to produce the zinc whiskers in time. A slower cooling process does not do this.
Oh, and remember, get in enough consultants, and you'll always find the answer you're looking for
Robert Anton Wilson
1. Make rework much harder. The plastic coating has to be carefully scraped away to fix a soldering error. Even with cheap Chinese labor it would be expensive.
2. Affect impedances. Commodity motherboards already have low (let's be honest: negative) timing margin. Random-thickness plastic coating over those long squiggly traces would make a bad situation worse. OTOH, it might not be so bad for the self-timed serial links that are becoming popular.
3. Have a higher failure rate. The spray will sometimes get onto connector and jumper terminals, and that sort of flakiness is way expensive.
First in years gone by there was rumours that seagate had a fluid filled drive, this was to protect against movement of the drive heads to allow more platters in the smaller frame. Second using a non-conductive liquid (such fluids exist 3M i think makes it, but right now it's expensive), you could have a completely sealed system, with a heat exchanger, outside of the case. This benefits the computer, in a few ways, no dust (which most people have a dust elephants in their computers); better performance(this fluid actually improves electrical flows with small gaps); quieter system, if needed you could move a fan to another area, or altenatively larger water reserves as well; lastly with a sealed case, you have better security of the system, which is more important in a work environment.
It's Fluorinert Here's an interesting thing for you overclockers, this guy got out of the Celeron 366 he got it to over 600 http://www.octools.com/index.cgi?caller=articles/s ubmersion/submersion.html
http://www.octools.com/index.cgi?caller=articles/s ubmersion2/submersion2.html
This is true in science as well as real life.
In this particular case, there is plenty of good evidence, which the poster who orginally went off on whiskers simply was unaware of and appeared to be unwilling to believe existed or could exist.
Being absolutely sure you know everything is a great way of surely being mistaken.
Zinc does grow whiskers. Ask anyone who's worked on a GE MASTR II radio (certain vintages) where the helical coils in the receiver front-end get shorted out over time by "dendrites" as they've been nicknamed in the radio world.
Walk over to the radio, give it a few solid whacks with your fist and the receiver gets a microvolt worth of additional sensitivity!
Interesting that NASA says they've seen these grow right back through their coatings they put on top of it. The "fix" for these radios is usually to remove the receiver casting and coils from the radio, clean thouroughly with whatever cleaner floats your boat, and then spray the inside of the receiver casting with clear spraypaint or shellac. Once that's done, the radio will operate correctly until the end of its service life.
+++OK ATH
...replacing the entire datacenter floor where I work because of zinc Whiskers. We had the underfloor area cleaned for the new A/C that blows from the floor up. We promptly lost 11 power supplies and the total count is up over 20 now. We did verify the presents of zinc whiskers in the dead powersupplies thanks to the onsite electron microscope.
You sure they are zinc and not steel? We had similar problems decades ago, where we got intermittant random memory errors. Turns out the blowers on the bottom of the frame were sucking up steel fibers left from the cleaners using steel wool pads on those huge floor buffers. They were being blown through the core memory arrays (yeah, REAL core memory).
Heisenberg may have been here.
Yes, it's solder mask ReK was thinking of (though soldermask used to be applied via a silkscreening process). The "silkscreen" is the text applied to the board: reference designators ("J2", "C25", etc.), PCB part numbers, company names, et. al.
.005" or less.
Regarding mlyle's comments about solder mask: production quality has improved greatly over the past decade. The unmasked areas are getting progressively smaller as the board manufacturers' ability to meet tight tolerances improves. It is now common for the gap between the mask and the exposed pads to be
Another post suggested the application of conformal coating to the board. This would work, but the folks in production - who build and test the boards - really don't like to do that. For those who don't know what conformal coat is, it's a plastic-like coating that is applied (in liquid form) to a circuit board. Once it is cured, it provides a relatively effective barrier to moisture, oxygen and corrosive pollutants.
It adds cost to the product, though, for several reasons: price of the coating material, man-hours spent masking areas not to be coated, applying the coating, sending it to cure, retrieving the cured boards, as well as added time spent repairing boards with faulty components found in the final board tests.
Personally, I support conformal coating boards - but then I'm a circuit board designer. The only work it adds for me is the addition of a note or two on the fabrication specs.
"A generation which ignores history has no past and no future." -- Robert Heinlein
This other page (from this /. comment) states that zinc whiskers are too small for normal dust filters, and it pretty much dismisses filtering as being a viable solution. (Of course, they're selling a non-filter solution so it's in their best interest to dismiss alternates...).
Check out Chad's News
NO.........We aren't stupid. My mom has been doing work with the electron microscope for more than 20 years now. you never re-use sample pads which is a standard rule when taking sample. Each sample was only used once and clearly marked where it was taken from so that they didn't get mixed up. In fact the power supply sample were taken a different day than the floor samples.
As I stated in a different post.....The electron microscope used for verification has a machine attached to it that can give you a breakdown of the metals in a givne sample. Each metal gives off a unique reading. It was very obviously zinc and not steel.