AT&T To Replace 17,000 Batteries

An anonymous reader writes "After four fires in two years — see earlier Slashdot discussions for background — AT&T is going against its own independent lab findings and declaring that the Avestor batteries powering its U-verse network aren't safe and need to be replaced. This is the network that SBC was building out prior to acquiring AT&T. Following the latest broadband equipment cabinet explosion in Wisconsin, the carrier says it will swap out 17,000 batteries deployed in several states across its network."

Comments

The Catch:

[Icarus1919]

AT&T will be replacing them with batteries that explode MORE often. The current frequency of explosion is unacceptable

Re:The Catch:

[Anonymous Coward]

Instead of exploding more often, they'll sell you a cheaper version of the same battery with some "features" locked. If you want to convert AC to DC, you'll need to pay extra.

It also comes with a hamster wheel if you feel like recharging it yourself.

Re:The Catch:

[robbiedo]

Are you referring to "Dr. Jacques von Hämsterviel?"

Avestor?

[Anonymous Coward]

Funny, I half-expected the batteries to have been made by Sony, not Avestor...

Metric

[Stanistani]

"Sir, we've experienced an explosive growth in customer satisfaction!"

Houston incidents

[I8TheWorm]

Said one AT&T employee in Houston...

"I don't know nuthin' about fautly batteries. But Houston is dang hot, and it ain't no dry heat. Things just catch on fire all the time. It's hotter than a whore in Sunday church down here."

Re:Houston incidents

[Locutus]

From what I heard, there are Microsoft Windows based servers in these boxes. So I wonder if they under estimated how hot that runs. poof! ;-)

And how come there are "Microsoft Windows" stickers all over those boxes like they are on all the OEM PCs and boxes? ;-)

LoB

But... why?

[Amorymeltzer]

Maybe I just don't know, but why on Earth do these things explode? It seems to happen with alarming frequency given the ubiquitous nature of these things - how hard is to make batteries or wires that don't catch fire when using them? Something like this has been happening a few times a year, and recalls or replacements aren't enough - punishments are in due order for making shoddy, dangerous products.

Re:But... why?

[$RANDOMLUSER]

Because customers demand quick-charge batteries - it's a dicey proposition - you have to measure the current draw pretty carefully, or the battery temperature itself. Picture filling a bucket with a firehose without overflowing the bucket and you get the idea.

Re:But... why?

[dgatwood]

Any time you store that much power in a confined space, you're taking a risk. If anything shorts out one of those batteries, whatever shorts it out will go up in flames because the battery dumps so much current through it. Usually, these fires are caused by an internal short triggered by dendrite growth, random metal fragments, impact, or chemical breakdown of the separator. When this happens, the little bit of shorting metal gets extremely hot. This starts a chain reaction, known as a thermal runaway in which the increase in temperature causes an increase in the chemical reactions in the battery, which, in turn, causes an increase in temperature.

http://ecsmeet3.peerx-press.org/ms_files/ecsmeet3/2007/01/03/00002421/00/2421_0_art_0_jbbdol.pdf
http://en.wikipedia.org/wiki/Lithium_ion_battery

Re:But... why?

[Locutus]

some have said that hot summer days were helping with the overheating of the batteries and the explosions. But, if you notice, the last explosion happened in late December in Wisconsin. Not likely to have been a very "hot" day. It could still be a heat issue if there's internal overheating do to "other" internal computer equipment, power supplies, and other equipment inside those boxes.

Don't expect AT&T to tell anybody what's really going on. After the dozens of images and stories that went out after the first explosion, AT&T is on top of these blown-up systems like white on rice.

Another interesting detail is that the company hired to examine the systems after the first two explosions said that the batteries and safety equipment were sound. They also said that they were likely better than most other batteries on the market. If this is the case, AT&T is going to have to start putting very large warning stickers on these boxes as they keep exploding. Maybe something like this:
http://www.flickr.com/photos/pluckytree/2186452007/in/pool-stickfiguresinperil/

or a version of this:
http://www.flickr.com/photos/blackloveinspace/494802125/in/pool-stickfiguresinperil/

LoB

Re:But... why?

[budgenator]

The Lithium itself is nasty stuff, but it's necessary to get the power densities needed, something that powerful is that powerful. The alternative is two have somebody try and call 911 and have a dead battery on the phone system.

Re:But... why?

[LWATCDR]

Except these are batteries in network cabinets. They are not in end user equipment. I doubt that they are quick charge batteries.

Re:But... why?

[DerekLyons]

Why wouldn't they be? Quick charge means they constantly (or nearly constantly) have maximum capacity available - which is the ideal state for a backup system (because it makes system performance predictable and maximizes uptime in the event of power outage).

Re:But... why?

[LWATCDR]

Quick charge means a lower life on the battery and really doesn't give a huge advantage. It just isn't a good tradeoff.

Re:But... why?

[Ignis Flatus]

several years ago, i had a job where we installed batteries in telecom shelters for Qwest, AT&T, etc. it was lead-acid batteries at the time. some of the guys were working on production, but even on hourly pay, there's still a lot of pressure to do the work in a timely manner. and invariably, the things get busted up. it may have happened during shipping, or during assembly. maybe the guys putting putting the connections on overtorqued them. maybe it was even a factory defect. but whoever is at fault, it inevitably happens. with lead-acid batteries, it's often pretty obvious. you get a crack in a case, and the thing leaks all over the floor. we always carried baking soda with us for this reason. maybe with a lithium battery, the defects are just not immediately evident. but from what i've heard wrt to lithium batteries on hybrid vehicles, lithium is just hard to deal with. too much energy density makes for more mishaps, i guess.

Re:But... why?

[rjune]

I live in the Milwaukee area and there was absolutely nothing about this in the local news. We subscribe to the local newspaper (Journal-Sentinal), visit the Journal-Sentinal website on a regular basis, and listen to the radio. If the local media is ignoring the story, why should AT&T talk about it. By the way, guess who bought a boatload of advertising on the above mentioned local media to lobby for a change in Wisconsin cable law? The new law allows AT&T to bypass local municipalities to get a franchise.

Re:But... why?

[nacturation]

Why wouldn't they be? Quick charge means they constantly (or nearly constantly) have maximum capacity available - which is the ideal state for a backup system (because it makes system performance predictable and maximizes uptime in the event of power outage). Think about what you're saying. These are backup batteries and will ideally never get used. What does it matter if your batteries charge in 30 minutes if they're only used two times a year? You could have batteries which take days to fully charge because at the time of an outage, what difference did it make that the batteries charged in 30 minutes and then sat there at full charge for half a year? Plus, quick charge batteries are likely more expensive, have shorter lifespans, and have worse performance.

Re:But... why?

[DerekLyons]

What makes you think I didn't think about what I said? That I proposed a solution based on sound engineering rather than guesswork?

Re:But... why?

[ScrappyLaptop]

Think about what you are saying. Think about your boss. Think about your boss making the decision of which batteries to purchase. Consider that he knows next to nothing about technology. Consider that he knows everyone under him knows *everything* about technology. Now tell me which ones he'll choose: The old-fashioned, slow ones or the faster ones labeled as 'advanced'? C'mon, everyone knows that when it comes to technology, faster is *always* better. The marketing folks (with whom your boss identifies more than with you) say so!

Re:But... why?

[cheater512]

Why would they use lithium?
Here in Australia I believe they use lead acid.

Makes far more sense. Lithium batteries lose charge over time regardless of usage.

Re:But... why?

[budgenator]

The batteries go in metal boxes scattered around most towns and are about 1M wide, 2M tall and about 2m long and they contain equipment to convert from optical fiber to copper for voice and DSL. Most people in the US live too far from the central office for DSL so the teleco's made mini-CO. The enviro-Nazi's would have a field day over lead-acid screaming "OOOHH, heavy metal, poison the kiddies, think of the children" and the "proper disposal" fees might make them more expensive per watt/hr of capacity.

Re:But... why?

[cheater512]

Are you serious or is that just speculation?

If your serious then Wow! God Americans are stupid. ;)
(Enviro-Nazis should realize that their cars use lead acid)

And btw Australia has the same system where every suburb or so has a telephone exchange.
I dont think they have batteries though. The small telephone buildings have the lead acid batteries.
They send the power to the exchanges.

If they use Lithium Ion then they need to be replaced every 2 - 3 years.
Lead acid can go for decades in good conditions (which exchanges are).

Re:But... why?

[nacturation]

What makes you think I didn't think about what I said? It was the contents of your post that indicate you didn't think it through. Yes, you're correct that having a system always charged is the best state to be in. However, having batteries fully charged is independent of how fast they get charged. If the power is going out so frequently that the only way to fully charge the batteries between outages is by having a quick charge battery, then I'll agree with you.

Re:But... why?

[nacturation]

Think about what you are saying. Think about your boss. Think about your boss making the decision of which batteries to purchase. Consider that he knows next to nothing about technology. Consider that he knows everyone under him knows *everything* about technology. Now tell me which ones he'll choose: The old-fashioned, slow ones or the faster ones labeled as 'advanced'? C'mon, everyone knows that when it comes to technology, faster is *always* better. The marketing folks (with whom your boss identifies more than with you) say so! Lucky for me, my boss isn't an idiot and appreciates solid engineering. Let's see... quick charge, but reduced cycles meaning more frequent replacements, higher cost, and decreased output which overall increases total cost of ownership. Or, slower charge, but longer life, lower cost, and increased output which reduces total cost of ownership. Either way, the batteries will have a full charge in the event of an outage.

Re:But... why?

[DerekLyons]

Ah, yes. Seeking to maximize uptime, system reliability, and performance predictabilty is a symptom that I don't think. You're an idiot.

Re:But... why?

[Anonymous Coward]

Lithium. The lead acid, NiCd, and NiMH batteries would certainly be damaged by short circuits and the like, but Lithium Ion batteries and the like tend to go into a runaway condition and explode or burn instead. (These battery fires on notebooks have usually been due to damaged or faulty construction of batteries causing internal shorts. The UVerse batteries are some lithium polymer.. but still, with construction faults or damage, probably blew due to internal shorts.)

          The quick charge doesn't help, but there used to be quick-charge NiMH batteries and they did not burn (they would still fail if charged to aggressively, just not violently fail.) The lithium is the main culprit. In particular, I think it's not specifically the energy density of the lithium ion and lithium polymer batteries, but the possible extremely high discharge rate.

Re:But... why?

[budgenator]

Are you serious or is that just speculation? Both, but these things are just huge ugly metal cabinets outside where they get too hot in the summer and too cold in the winter.
If your serious then Wow! God Americans are stupid. ;) yes indeed we've had some towns ban dihydrogen monoxide as a hazardous chemical!

Re:But... why?

[nacturation]

Ah, yes. Seeking to maximize uptime, system reliability, and performance predictabilty is a symptom that I don't think. You're an idiot. Actually you did think of that. What it appears you're missing is that a slow charging battery achieves the exact same characteristics as a quick charge battery. Both will be fully charged at the point they're needed. And if slow charge and quick charge batteries deliver the exact same performance characteristics and longevity with the only difference being charge time, then of course you'd be right... go with quick charge. Since I'm not a telecom engineer nor do I design batteries, I'll shut up now. :)

Re:But... why?

[macdaddy]

Think about what you're saying. These are backup batteries and will ideally never get used.

The batteries are not just for backup purposes. They are in service 24/7/365. The cabinets themselves run off of -48VDC provided by the string of batteries. The cabinets do not run off of VAC or unregulated VDC. Therefore the batteries are always in production, not just when you VAC input is unavailable. I work for a telco.

not a mention of the recycling details

[Locutus]

I wonder if anybody is watching to see what they do with all these batteries?

LoB

Re:not a mention of the recycling details

[calebt3]

what they do with all these batteries? Sell them to Sony.

Re:not a mention of the recycling details

[ScrewMaster]

I wonder if anybody is watching to see what they do with all these batteries?

Probably send them to Iraq. Just let the terrorists steal them to power their equipment.

Re:not a mention of the recycling details

[epp_b]

what they do with all these batteries?

Sell them to Sony.

You mean sell them back to Sony?

Re:not a mention of the recycling details

[calebt3]

No. These batteries came from a company called Avestor. Unless that is the name of the brand under which Sony sells batteries.

Re:not a mention of the recycling details

[lhaeh]

I seem to recall reading a National Geographic article about kids in Mexico employed smashing open lead-acid batteries with hammers to extract the lead. I'm sure there are greener ways of recycling them, but the heaps of e-trash in Gungdong leads me to think a lot of old batteries end up there for that kind of recycling. Since these batteries are still good, I can see them ending up in a surplus auction. I tend to stock my UPS with lightly used surplus batteries and they are good for nearly as long at a fraction of the price.

There's nothing good hearted about this

[ip_freely_2000]

Call me a cynic, but I'm sure they put a formula into a spreadsheet and discovered the liability issues outweighed the "do nothing" option. I'm sure there's a Ford Pinto kind of memo on a AT&T server somewhere.

Re:There's nothing good hearted about this

[calebt3]

Your a Cynic

Re:There's nothing good hearted about this

[calebt3]

Ghaa! You're, not Your.

Re:There's nothing good hearted about this

[Anonymous Coward]

Narrator: A new car built by my company leaves somewhere traveling at 60 mph. The rear differential locks up. The car crashes and burns with everyone trapped inside. Now, should we initiate a recall? Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.
Business woman on plane: Are there a lot of these kinds of accidents?
Narrator: You wouldn't believe.
Business woman on plane: Which car company do you work for?
Narrator: A major one.

Re:There's nothing good hearted about this

[petermgreen]

which works fine until one of the cases actually hits court and this behaviour is disconvered during the case.......

Re:There's nothing good hearted about this

[maxume]

The other way of looking at it is that it means the legal system is more or less working...

Re:There's nothing good hearted about this

[VTMarik]

Still, it's a rare occasion when a company goes against the advice of a study that says "Everything is fine" instead of insisting that people trust the study and not worry about the potential explosive in their devices.

Sure, it's cheaper to swap it out then pay a lawsuit but when has that stopped a large company? It was cheaper to make sure the Ford Pinto didn't explode than to lose millions in sales when people just stopped buying them.

Re:There's nothing good hearted about this

[DerekLyons]

It's not cynical - it's just recognition of good business practice.

Re:There's nothing good hearted about this

[Anonymous Coward]

Pinto? More like Ford Everything. Last year I got a recall letter for my Ford Ranger that says the cruise control deactivation switch is faulty and cause fires. Not just when the car is running, but when it's parked, the engine shut down, keys out of the ignition. After doing some research, I found out that this faulty switch has caused hundreds of fires (not 4 like these AT&T batteries), sometimes burning down houses when the fire starts in a garage.

Not only that, but Ford has known about this since at LEAST 2005, which was when they issued the first recalls - but these recalls only applied to a portion of their cars that used the switch. Since then, they have recalled more models (my Ranger being one of them), but there are STILL Fords on the road with this faulty switch that haven't been recalled yet. But you won't hear about that in the media, I guess they're all in bed with Ford now.

By the way, the recall letter said I could go to the dealership and get a new fault-free cruise control deactivation switch installed. But of course, they didn't have any. They said they would have them in by November (funny thing, it's January and they still don't have them). So all they did was disconnect the switch, thereby disabling my cruise control and making the ABS light come on. A problem which Ford is denying exists, but dozens of members on auto enthusiast forums confirmed it is an issue. Sort of like the Comcast BitTorrent throttling, where they deny it up and down, but independent tests confirm it's obviously happening. Hopefully my ABS still works and it's just the dashboard light that's going off.

Fuck Ford. I'm buying Japanese next time.

Re:There's nothing good hearted about this

[ScrewMaster]

That and earning a lot of good PR. Most of us don't mind so much when companies make mistakes (we all recognize that, well, sometimes shit happens.) It is when a corporation goes into deny/ignore mode that we get pissed off. This will cost AT&T a bit of cash, sure, but they deserve that for buying crap batteries in the first place. Furthermore, imagine the hot water they'd be in if this started happening again. Somebody upstairs decided not to take the risk.

Smart move.

Re:There's nothing good hearted about this

[p0tat03]

Easy... estimate the dollar value of the public relations disaster you will face, times the probability of being discovered... if it's STILL less than the cost of recall...

Re:There's nothing good hearted about this

[iminplaya]

Fuck Ford. I'm buying Japanese next time.

Yeah! Get a Mazda

Re:There's nothing good hearted about this

[zippthorne]

Ah, but is this really an immoral policy?

If your ethical system involves doing the most good for the greatest number of people, you're going to have to do a lot of calculations like that. I'd say, depending on how the settlement process works, it's at worst, amoral. At best, it's actually the most ethical course of action possible. Coldly, calculatingly ethical.

Re:There's nothing good hearted about this

[EmagGeek]

I can think of several reasons not related to civil liability to go ahead and replace the batteries

1) The cost of the VRAD equipment is astronomical when you consider the labor of sorting out hundreds upon hundreds of burnt copper and fiber cables and replacing them at union labor rates
2) The cost of losing customers due to the unreliability caused by failing VRAD modules
3) The indirect cost of investors worrying about future revenue, depressing share price

The batteries from Avestor were NOT cheap, as has been insinuated elsewhere in this thread. As much as people bitch about telecom companies doing things "on the cheap" to make money, it's not true. The cost of servicing "cheap shit" in the field is far higher than simply buying "good shit" in the first place. This is why Comcast uses cable that costs $2-$3/ft in underground utility installations, rather than using cheap RG-9X from Home Depot.

The same logic goes for the batteries. They would much rather pay $500 for a battery than pay $250 for a battery and then another $500 to have a union contractor go out and replace it with another $250 battery - or God forbid, another $20K to replace the whole VRAD module when it spontaneously goes into low Earth orbit. That's what we call "cheap insurance."

In this case, it just so happened that the "cheap insurance" didn't work out.

They're getting 17,000 new batteries?

[Chris Mattern]

No wonder my drugstore is out of AAs!

Chris Mattern

Achmed the Terrorist Energizer Bunny

[Anonymous Coward]

Come on, you all want to see the commercial where the voiceover says: "Energizer Bunny keeps going and going and go- KABOOM! For fuck's sake, can't we find a bunny who hasn't turned suicide bomber?! Didn't airport security check the interior of his drum?"

Even better would be Achmed the Dead Terrorist dressed up in an Energizer Bunny outfit. "Hi, I am Achmed. I keep going and going and... I KEEEEEIIILL YOU!!". If you haven't yet seen it, check out Jeff Dunham's Achmed the Dead Terrorist act. IMO, it's absolutely hilarious.

Re: Fight Club quote updated

[NFN_NLN]

which works fine until one of the cases actually hits court and this behaviour is disconvered during the case.......

Take the number of [soldiers] in [Iraq], A, multiply by the probable rate of [deaths], B, multiply by the average [veteran pay to the family], C. A times B times C equals X. If X is less than the cost of [buying proper bullet proof jackets and light armor], we don't do one.

Nope, looks like it doesn't matter even if people know.

NFN_NLN

Another sign of VRAD overheating observed?

[Lauren Weinstein]

Out here in the West San Fernando Valley (Los Angeles) -- and it can get *very* hot here during the summer -- where AT&T has widely deployed VRAD cabinets (though U-verse is not activated here yet), I've recently noticed another sign of possible overheating problems. On the side of the VRAD cabinets is apparently a large air intake with an exposed filter element. On several units I've observed recently, the filter element first vanished completely, and then was replaced shortly thereafter with what appear to be rather bulky external fan units. Interesting. --Lauren--

Test run?

[CustomDesigned]

Amen. Shouldn't they replace say 20% - 50%, and see if any of the problems are in the new set? Maybe the problem is the batteries environment, and not the battery itself.

New Godzilla

[Anonymous Coward]

I can already picture a new Godzilla movie with exploding cell towers as he rampages through the city. Now, that would be fun.

And citizens throw their laptops at Godzilla to stop him...

This is exactly why it is a bad idea

[neapolitan]

to offer 'integrated' batteries into devices.

Imagine what a recall of the iPod or Macbook Air battery would do to Apple's share price.

(Now smile to yourself, quietly.)

New Battery technology

[KD7JZ]

The Avestor batteries were a new innovation for powering telecommunications network elements. They were a lithium based chemistry (LiIon) instead of traditional lead acid gel cells. The advantage was supposed to be about twice the service life (6-8 years) instead of the traditional 3 to 5 years. Lithium is a reactive element and if they were having a problem where an internal short was causing thermal runaway, you have a reactive element combined with heat. Not a good thing...

Re:New Battery technology

[afidel]

Hmm, I wonder where I can get some of the pulled batteries. Stuff like that usually goes for a penny on the dollar or less. I know it would be plenty cool in my basement and would do wonders to extend the life of my Matrix 5000 =)

Avestor batteries caughting on fire, really !!!

[Anonymous Coward]

Have a look at the pictures of two of the fires at Avestor plant:
http://www.incendielongueuil.com/interventions/galerie/index.php?gallery=./Interventions%202006/06-%20Juin/1600decoulomb
http://www.zone911.com/ARCHIVES/sept-2005-18et24-Archives.html (top 2 pictures)

Who could have thought that Avestor's batteries could caught on fire ??

Oh, just great...

[Anonymous Coward]

...another excuse for them to crank-up their billing charges.

More info on WI explosion

[bigbadunix]

Being from Wisconsin, and not hearing about this, I did a little search and found this.

Funny thing is, it was a less than a mile from my house, and probably a week after 2 friends of mine in the area signed up for u*verse.

The article does a good job basically re-iterating TFA with a bit more detail.

Re:More info on WI explosion

[scottv67]

Funny thing is, it was a less than a mile from my house, and probably a week after 2 friends of mine in the area signed up for u*verse.

I hope you live a mile west or a mile south of 64th and North. A mile east would put you in the ghet-to. And a mile north of there ain't such a sweet place to live either. Can you post some photos of the new cabinet that ATT installed at that location? You mentioned that some of your friends ordered uverse. Are they happy with uverse? I have seen a million white ATT trucks running around Waukesha but I don't know anyone who has uverse.

Re:More info on WI explosion

[Anonymous Coward]

The Free Press article's source is saveaccess: http://saveaccess.org/node/2018 . There are photos of the burning telco cabinet with the article (the photos were sent by a local resident to the site). There's also a follow-up story at: http://saveaccess.org/node/2044. Apparently local residents were well informed and warned of potential problems as much as a year before.

I really doubt if external heat played a factor, at the time of the explosion in Wisconsin there was snow on the ground, though it may be possible that this creates an even more unstable situation in warmer climates.

In any event, AT&T has now made a public statement that they will replace all 17,000 batteries - hopefully before another incident and any potential injuries.

AT&T Servicing iPhone batteries?

[kcbanner]

Read the summary? Surely you jest!

Placing shields over a 2 meter vent...

[matrixghost1286]

Looks like the Empire decided to take preventative measures to ensure power generation.

I guess the Rebels will have to find a new way to blow up the Death Star.

Avestor was already bankrupt

[Anonymous Coward]

This is costing AT&T a lot of money, Avestor is bankrupt so they won't pay for it.
http://www.greencarcongress.com/2006/11/avestor_shuts_d.html

Avestor went under in 2006

[EmagGeek]

I wonder why...

Needless to say, AT&T will not be able to recover anything from them. Hopefully none of the NiCd replacements explode - being incredibly toxic to the environment and all...