Another Battery Fire in AT&T's Network

An anonymous reader writes "AT&T has disclosed another fire started by one of the 17,000 Avestor batteries in its broadband network. The first fire caused a violent explosion in suburban Houston. This second incident occurred just 20 miles away."

Comments

Good thing Houston gets rain...

[192939495969798999]

I can't imagine if one of these went off in say, northern El Cajon California. There would probably be a huge brush fire at least, all it takes is one spark to hit that bone-dry foliage. Does anyone know if these batteries are installed all over or just in TX?

Re:Good thing Houston gets rain...

[Anonymous Coward]

But think of all the hookers we could get off of El Cajon Blvd. if one of those went off around there.

Not called "Devil's Coffin" for nothing

[Anonymous Coward]

Not called "Devil's Coffin" for nothing. Way too many panhandlers to boot.

Verizon?

[Anonymous Coward]

Wasn't the other under Verizon?

Re:Verizon?

[Anonymous Coward]

No.

Wow

[Durrok]

From TFA: "Outside of these two incidents, there have been no similar incidents involving these batteries," AT&T's spokesman writes via email.

Anyone else read ".... YET" in that sentence. I'm hoping they are doing further testing in whatever conditions these batteries were in that made them explode. TFA did not mention anything but then again it is light reading.

Re:Wow

[smashin234]

"Anyone else read ".... YET" in that sentence"

Yes, I did, take the quote from the bigwig:

"the battery design was sound, as were the safety features," and concluded that "the risk of hazardous failures with this battery is as low, if not lower, than the risk with alternative batteries, which are used by other telecommunications and cable companies in similar applications."

IF the battery design was sound, why did 2 of them start fires?

Re:Wow

[Monoliath]

Just wait for it, the word of the decade (terrorism) is going to come up in just a matter of time...you watch.

Re:Wow

[Durrok]

Well it's hard to say, many electric devices can start fires/malfunction if treated incorrectly. Were they exposed to water? Intense heat? Corrosive chemicals? If the situation for those two batteries was far beyond the norm of use then it's not necessary to replace all the batteries, just to post notice or possibly move the boxes to a better location (Which would be a lot more work then just replacing the batteries I'm sure). I'm not saying they should go recall all of these immediately but I really hope they are looking into it further.

Re:Wow

[smashin234]

Maybe I am just pessimistic, but the fact that the bigwigs do not explain the circumstances behind the fires makes me unsettled. Sure, there could be a natural explanation (Summer heat + some unfortunate other circumstance) or misuse, but what troubles me is that the bigwig said the batteries were safe without a full investigation.

Now maybe this is for legal reasons, but at the very least they should explain why this happened and how to prevent it from happening again. Or if the batteries are indeed safe, tell us that with proof.

Re:Wow

[gratemyl]

Don't be silly - nobody would make an unreasonable conclusion that this was a terrorist attack - after all, everybody knows that Al Qaeda planned this attack so carefully that nobody would notice.

Re:Wow

[grumling]

If they were within 20 miles of each other, I'd look at the other ones nearby and find out who installed them. Maybe the contractor needs a little more training, maybe they got a bad batch of batteries.

It is likely they were installed by the same group of people (person?), and if they (he) didn't know what he was doing, it may very well happen again. It is also likely the batteries were all part of the same batch.

Re:Wow

[Jeff DeMaagd]

I think it definitely needs to be investigated. Two batteries in 17,000 randomly failing is one thing (and probably not a bad rate, assuming none of the other 17,000 only failed), but starting fires is another issue altogether.

Re:Wow

[The_Wilschon]

Were they exposed to intense heat? Well, they were in Houston after all.

Re:Wow

[maeka]

If they were within 20 miles of each other, I'd look at the other ones nearby and find out who installed them. Maybe the contractor needs a little more training, maybe they got a bad batch of batteries.

It also should be noted that (depending on your source) Houston, TX is the second hottest city in the United States, when looking at daily mean.

Re:Wow

[P3NIS_CLEAVER]

I would suspect that heat has something to do with the failures. Houston is pretty hot.

Re:Wow

[Anonymous Coward]

it's very likely it was the same crew within a contracting company, or even the same individual who installed both of these batteries and many more nearby. it's quite common for one crew to go to an area and complete several jobs as part of a larger project and be staying in the area for quite a while. the longest i have personally heard of is 5 months for a string of 40 or so jobs.

Re:Wow

[Anonymous McCartneyf]

Were these batteries exposed to water? Maybe--there has been a lot of rain in the Southern Plains this year. Tropical Storm Erin just passed through, but there was enough rain before that.
Were they exposed to heat? Yes. It's been a very hot summer this year, or at least this month, in the Southern Plains when it hasn't been raining. Nineties or 100s(Fahrenheit).
Corrosive chemicals? Probably not. But these batteries are li-poly! They can explode without help from other chemicals.

Re:Wow

[ScrewMaster]

It is also likely the batteries were all part of the same batch.

Yes, I understand that Avestor made some twenty thousand of the things before going bankrupt.

Why that kind of battery?

[Anonymous Coward]

"lithium metal polymer"???

If you're doing battery backup in a fixed location, what advantage do you get from using sophisticated battery chemistries? Weight shouldn't matter. Size shouldn't matter. If you're powered off the grid, efficiency doesn't even matter. The last time I checked (quite a while ago), the standard choice was good old lead acid. Properly maintained they last forever and are the most cost effective choice. Does AT&T know something I don't or did someone just make a bad engineering decision.

I forgot to mention: lead acid doesn't suffer from thermal runaway which is the major cause of battery explosions.

Re:Why that kind of battery?

[zeromorph]

You have to distinguish between two types of lead-acid batteries and then the whole thing is not that optimal anymore.

With wet-cell lead-acid batteries you'll get evaporation and resulting loss of capacity. Beside that you have a precipitation of lead(II) sulfate that can ultimately kill your battery.

With maintenance-free (sealed) batteries like Valve Regulated Lead Acid batteries you do not get these problems but you could get thermal runaway and they do explode. Gel-batteries are less inclined to explode but especially older ones do that too.

On the other hand lithium metal polymer are said to "have service lives as long as 10 years, under ambient temperatures from -40C to +65C."

Re:Why that kind of battery?

[Anonymous Coward]

The lead-acid batteries I'm familiar with are large, have clear cases, lots of room below the plates and probably quite wide spacing between the plates. Banks of them fill large rooms. They are one of the reasons that POTS (plain old telephone service) works even when the power goes out everywhere else. They are also the reason why air traffic control towers continue to work even if the diesel generator doesn't start quickly. All of the problems you mention don't affect them.

Having said the above, even regular car batteries last a very long time in backup service. There's not a lot of current happening so they don't have much of a problem with sulphation. You do have to check the water level every x months. My former employer had a lot of remote equipment and used heavy duty car batteries for backup. In the ten years I spent with them, I don't remember having to replace even a single battery.

The one thing I do remember having to address was acid vapor. The batteries were never in the same space as the equipment.

Re:Why that kind of battery?

[Anonymous Coward]

The last time I checked (quite a while ago), the standard choice was good old lead acid. Properly maintained they last forever and are the most cost effective choice.

No, lead-acid batteries are far from immortal. Rated life under ideal conditions is 5-10 years for "maintenance free" VRLA batteries, but real world use in an outdoor installation is nothing like a laboratory test (20-25C, few discharge cycles, never deep-discharged, full recharge before every discharge...). Typical field service life is less than that published in the spec sheet. Compare the warranty period to the advertised life rating: normal service life is between those limits. In hot climates such as Houston, two years of reliable service is doing well. The reliability engineer's rule-of-thumb that increasing temperature by 10C cuts life in half applies: this is a case where the wear-out mechanisms really are chemical processes governed by the Arhennius equation.

Lead acid batteries, especially "maintenance free" VRLA types, are subject to thermal run-away. High charging voltage (lack of proper temperature compensation) or shorted cells in the battery can initiate a run-away failure, generate excessive gas, even rupture the case if the gas generation is more than the pressure-relief safety vents can handle. Accidents can and will happen with any type of high power battery.

Re:Why that kind of battery?

[NoMaster]

On the other hand lithium metal polymer are said to "have service lives as long as 10 years, under ambient temperatures from -40C to +65C."

Funny thing is, I've got here plenty of manufacturer data sheets claiming pretty much the same thing for SLA / VRLA batteries.

In practice, for ambient temps between 15C & 40C, service life seemed to be ~ 3 years max. And not much, if any, better in a regulated 25C ~30C environment.

OTOH, I remember these being used to replace batteries of flooded lead-acid cells, subject to a fairly well regulated 20C ~ 30C, that were still in near-perfect condition - no silting, no visible sulphation, no chlorine poisoning, minimal post corrosion, discharge test results of > 90% capacity @ C10 rate, etc - despite being more than 30 years old.

Yeah, traditional flooded cells require at least yearly maintenance; preferably 1/2 yearly or quarterly. But, as I've mentioned here before, I've seen SLA batteries go from visually/quantatively OK to "oozing their guts" in less than 3 months...

AT&T: One Hot Telephone Network!

[davidwr]

Of course, when the hottest thing you have to offer your customers is an actual fire it's not quite the same.

Did Anyone

[phoenixwade]

The two batteries referred to were housed in two different VRAD cabinets, 20 miles apart, in the same city within a few months of each other. But, the carrier says, apart from those two batteries, all else is well. In its statement, published earlier today, AT&T said "the battery design was sound, as were the safety features," and concluded that "the risk of hazardous failures with this battery is as low, if not lower, than the risk with alternative batteries, which are used by other telecommunications and cable companies in similar applications." Really? the ubiquitous SLA has the same track record? What a crock 'o poopie. Another one of those "I think I'll pull statistics out of my ass and spout them off" moments....

What I can't figure out is why they aren't using SLA's in those cabinets in the first place. They are, by far, the most common battery in use in almost every application. The advantage is that SLA's are safe, predictable and cheap. The disadvantages are volume and weight, but in a stationary cabinet that shouldn't make a difference.

Doesn't surprise me....

[LWolenczak]

It really doesn't surprise me. Sealed cell VRLA Battery + High Temperatures (Summer) + Hydrogen creation inside battery from the Water/Acid mix + brown out causing battery to be utilized = practical bomb. I had one cook off in a generator this week. One of my data center generators went to start up for its weekly exercise this week and it never started, but one of the batteries exploded. It didn't lead to a fire, but I'm sure it would have if it was in a more confined space.

Re:Doesn't surprise me....

[SigILL]

Sealed cell VRLA Battery

Er, read the article (again). They're Lithium Polymer (LiPo) batteries. Remember those exploding Lithium-Ion laptop batteries a couple of months ago? These are even more dangerous. This should give you an idea... and that's a small one.

Re:Doesn't surprise me....

[Anonymous Coward]

The article said it was a Lithium battery, it happened in Houston in January. Maybe your theory is wrong.

Re:Doesn't surprise me....

[LWolenczak]

Whoops, I have been corrected. I would have assumed VRLA given thats the standard, at least inside central offices and DC powered data centers. If its a lithium battery, then they are just plum overcharging it.

Re:Doesn't surprise me....

[LWolenczak]

FWIW anyway, there is a similar risk with wet cell batteries, but they are not small, and they are typically monitored very closely. Most rooms I've seen wet cells in have hydrogen sensors and ventilation.

Re:So that's what happens if you run Windows Embed

[Anonymous Coward]

I guess I'm allowed to have a joke once a day

Jokes are meant to be funny.

Put a lot of energy in a small place...

[grumling]

And you might end up with a little problem.

http://www.askthepilot.com/upsfire.html

Story that explains the picture:
http://www.salon.com/tech/col/smith/2006/09/22/ask thepilot202/print.html

Granted, Smith is a pilot, not an engineer, but he gives a somewhat good explanation of thermal runaway.

Re:Put a lot of energy in a small place...

[Anonymous Coward]

How do people read Salon.com and manage to walk around with a straight face? I've never seen such a liberal piece of rubbish in my entire life.
They try to pass this stuff as news. They should be forced to put the word "Editorial Opinion" on every "news" article.

Are we sure people aren't the accelerant?

[writermike]

Holy GOD it's hot here.

Re:So that's what happens if you run Windows Embed

[X0563511]

You could try cracking a joke that had something to do with the article, or at the very least to a comment.

That post was a waste of all our time, and a moderators mod points, and your Karma.

Why does AT&T use such huge cabinets?

[kilodelta]

Verizon's FIOS is in my area. I won't subscribe because I hate Verizon with a passion, but I've got photos of their cabinets and they look nothing like at&t's cabinets.

http://www.flickr.com/photos/kd1s/553921527/in/set -72157594534810671/

Re:So that's what happens if you run Windows Embed

[Loconut1389]

FWIW- I see the joke you were trying to make. Just imperfect delivery I guess. Mods can be very jumpy with the MS topic

Avestor Itself Blows up

[index.html]

Aventes has filed for bankruptcy:

http://www.greencarcongress.com/2006/11/avestor_sh uts_d.html

Avestor, the Canadian developer of Lithium-Metal-Polymer (LMP) battery technology, is shutting down. The company filed with the Office of the Superintendent of Bankruptcy in Montréal with a view to making a Proposal to its creditors on 31 October.

In August, the company had produced and shipped its 20,000th battery. At that time, Avestor said it had signed multimillion dollar, multiyear contracts with major telecommunications service providers in North America and also was a provider to several other telecommunications customers in North America.

"Second Hottest"

[Anonymous Coward]

Houston, TX is the second hottest city in the United States, when looking at daily mean. What about when you look at women?

Re:"Second Hottest"

[Anonymous Coward]

Then it's somewhere below your local Curves gym.

avestor... isn't that the outfit that went bust?

[swschrad]

yep

http://www.greencarcongress.com/2006/11/avestor_sh uts_d.html

our telecom outfit had ship issue after ship issue and RMA issue after RMA issue, before we finally got a tech advisory that the vendor was shyt outta luck, and use sealed lead-acid for field replacements.

which will probably improve reliability immensely.

if you have any, seriously consider replacing them with non-incendiary technology.

BUY FLYWHEELS YOU FUCKING INGRATES

[F34nor]

Wake up take your head out of your ass and buy Flywheel batteries. Caterpillar sells them for god sakes. No maintenance, rapid charge and discharge, and so reliable that you can send them into space at 10,000 USD per pound to orbit.

Re:So that's what happens if you run Windows Embed

[Tom DBA]

I was at Sprint PCS a few years ago. I remember hearing about the power failure in Ft. Worth, TX. The batteries drained down in the UPS, so the power generator was started. It caught fire and people were air lifted to hospital. When I'm in the Ft. Worth area I go to visit that data center and just think about the horror Sprint, EMC, Sun and Veritas people suffered one night a few years ago.