Super Bowl Blackout Caused By Defective Protective Relay

New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."

Comments

It was a fail safe

[eksith]

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

Re:It was a fail safe

[Anonymous Coward]

Apparently the circuit breaker failed even when there was no short-circuit event.

I work as an electrical engineer with an electric utility and it seems to me the circuit break perform as intended. The relay told the breaker to trip (open), so it did. After a series of check to make sure no equipment was damaged, electricians were able to close the breaker again.

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Re:It was a fail safe

[Skapare]

It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.

Re:It was a fail safe

[LoRdTAW]

Just to add to this, the term "circuit breaker" refers to a switch that can open (interrupt) and close a circuit under load without arcing or damage. They are typically designed with springs or gas charges to very rapidly open the contacts without arcing or have channels around the contacts called arc chutes that extinguish arcs (older DC breakers had "blow out" coils, when opened the arc would be magnetically pushed out until it extinguished) . "Over current protection" refers to a device that guards against over load and short circuits. It may be part of the circuit breaker or external. Home circuit breakers are a combination of a circuit breaker with over current protection and are always a fixed amperage for the sake of simplicity. Disconnects are switches that are not designed to interrupt a circuit under load but to isolate circuits and circuit breakers.

Industrial breakers can be adjusted to within +-10% of their rating and some have the ability to be remotely opened and closed. Remember the scene in Jurassic Park where Ellie has to turn the power back on? Well that breaker charging scene is what you really do. The handle is pumped to charge the springs inside which open/close the breaker. You pump until "charged" appears in a window and then you push a "close" button to "turn on" the breaker. Those circuit breakers come in a variety of sizes and can handle 4000+ amps at 480 or 600 volts. After hurricane Sandy my company rented a trailer generator which had a pump type breaker. It was rated to 600 Amps and controlled by a PLC. The remote trip is handy if you want to dial in a maximum current which is lower then the breaker maximum. This allows you to match the capacity of the system being powered. Unfortunately the technician set the PLC to trip the breaker around 350 Amps which kept us from using the generators complete capacity as well as all of our machines. They tried to reset it but there was a software glitch which prevented them for turning the current up.

Utility circuit breakers used in switch gear are either oil or sulphur hexafluoride (SF6) filled. They are not designed with over-current protection built in, rather they are commanded by an automation system which has sensors for everything ranging from volts, amps to temperature phase loss and power factor. If you ever happen to get a good look at an outdoor substation, you can easily spot a circuit breaker. It has six insulators for the three phases coming in and three phases coming out and is rather small compared to a transformer. On either side there are disconnects, knife switches who's job it is to allow the circuit breaker to be completely isolated from the system. This allows technicians to service or replace it.

Check out this video: http://www.youtube.com/watch?v=_2LpCdhuOyQ The first segment is a disconnect opened under load. You see it just arcs until a phase-to-phase short occurs which causes an upstream breaker to open. I recall it was a 138kV circuit. The third segment is a 500kV set of disconnects which were opened while still energized. The original story I read was that one of the poles in the SF6 breaker failed to open so they had no choice but to open the disconnect.

The TL;DR

[Anonymous Coward]

Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).

Re:The TL;DR

[cblguy2]

Circuit breaker was not "miscalibrated". The protective relay (which is separate from a breaker) possibly had a setting in it that was too low. Protective relay settings are based on time curves (which are plotted on logarithmic paper). For, say, 300 amps, it trips after 10s or 100s of seconds of continuous operation past the setting. For 10,000 amps, it may after .03 seconds (or you may have an instantaneous setting, or a definite time delay based on cycles). That kind of curve. If the load was drawing so much current, for so long of time, then yes, it will send a command for the circuit breaker to trip. Anyhow, it's easy to screw up a protective relay setting - and yes, I've done it. That's why relay settings are always checked by a second engineer as well, just to make sure you didn't miss something. IAAPE (I am a protection engineer, and a P.E.), though we don't use S&C relays (Schweitzer here).

Re:The TL;DR

[Anonymous Coward]

The protective relay is not at all a part of the circuit breaker. The breaker is a separate device completely, it might not even be in the same cubicle with the protective relay. Also, one protective relay may be commanding several breakers to open on a fault, or it may not actually be commanding a breaker per se, but starting a chain of operations, opening the overloaded breaker, notifying a transfer switch to close tie breakers and go to an alternative power source, etc.

Electrical controls are complex and nuanced, that is why there are professionals to do it. I work in the industrial process control industry, and have programmed my fair share of protective relays, both for switchgear and for motors. (Schweitzer, GE, Square-D/Schneider and ABB specifically.)

Re:The TL;DR

[Anonymous Coward]

You are being pedantic to make yourself feel important. Sure they may not be part of the racked out breaker for high voltage breakers (4160, 13.8kv, and above), but in those cases the relays are associated with the breaker and include it in their designation (51-bkr designation, 86-breaker designation, 27-breaker designation, etc.). The control logic for those breakers will usually be in the breaker cubicle and the relays will usually be mounted on the front or with the control logic. For lower voltage breakers you will have a relay cabinet and control power fuses that feed the breakers and certain relays mounted into the breakers (so that even when you rack them out, the relays come along). 99% of breakers that you deal with will be this variety. 0.9% will be the more complex variety above. and 0.1% will be complex enough that control logic is done in different cabinets like you describe (for example, nuclear plant protective logic or for extremely high voltage like 345kv breakers where you want all of the logic controlled in a switchhouse). How does this apply? 99.9% of the 'breakers' will be housed in one integral cabinet or have a relay cabinet in the same bus housing. An operator will call it a 'breaker'. An electrical tech will call it a 'breaker'. An engineer when communicating with anybody else will call it a 'breaker'. Only an engineer when communicating with another engineer would ever be pedantic enough to point out that the relay isn't part of the breaker. For everyone using it, it is. Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?

Re:The TL;DR

[MorePower]

In my experience, most relays have a "Instantaneous" setting that goes off as fast as possible if you have like 20-30 times as much current as should be there, a "Short Time" setting that goes off in few seconds (a fixed time, exactly how long is settable) if the current is several times times what it should be (exactly how much current is settable) and the "Long Time" setting which follows $Fixed_value = [Current]^2 * time ("I squared T").

The "Long Time" setting integrates current squared when ever the current is above the "Pick-up" value which is typically around 20% over normal rated current. Exactly how much the integrated value has to reach to trip on "Long Time" is very complex and has to be coordinated all the other relays and systems. Generally, the lowest level of breakers are given time to trip first, in hopes that the problem is solved while only interrupting a single circuit. The upstream breakers are set with a higher value so they will trip after the downstream breakers had their chance.

Re:The TL;DR

[sco08y]

You are being pedantic to make yourself feel important. ... Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?

Talk about projection.

Re:The TL;DR

[inasity_rules]

You sir, are a bit of an ass. He is giving relevant and interesting information, which is true. I know, I'm also a process control engineer. The protection relay is quite a complex device (normally approaching the complexity of a small PLC) and very easy to set up wrong.

Re:The TL;DR

[adolf]

But calling it a "protection relay" instead of a "circuit breaker" really is missing the point with pedantry.

As I understand it, such a protection relay is one component tells the switchgear and/or other components what to do, based on loading and other parameters.

The entire system, viewed as a black box, is a circuit breaker. And in practice, the entire system behaves in a manner not dissimilar to the circuit breakers in my own house: Detect fault (either current fault or ground fault or arc fault or some manner of goddamn fault) and direct stuff to turn off.

Just like any other circuit breaker in common parlance.

On any scale, there are many parts to the systems that we call circuit breakers. Any failure of any one of them to behave as expected can still accurately described as a failure of the circuit breaker as a whole.

Anything else is pedantic. A car analogy:

"My car is broken. It left me stranded on my way to the Super Bowl."

    "What's wrong with it?"

"The fuel pump relay."

    "Oh, well the car isn't broken then. It's just a fuel pump relay."

"But the car doesn't work."

    "Sure it does. Only the fuel pump relay failed. The rest of the car is fine, isn't it?"

"Stupid fucking pedant."

    "You're a bit of an ass for saying that."

"And you don't seem to understand that no matter what, the whole car doesn't work without that part."

Re:The TL;DR

[inasity_rules]

You can have a breaker without a protection relay, but not a car without a fuel pump. A fuel pump is often mechanically driven, and a protection relay on it would be silly. Even the electric ones use a simple fuse. A car with a blown fuse isn't necessarily broken (and normally a spare is carried anyway), and can be up and running again in minutes. A breaker will work fine without a protection relay. It simply won't trip on fault conditions which is considered dangerous. I have seen a couple like that. You clearly not entirely sure what you're talking about, so his "pedantry" was necessary to educate you. QED. :P

CYA

[msauve]

Yet, the manufacturer of the trip relay says "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power..." Based on Entergy's incorrect initial claims that "it wasn't us," I tend to think they're not being honest.

Re:Did someone lost his job?

[isorox]

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

Seems like system failures

[matty619]

Are frequently caused by the devices installed to prevent them. Quite ironical.

Re:Seems like system failures

[the eric conspiracy]

That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.

I think the turning point was when journaling file systems came to Linux.

Re:Seems like system failures

[MightyYar]

True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.

Re:Seems like system failures

[dbIII]

I've had plenty with APC and plenty more with a cheap and nasty Chinese brand that has probably given up in shame.
Have enough of them and the high chance of failure stacks up so you start seeing a few. Have them for long enough and the batteries will die and won't always die gracefully. A frequent battery replacement schedule instead of waiting for a warning could get you around the last one. The largest problem I had was when there were minor power fluctuations that would make lights flicker and make the UPS's panic and shutdown instantly - thus leaving everything up that was not on a UPS while the "protected" stuff suddenly lost power. While that was due to unusual loading on the power network (I suspect it was a nearly crane regenerating as it rapidly lowered a load) it happened quite a couple of dozen times one year and we couldn't find a way to filter it out so eventually nearly everything was migrated off UPS. UPS, when it works, gives you time to write to a filesystem but that's no longer as important with things like batteries in controller cards and far more stable filesystems. Expecting it to tide you over a power outage is far too optimistic - that's what a generator with a big fuel tank is for.

Re:Seems like system failures

[greg1104]

It is possible to buy a UPS for around $40, and all of these items are crap. CyberPower, DirectUPS, Tripp-Lite, Opti-UPS, Minuteman, Powercom...any of these can end up becoming the least reliable component to a Linux system.

Even with APC, who sells reliable units if you spend enough, you're looking at a few hundred dollars for one that is usefully reliable. And even there you have to be careful, do your research, and test to be sure. The biggest issue right now is how power is generated, which determines whether they will support active PFC power supplies. The models from APC that say "stepped approximation to a sinewave" for example are nothing but trouble. You want "pure sine wave". The Smart-UPS PCW-SUA1000 has the right design at $370. It is depressing how bad their cheaper models are nowadays, and how much you can spend on a unit that's still junk.

Re:Did someone lost his job?

[Hamsterdan]

Yup, that's the way it goes in some parallel universe :)

Re:Alphas

[Mitreya]

Reminds me of an episode of the Syfy-channel show Alphas

Yeah, Superbowl reminds me of SyFy shows too.

Re:Did someone lost his job?

[davester666]

No, they are not detached from company performance.

If the company performs well, the bonus becomes astronomical. If it performs less well, the bonus is merely unbelievable.

To regular people, it appears to be detached from reality.

Re: Explanation

[boundary]

Thanks. I assume it's for crown green bowls, or something similar?

Next Year...

[Anonymous Coward]

The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.

TFA

[Anonymous Coward]

You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.

Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.

Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.

The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.