Super Bowl Blackout Caused By Defective Protective Relay

New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."

Comments

It was a fail safe

[eksith]

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

Re:It was a fail safe

[Anonymous Coward]

Apparently the circuit breaker failed even when there was no short-circuit event.

I work as an electrical engineer with an electric utility and it seems to me the circuit break perform as intended. The relay told the breaker to trip (open), so it did. After a series of check to make sure no equipment was damaged, electricians were able to close the breaker again.

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Re:It was a fail safe

[Skapare]

It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.

The TL;DR

[Anonymous Coward]

Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).

Re:The TL;DR

[cblguy2]

Circuit breaker was not "miscalibrated". The protective relay (which is separate from a breaker) possibly had a setting in it that was too low. Protective relay settings are based on time curves (which are plotted on logarithmic paper). For, say, 300 amps, it trips after 10s or 100s of seconds of continuous operation past the setting. For 10,000 amps, it may after .03 seconds (or you may have an instantaneous setting, or a definite time delay based on cycles). That kind of curve. If the load was drawing so much current, for so long of time, then yes, it will send a command for the circuit breaker to trip. Anyhow, it's easy to screw up a protective relay setting - and yes, I've done it. That's why relay settings are always checked by a second engineer as well, just to make sure you didn't miss something. IAAPE (I am a protection engineer, and a P.E.), though we don't use S&C relays (Schweitzer here).

Re:The TL;DR

[Anonymous Coward]

The protective relay is not at all a part of the circuit breaker. The breaker is a separate device completely, it might not even be in the same cubicle with the protective relay. Also, one protective relay may be commanding several breakers to open on a fault, or it may not actually be commanding a breaker per se, but starting a chain of operations, opening the overloaded breaker, notifying a transfer switch to close tie breakers and go to an alternative power source, etc.

Electrical controls are complex and nuanced, that is why there are professionals to do it. I work in the industrial process control industry, and have programmed my fair share of protective relays, both for switchgear and for motors. (Schweitzer, GE, Square-D/Schneider and ABB specifically.)

Re:The TL;DR

[Anonymous Coward]

You are being pedantic to make yourself feel important. Sure they may not be part of the racked out breaker for high voltage breakers (4160, 13.8kv, and above), but in those cases the relays are associated with the breaker and include it in their designation (51-bkr designation, 86-breaker designation, 27-breaker designation, etc.). The control logic for those breakers will usually be in the breaker cubicle and the relays will usually be mounted on the front or with the control logic. For lower voltage breakers you will have a relay cabinet and control power fuses that feed the breakers and certain relays mounted into the breakers (so that even when you rack them out, the relays come along). 99% of breakers that you deal with will be this variety. 0.9% will be the more complex variety above. and 0.1% will be complex enough that control logic is done in different cabinets like you describe (for example, nuclear plant protective logic or for extremely high voltage like 345kv breakers where you want all of the logic controlled in a switchhouse). How does this apply? 99.9% of the 'breakers' will be housed in one integral cabinet or have a relay cabinet in the same bus housing. An operator will call it a 'breaker'. An electrical tech will call it a 'breaker'. An engineer when communicating with anybody else will call it a 'breaker'. Only an engineer when communicating with another engineer would ever be pedantic enough to point out that the relay isn't part of the breaker. For everyone using it, it is. Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?

Re:The TL;DR

[MorePower]

In my experience, most relays have a "Instantaneous" setting that goes off as fast as possible if you have like 20-30 times as much current as should be there, a "Short Time" setting that goes off in few seconds (a fixed time, exactly how long is settable) if the current is several times times what it should be (exactly how much current is settable) and the "Long Time" setting which follows $Fixed_value = [Current]^2 * time ("I squared T").

The "Long Time" setting integrates current squared when ever the current is above the "Pick-up" value which is typically around 20% over normal rated current. Exactly how much the integrated value has to reach to trip on "Long Time" is very complex and has to be coordinated all the other relays and systems. Generally, the lowest level of breakers are given time to trip first, in hopes that the problem is solved while only interrupting a single circuit. The upstream breakers are set with a higher value so they will trip after the downstream breakers had their chance.

Re:The TL;DR

[inasity_rules]

You sir, are a bit of an ass. He is giving relevant and interesting information, which is true. I know, I'm also a process control engineer. The protection relay is quite a complex device (normally approaching the complexity of a small PLC) and very easy to set up wrong.

CYA

[msauve]

Yet, the manufacturer of the trip relay says "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power..." Based on Entergy's incorrect initial claims that "it wasn't us," I tend to think they're not being honest.

Re:Did someone lost his job?

[isorox]

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

Re:Seems like system failures

[MightyYar]

True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.

Re:Did someone lost his job?

[Hamsterdan]

Yup, that's the way it goes in some parallel universe :)

Re:Alphas

[Mitreya]

Reminds me of an episode of the Syfy-channel show Alphas

Yeah, Superbowl reminds me of SyFy shows too.

Re:Did someone lost his job?

[davester666]

No, they are not detached from company performance.

If the company performs well, the bonus becomes astronomical. If it performs less well, the bonus is merely unbelievable.

To regular people, it appears to be detached from reality.

Next Year...

[Anonymous Coward]

The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.

TFA

[Anonymous Coward]

You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.

Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.

Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.

The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.