Super Bowl Blackout Caused By Defective Protective Relay

New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."

Comments

It was a fail safe

[eksith]

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

Re:It was a fail safe

[Anonymous Coward]

Apparently the circuit breaker failed even when there was no short-circuit event.

I work as an electrical engineer with an electric utility and it seems to me the circuit break perform as intended. The relay told the breaker to trip (open), so it did. After a series of check to make sure no equipment was damaged, electricians were able to close the breaker again.

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Re:It was a fail safe

[Skapare]

It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.

The TL;DR

[Anonymous Coward]

Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).

Re:The TL;DR

[Anonymous Coward]

The protective relay is not at all a part of the circuit breaker. The breaker is a separate device completely, it might not even be in the same cubicle with the protective relay. Also, one protective relay may be commanding several breakers to open on a fault, or it may not actually be commanding a breaker per se, but starting a chain of operations, opening the overloaded breaker, notifying a transfer switch to close tie breakers and go to an alternative power source, etc.

Electrical controls are complex and nuanced, that is why there are professionals to do it. I work in the industrial process control industry, and have programmed my fair share of protective relays, both for switchgear and for motors. (Schweitzer, GE, Square-D/Schneider and ABB specifically.)

Re:The TL;DR

[MorePower]

In my experience, most relays have a "Instantaneous" setting that goes off as fast as possible if you have like 20-30 times as much current as should be there, a "Short Time" setting that goes off in few seconds (a fixed time, exactly how long is settable) if the current is several times times what it should be (exactly how much current is settable) and the "Long Time" setting which follows $Fixed_value = [Current]^2 * time ("I squared T").

The "Long Time" setting integrates current squared when ever the current is above the "Pick-up" value which is typically around 20% over normal rated current. Exactly how much the integrated value has to reach to trip on "Long Time" is very complex and has to be coordinated all the other relays and systems. Generally, the lowest level of breakers are given time to trip first, in hopes that the problem is solved while only interrupting a single circuit. The upstream breakers are set with a higher value so they will trip after the downstream breakers had their chance.

Re:Seems like system failures

[MightyYar]

True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.

TFA

[Anonymous Coward]

You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.

Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.

Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.

The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.