Super Bowl Blackout Caused By Defective Protective Relay
New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."
That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?
I'm not a complete idiot... Some parts are missing.
Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.
If computers were people, I'd be a misanthrope.
Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).
Yet, the manufacturer of the trip relay says "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power..." Based on Entergy's incorrect initial claims that "it wasn't us," I tend to think they're not being honest.
"National Security is the chief cause of national insecurity." - Celine's First Law
Are frequently caused by the devices installed to prevent them. Quite ironical.
In combat sports where people get hurt for the amusement of spectators, we use Roman numerals.
Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
"Just when you thought the NFL couldn't get any blacker"
"God punishes Beyonce for acting like a stripper."
-- Bill Maher
There was this engineer who was in charge of the production studio that did the live broadcast of the presidential debate between Jimmy Carter and Gerald Ford. I wonder if this stadium maintenance engineer is that guy's son or something...
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
It was not any piece of technology that caused that outage, it was vengeful spirits!
The Uncoveror: It's the real news.
Reminds me of an episode of the Syfy-channel show Alphas
If you have trouble figuring out what is ironic and what isn't: This is ironic.
Yes, apologies, English is not my first language. I only scored 115 on my TOEFL test.
I'm not a complete idiot... Some parts are missing.
Comment removed based on user account deletion
From TFA
The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.
If the device tripped out because the load it saw exceeded its settings, then I'd say that the device was functioning perfectly fine at the Superbowl.
Now were those setting suitable for Beyonce's half time show? That would be the question to ask.
I am Slashdot. Are you Slashdot as well?
I'll start by raising mine. Seriously. It's just funny to me now.
I did not know that. I thought you needed a Superb Owl to do that.
Village idiot in some extremely smart villages.
So it wasn't Beyonce's blow-dryer after all?
Table-ized A.I.
nuf sed
Table-ized A.I.
Thanks. I assume it's for crown green bowls, or something similar?
The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.
As a technical troubleshooter I can not believe how they try to say it was not caused by the grand surge of electricity needed.
Maybe the device saw the usage as abnormal and just did it's job a little bit late.
Just admit it, you didn't think it would draw that much amps/be a problem with stage over-hype.
First thing I did after seeing it was check solar activity. Once I saw that it was nominal and not CLOSE to a major power fluctuation event, I determined it was caused by Human Error. Don't fucking care what the error was - people will fix broken shit. It was just an error.
re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels, so the firing most often will happen to those at the lower level who executed the commands, including putting in crappy materials that were ordered when the higher-ups want to save money. At least that seems to be the way of the USA; Japan's older way would have those responsible all the way up to the chair/CEO stepping up and taking blame and getting out of the way or resigning. Over here, the standard is to blame someone else.
.
"The buck stops here" for Eisenhower; but notice how Obama stayed out of the way when Hillary Clinton tried to initially take the blame for Benghazi, but ultimately tried to tap-dance her way out of all responsibility when it actually came time for the congressional hearings.
You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.
Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.
Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.
The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.
tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.
yeah, some people thought it was suspicious because stopping Baltimore's momentum helped enable that San Francisco rally/near-comeback.
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
BS
Commentators talked about the momentum shift, and then that happened.
Yeah, if the spread was Baltimore by 3 or less, this would affect the betting outcome but not the game outcome, which makes sense as a lower-impact way to 'fix' it. However, most spreads I saw were San Francisco by 4. Maybe the attempt to 'fix' it went wrong, maybe it was about some side bet.
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
*if the spread was Baltimore by 3 or more*
I listen to both RIAA and non-RIAA stuff if I like the music, tangential business/politics nonwithstanding.
Someone should inform Jewel that this is an appropriate example of irony!
You're thinking of Alanis Morissette...
"Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages."
Isn't the point of the protective relay to CAUSE power outages when the load is too high in order to prevent damage to equipment and fires from the line carrying more load than it should.
"The more they overthink the plumbing, the easier it is to stop up the drain."
Mr. Leonard, Its me, jim. you might not know me very well, im not exactly C-level so i never really met you. I just wanted to ask, hows that Relay sub-subcontractor thing working out for you? Me? oh ive been pretty successful since the termination with my own engineering consulting firm. We work on switchgears, relays, you name it!
Hope your team won the superbowl, Jim Ex master relay engineer, Entergy INC Local Union affiliated.
Good people go to bed earlier.
... in general. It is just the same as with security: If you do not design it in from the beginning, it does not really help and ha s a good change of making things worse.
As it looks like the circuit-breaker type device was configured wrongly, that holds even more: The more components that need a setting, the higher the probability somebody messes up or has his/her capability for understanding how the system works exceeded.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
Even EL&P's coverage doesn't say anything substantive. When we start seeing articles with diagrams of the feeders, maybe we'll know something.
An enclosed stadium in Louisiana in winter shouldn't be anywhere near its electrical load peak. No air conditioning load.
Seems TOEFL is for UK and Australia mainly. So what do you care?
Learn to love Alaska
Apparently not, I was able to see and hear it 100%.
You do know that the outage occurred after the second half kicked off, right?
Confucius say, "Find worm in apple - bad. Find half a worm - worse."