Super Bowl Blackout Caused By Defective Protective Relay

New submitter wilby writes "Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages. A device designed to improve the Superdome electrical system reliability instead caused it to shut down dramatically during Super Bowl 47. [The company] said testing traced the source of the problem to an 'electrical relay device' it had installed in December to protect Superdome equipment in case a cable failure occurred between the company's switchgear and the stadium."

Comments

Did someone lost his job?

[sabri]

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Re:Did someone lost his job?

[isorox]

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

Re:Did someone lost his job?

[Hamsterdan]

Yup, that's the way it goes in some parallel universe :)

Re:Did someone lost his job?

[Mitreya]

That's the first thing that came to my mind when I saw this happen: someone is going to get fired over this... So, who got fired?

Presumably the person that receives the big end-of-year bonus when everything goes well?

They probably get a higher end-of-year bonus for resolving the problem.
Upper level compensation works in mysterious ways - plenty of CEOs manage to get guaranteed bonuses that are completely detached from company performance.

Re:Did someone lost his job?

[davester666]

No, they are not detached from company performance.

If the company performs well, the bonus becomes astronomical. If it performs less well, the bonus is merely unbelievable.

To regular people, it appears to be detached from reality.

Re:Did someone lost his job?

[Anonymous Coward]

Somebody got fired? Did they show another nipple at the half time show instead of violence?

Re:Did someone lost his job?

[Runaway1956]

We watch companies fail, then get bailed out by the government, after which CEO's still get astronomical bonuses.

Yes, the bonus system is detached from reality. Any CEO whose company fails as a result of his actions, inactions, and poor judgement calls should be sacked, then sued for dereliction of duty. Said CEO should never again have access to resources which he hasn't earned, himself. Let him push a frigging broom and mop for the rest of his life.

Those astronomical bonuses to which you refer have no place in a sane world.

Re:Did someone lost his job?

[Anonymous Coward]

And yet my "longevity" bonus is so tied to my performance I can never get it. They said everyone will make the same, but the people who have been employed longer will get an ever increasing bonus tied to how long you have been with the company. Well than they said that you have to do extra to earn 100%, just doing your job gets you 75%. Than they started taking percentages for anything bad done. I guess it is nice to make 6 figures and complain those who make 1/3rd or more less aren't happy with what you give them. Oh yeah, 6 figure person gets there bonus in full every time.

Re:Did someone lost his job?

[Hognoxious]

I suggest you get someone to read davester666's post to you, and then explain it. Someone from England would be a good choice.

Re:Did someone lost his job?

[Runaway1956]

I suggest you read Davestar's post, as well as my own again. I've disagreed with Davestar. He claims that the failures who ran banks into the dirt didn't get astronomical bonuses, merely unbelievable bonuses. I stated that they did indeed get astronomical bonuses.

I have little idea which language might be your own native language, but respectful disagreement seems pretty obvious to me here. Davestar basically split a hair, and I told him that execs always get the bigger half of the hair.

Re:Did someone lost his job?

[Anonymous Coward]

Yes, Minister.

It was a fail safe

[eksith]

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

Re:It was a fail safe

[Anonymous Coward]

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

So the system blew a fuse / tripped a circuit breaker? Why is this worth reporting on?

Re:It was a fail safe

[jones_supa]

Apparently the circuit breaker failed even when there was no short-circuit event.

Re:It was a fail safe

[Dr. Tom]

It was too many people tweeting at the same time.

Re:It was a fail safe

[Anonymous Coward]

Apparently the circuit breaker failed even when there was no short-circuit event.

I work as an electrical engineer with an electric utility and it seems to me the circuit break perform as intended. The relay told the breaker to trip (open), so it did. After a series of check to make sure no equipment was damaged, electricians were able to close the breaker again.

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Re:It was a fail safe

[Seumas]

My only thought on this whole event was... "OH MY GOD! A sporting event went without lights for a whole half of an hour! WHAT A TRAGEDY?! THERE MUST BE INVESTIGATIONS! Someone MUST be held accountable! WE WILL NEVER FORGET!" *yawn*

Re:It was a fail safe

[Anonymous Coward]

Basically to power down the system before catastrophic failure will cause wires to melt, cause fires and other bad things. So essentially, it did its job. They just needs to dial down the sensitivity.

So the system blew a fuse / tripped a circuit breaker? Why is this worth reporting on?

Because, for some strange reason, we here in America feel that we have to always be 100% in control of and never make mistakes on anything. Everything must be designed to the level of a dream world of perfection. When someone makes a mistake (or something that someone made fails), we must make a huge deal out of it and act as if the world will end if it ever happens again.

No, I am not proud of my country on quite a few methods. This would be one of them.

Re:It was a fail safe

[Skapare]

It apparently did it's job. But apparently it was given the wrong job. It is accused (by the manufacturer, of course) that someone entered the wrong amperage that it should do its job at. Unlike home circuit breakers which come in specific amperage levels (and vary from unit to unit by plus or minus 10 percent which is considered acceptable), these relay devices, which are a component in an overcurrent protection system, cannot be made at fixed amperage levels due to economics. They are quite expensive to replace with another just to tweak the settings due to changes made elsewhere in the power distribution network, and the number of different amperage values needed would be very large. They can be expensive also because either they directly connect to current transformers that have high open circuit voltage potential, or operate from digital sensors on the current transformers. They are also expected to have accurate at better than one percent.

Re:It was a fail safe

[Anonymous Coward]

Surprise. The manufacturer is defending itself. They say the trip point was set too low and the equipment performed as designed.

Posting anon because I RTFA and I don't want to ruin my rep. They still ban posters for that don't they?

Re:It was a fail safe

[JWSmythe]

    My only thought was, "The power went out during the Superbowl? I'm surprised I haven't heard more bitching".

    That was about it. Although there are apparently millions of Americans who care, there are also millions of Americans would don't.

Re:It was a fail safe

[JWSmythe]

    Wasn't there a practice run done? Really, if you're responsible for a multi-billion dollar event, it might be a good idea to fire everything up a few times and make sure it behaves as intended.

Re:It was a fail safe

[eksith]

It probably doesn't help that electricity, especially at these levels, is a tricky thing to begin with. Even in the article, they mention they're not entirely sure why the relays fail.

Re:It was a fail safe

[AK Marc]

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Tested? Who tests anything?

Re:It was a fail safe

[DKlineburg]

The end user. I think that is how they do game production at least.

Re:It was a fail safe

[Seumas]

Even those who are die-hard fans of football . . . I don't see why "oh no, the facility lost electricity for a half hour" is such a compelling thing that it has to be talked about for a week or two and investigations have to be undertaken. What does it have to do with the game any more than the amount or expense of parking at the facilities the game is played at do?!

It's just this dumb, ridiculous, meaningless, trivial, idiotic thing that the whole media and all of society has somehow latched onto as being even remotely relevant or interesting at all.

Re:It was a fail safe

[Anonymous Coward]

I'm just surprised that the stadium was designed with an entire half of the lighting on one circuit like that. (That's where the real design flaw is.) For such a large venue, you'd think they'd break it down into smaller sections like eighths or sixteenths. At least then a circuit's failure mode wouldn't cause such a big disruption in service.

Re:It was a fail safe

[denobug]

The relay is the device in question, and they haven't released enough information for anybody outside to know what happened. It looks to me that a relay was installed and it either was setup up at the wrong trip point, or it wasn't tested properly.

Tested? Who tests anything?

In my line of work. If you don't properly commission your stuff, it has high consequences. Both in casualty and in economic consequences.

As a result, if someone works for me and not properly test/commission the equipment, I will make sure they don't work for me ever again, period. If they fuss then I will see to it that the rest of the company knows how incompetent you are and why your company/affiliate tolerate such unprofessional attitude and conduct. It's a fairly small circle in my line of work that eventually no one will hire them unless their attitudes change.

Re:It was a fail safe

[GNious]

I was expecting millions of non-Americans to not care ... seems I was wrong, judging from news-sites etc.

Re:It was a fail safe

[LoRdTAW]

Just to add to this, the term "circuit breaker" refers to a switch that can open (interrupt) and close a circuit under load without arcing or damage. They are typically designed with springs or gas charges to very rapidly open the contacts without arcing or have channels around the contacts called arc chutes that extinguish arcs (older DC breakers had "blow out" coils, when opened the arc would be magnetically pushed out until it extinguished) . "Over current protection" refers to a device that guards against over load and short circuits. It may be part of the circuit breaker or external. Home circuit breakers are a combination of a circuit breaker with over current protection and are always a fixed amperage for the sake of simplicity. Disconnects are switches that are not designed to interrupt a circuit under load but to isolate circuits and circuit breakers.

Industrial breakers can be adjusted to within +-10% of their rating and some have the ability to be remotely opened and closed. Remember the scene in Jurassic Park where Ellie has to turn the power back on? Well that breaker charging scene is what you really do. The handle is pumped to charge the springs inside which open/close the breaker. You pump until "charged" appears in a window and then you push a "close" button to "turn on" the breaker. Those circuit breakers come in a variety of sizes and can handle 4000+ amps at 480 or 600 volts. After hurricane Sandy my company rented a trailer generator which had a pump type breaker. It was rated to 600 Amps and controlled by a PLC. The remote trip is handy if you want to dial in a maximum current which is lower then the breaker maximum. This allows you to match the capacity of the system being powered. Unfortunately the technician set the PLC to trip the breaker around 350 Amps which kept us from using the generators complete capacity as well as all of our machines. They tried to reset it but there was a software glitch which prevented them for turning the current up.

Utility circuit breakers used in switch gear are either oil or sulphur hexafluoride (SF6) filled. They are not designed with over-current protection built in, rather they are commanded by an automation system which has sensors for everything ranging from volts, amps to temperature phase loss and power factor. If you ever happen to get a good look at an outdoor substation, you can easily spot a circuit breaker. It has six insulators for the three phases coming in and three phases coming out and is rather small compared to a transformer. On either side there are disconnects, knife switches who's job it is to allow the circuit breaker to be completely isolated from the system. This allows technicians to service or replace it.

Check out this video: http://www.youtube.com/watch?v=_2LpCdhuOyQ The first segment is a disconnect opened under load. You see it just arcs until a phase-to-phase short occurs which causes an upstream breaker to open. I recall it was a 138kV circuit. The third segment is a 500kV set of disconnects which were opened while still energized. The original story I read was that one of the poles in the SF6 breaker failed to open so they had no choice but to open the disconnect.

The TL;DR

[Anonymous Coward]

Overcurrent tripped a miscalibrated circuit breaker (trip setting was too low).

Re:The TL;DR

[Dr. Tom]

too many tweets

Re:The TL;DR

[Anonymous Coward]

too many dupes

Re:The TL;DR

[cblguy2]

Circuit breaker was not "miscalibrated". The protective relay (which is separate from a breaker) possibly had a setting in it that was too low. Protective relay settings are based on time curves (which are plotted on logarithmic paper). For, say, 300 amps, it trips after 10s or 100s of seconds of continuous operation past the setting. For 10,000 amps, it may after .03 seconds (or you may have an instantaneous setting, or a definite time delay based on cycles). That kind of curve. If the load was drawing so much current, for so long of time, then yes, it will send a command for the circuit breaker to trip. Anyhow, it's easy to screw up a protective relay setting - and yes, I've done it. That's why relay settings are always checked by a second engineer as well, just to make sure you didn't miss something. IAAPE (I am a protection engineer, and a P.E.), though we don't use S&C relays (Schweitzer here).

Re:The TL;DR

[Anonymous Coward]

Yes and the protective relay is part of the circuit breaker. Don't get too pedantic on this. When you swap breakers, you recalibrate the relays to the proper trip setpoints. Engineers determine the settings and electrical or I&C techs put it in.

Re:The TL;DR

[Anonymous Coward]

"To the rest of the world nothing of value was lost". Your precise is now complete.

Re:The TL;DR

[girlinatrainingbra]

Q for you: is the circuit breaker actually set for $Fixed_value = Current \times Time$ or for a thermal setting which just happens to almost be equivalent to (current draw)x(time drawn) ? Of course I do realize that since the current draw won't be continuously constant, my question simplified what should be the integral of current draw over a running window of time being over a threshold into just the product of time and current draw, but you get the idea...
.
I know that a fuse is thermally activated with the current draw through a resistive element which melts when the current flow is high enough to raise the temperature of the resistive element enough.

Re:The TL;DR

[Anonymous Coward]

The protective relay is not at all a part of the circuit breaker. The breaker is a separate device completely, it might not even be in the same cubicle with the protective relay. Also, one protective relay may be commanding several breakers to open on a fault, or it may not actually be commanding a breaker per se, but starting a chain of operations, opening the overloaded breaker, notifying a transfer switch to close tie breakers and go to an alternative power source, etc.

Electrical controls are complex and nuanced, that is why there are professionals to do it. I work in the industrial process control industry, and have programmed my fair share of protective relays, both for switchgear and for motors. (Schweitzer, GE, Square-D/Schneider and ABB specifically.)

Re:The TL;DR

[Anonymous Coward]

It might be both or neither. There is more than just high current that can trip a protective relay, it could be a phase imbalance caused by a voltage sag in one phase it is monitoring, or a sudden increase in ground current, or a high temperature in the MCC, or any number of other things.

Protective relays have dozens to hundreds of settings, depending on the type and model. It is not at all as simple as TFA makes it out to be, nor are they are simple as a fuse.

Also, in power distribution, even fuses are not that simple. Generally a fuse in an important application is "dual element." One element allows more than the rated current to flow through without tripping, but builds up thermal energy quickly so that an overcurrent condition won't last for long. Most devices draw a sudden burst of current when they are started, this is called "inrush." For motors this can be 6 (or more) times the current it would draw once it is running at full load, for loads with a high capacitive component, it can be thousands of times the current the device will use in operation. The second element is usually sized so that full load will not trip it, but a sudden, HUGE overcurrent will cook it instantly. This protects against dead shorts, etc.

Re:The TL;DR

[Anonymous Coward]

You are being pedantic to make yourself feel important. Sure they may not be part of the racked out breaker for high voltage breakers (4160, 13.8kv, and above), but in those cases the relays are associated with the breaker and include it in their designation (51-bkr designation, 86-breaker designation, 27-breaker designation, etc.). The control logic for those breakers will usually be in the breaker cubicle and the relays will usually be mounted on the front or with the control logic. For lower voltage breakers you will have a relay cabinet and control power fuses that feed the breakers and certain relays mounted into the breakers (so that even when you rack them out, the relays come along). 99% of breakers that you deal with will be this variety. 0.9% will be the more complex variety above. and 0.1% will be complex enough that control logic is done in different cabinets like you describe (for example, nuclear plant protective logic or for extremely high voltage like 345kv breakers where you want all of the logic controlled in a switchhouse). How does this apply? 99.9% of the 'breakers' will be housed in one integral cabinet or have a relay cabinet in the same bus housing. An operator will call it a 'breaker'. An electrical tech will call it a 'breaker'. An engineer when communicating with anybody else will call it a 'breaker'. Only an engineer when communicating with another engineer would ever be pedantic enough to point out that the relay isn't part of the breaker. For everyone using it, it is. Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?

Re:The TL;DR

[MorePower]

In my experience, most relays have a "Instantaneous" setting that goes off as fast as possible if you have like 20-30 times as much current as should be there, a "Short Time" setting that goes off in few seconds (a fixed time, exactly how long is settable) if the current is several times times what it should be (exactly how much current is settable) and the "Long Time" setting which follows $Fixed_value = [Current]^2 * time ("I squared T").

The "Long Time" setting integrates current squared when ever the current is above the "Pick-up" value which is typically around 20% over normal rated current. Exactly how much the integrated value has to reach to trip on "Long Time" is very complex and has to be coordinated all the other relays and systems. Generally, the lowest level of breakers are given time to trip first, in hopes that the problem is solved while only interrupting a single circuit. The upstream breakers are set with a higher value so they will trip after the downstream breakers had their chance.

Re:The TL;DR

[sco08y]

You are being pedantic to make yourself feel important. ... Why an engineer talking to the general public on Slashdot feels the need to point out the difference is unknown. Perhaps this engineer feels under appreciated?

Talk about projection.

Re:The TL;DR

[inasity_rules]

You sir, are a bit of an ass. He is giving relevant and interesting information, which is true. I know, I'm also a process control engineer. The protection relay is quite a complex device (normally approaching the complexity of a small PLC) and very easy to set up wrong.

Re:The TL;DR

[Anonymous Coward]

No pedantic-ness or self importance, it is just how it is. We (operators, techs, engineers and tea ladies) never call the relay the breaker, or part of the breaker. Also, there are plenty of relays which are typically well separated from the breaker, line diff, tranny diff, distance and bus protection relays come to mind - all used at distribution level voltages. Also, not sure about other companies, but why would you need to recalibrate the relay if you swap the breaker? Unless the CTs are part of the breaker assembly and have a different ratio this is not necessary. I suspect you don't work in this industry.

Re:The TL;DR

[adolf]

But calling it a "protection relay" instead of a "circuit breaker" really is missing the point with pedantry.

As I understand it, such a protection relay is one component tells the switchgear and/or other components what to do, based on loading and other parameters.

The entire system, viewed as a black box, is a circuit breaker. And in practice, the entire system behaves in a manner not dissimilar to the circuit breakers in my own house: Detect fault (either current fault or ground fault or arc fault or some manner of goddamn fault) and direct stuff to turn off.

Just like any other circuit breaker in common parlance.

On any scale, there are many parts to the systems that we call circuit breakers. Any failure of any one of them to behave as expected can still accurately described as a failure of the circuit breaker as a whole.

Anything else is pedantic. A car analogy:

"My car is broken. It left me stranded on my way to the Super Bowl."

    "What's wrong with it?"

"The fuel pump relay."

    "Oh, well the car isn't broken then. It's just a fuel pump relay."

"But the car doesn't work."

    "Sure it does. Only the fuel pump relay failed. The rest of the car is fine, isn't it?"

"Stupid fucking pedant."

    "You're a bit of an ass for saying that."

"And you don't seem to understand that no matter what, the whole car doesn't work without that part."

Re:The TL;DR

[inasity_rules]

You can have a breaker without a protection relay, but not a car without a fuel pump. A fuel pump is often mechanically driven, and a protection relay on it would be silly. Even the electric ones use a simple fuse. A car with a blown fuse isn't necessarily broken (and normally a spare is carried anyway), and can be up and running again in minutes. A breaker will work fine without a protection relay. It simply won't trip on fault conditions which is considered dangerous. I have seen a couple like that. You clearly not entirely sure what you're talking about, so his "pedantry" was necessary to educate you. QED. :P

Re:The TL;DR

[inasity_rules]

I have never seen CTs as part of the breaker assembly. They'll be in the panel, or in the case of some large motors by the motor as well. Admittedly I haven't worked with all the breakers that you can get, but this seems normal so far... (Also for the mods, mod parent up. Nobody worth their salt thinks a protection relay is part of a breaker. It simply is not.)

Re:The TL;DR

[adolf]

You can have a breaker without a protection relay,

Of course you can: A simple circuit breaker is just a heater that is mechanically coupled to a switch that locks in the off ("tripped") state: Just like a fuse, but resettable -- no fancy-pants "protection relay" required.

But: If a protection relay exists in this context, it is part of the circuit breaker system. Thus the generalized term "circuit breaker" certainly applies.

A fuel pump is often mechanically driven

"Often?" None of the modern cars I've owned have had a mechanically-driven fuel pump: It is always in the fuel tank (with the fuel), with wires going to it, connected to some manner of switch or relay. (I've owned antique cars with mechanical fuel pumps, but they don't count for any meaningful quantity of "often.")

A breaker will work fine without a protection relay. It simply won't trip on fault conditions which is considered dangerous.

Thank you, Captain Obvious.

You clearly not entirely sure what you're talking about,

Funny, because other than the definition of "often" as it relates to automobile fuel pumps, you seem to agree with me.

QED. :P

Indeed.

Re:The TL;DR

[inasity_rules]

But: If a protection relay exists in this context, it is part of the circuit breaker system. Thus the generalized term "circuit breaker" certainly applies.

Incorrect. In industry they are definitely separate items. The protection relay is in the panel. The breaker is a removable item that fits into the panel, and normally gets wheeled around a trolly. You clearly have never worked with this stuff. perhaps you should before you talk about it? Why do you insist that the breaker is the panel? I really don't understand how you are confusing the two?

let me help you out here:

This is what a breaker looks like. Notice the distinct lack of a protection relay.

Re:The TL;DR

[Anonymous Coward]

I suspect you only work on the distribution side which has colored your views to how breakers are used in generation and by end users. Your talk about CTs confirms this. Engineers are needed to make sure expensive switchyard breakers work properly. They aren't needed to check the operation of bucket breakers in distribution panels. Thus, an engineer who talks about breakers and doesn't work for a manufacturer will assume every breaker is the type he knows. If the engineer gets down to the 480V level at all, he is probably working on complicated protection schemes on distribution busses, not the typical load breakers.

Re:The TL;DR

[Anonymous Coward]

And Schweitzer is the S in S&C, no? (But they are now separate companies with Schweitzer having specialized in relays and S&C in fuses and cutouts.)

FWIW my father worked for S&C and I've done some work for them as well. I did read the article. It has not been determined (to the extent that I can trust the article) that the relay was faulty or had faulty settings. Of course everyone involved is pointing their finger at someone else.

Re:The TL;DR

[Anonymous Coward]

Most likely the failure to properly set the device can be traced to use of a "low bid contractor". Assuming Entergy Corp. is not itself a "low bid contractor".
I'll bet the crew that installed the relay just slapped it into the circuit and did not RTFM.

Re:The TL;DR

[OffTheWallSoccer]

Well put, sir.

Re:The TL;DR

[mirix]

"Often?" None of the modern cars I've owned have had a mechanically-driven fuel pump: It is always in the fuel tank (with the fuel), with wires going to it, connected to some manner of switch or relay. (I've owned antique cars with mechanical fuel pumps, but they don't count for any meaningful quantity of "often.")

Diesel. Even modern ones are mechanically driven... Which is over half of vehicles in some places, which I'd call 'often'.

Re:The TL;DR

[adolf]

Which is over half of vehicles in some places, which I'd call 'often'.

"Some places" isn't the US. TFA is about the US.

And being in the US, I do see a few diesel cars around every now and then. I do not see them "often."

Re:The TL;DR

[adolf]

My whole argument is against unnecessary pedantry.

You're supporting it very well.

Re:The TL;DR

[inasity_rules]

It is not unnecessary. They are different things. It is like you saying "my garage doesn't work" "Why?" "My car is not in it." I give up and hope to goodness you don't do anything technical, especially not anything that someone's life might depend on.

Re:The TL;DR

[inasity_rules]

Pot? This is kettle.

Re:The TL;DR

[adolf]

Don't worry. My day job is about as non-technical and unimportant as they come -- mostly all I do is wireless communication systems for emergency dispatch centers.

So when you call 911 and an ambulance never shows up, you can refer back to this thread and pat yourself on the back for having foretold the eventuality.

Meanwhile, I maintain that arguing about the exact definitions of a "circuit breaker" or "protection relay" is unnecessary pedantry because any thing or collection of things capable of autonomously breaking a circuit in response to abnormal conditions can be (rightly!) termed as a "circuit breaker."

This thing, or collection of things, might be in a panel. It might be on a trolley or cart. It might be a collection of gear filling entire room, or with its own isolated building, or be contained within two or more buildings. It could stretch across continents. I don't really care: The collection of stuff acts as a circuit breaker, and therefore it is a circuit breaker.

Nobody outside of EE fields cares about what parts this circuit breaker might consist of, or that there might be a portable apparatus with an overlapping name, or that the term is too generalized to satisfy every pedant in the world.

Re:The TL;DR

[inasity_rules]

To say nobody outside of the EE feild cares might not be true.. Go order one and see how far you get. The collection of stuff does not act as a circuit breaker. The collection of stuff acts as switchgear, or a MCC or heck, even a substation. The name is not "too generalized" it is simply and plainly wrong. The woman who cleans the substation at the one place I work knows the difference and she barely has any qualifications. I am trying to think of a way to put this so you will understand, but it seems you have a block on this. There are lots of panels around the same size and shape as switchgear, but which aren't switchgear. They don't have breakers in them. They contain other control equipment. The collection of breaker and panel could be correctly reffered to as "switchgear" but not a "breaker." This is not pedantry, it simply is what it is, and trying to overgeneralise here will not work, because (and going back to TFA) what happened will make no sense. I can garruntee you, as someone who works in the industry, that your understanding of this is very very wrong. And I have met enough people out of the industry with a far better understanding of this than you have.

Re:The TL;DR

[adolf]

It is pedantry. And I don't care who you've met, or what the facilities look like (didn't I just say that?).

You're still wrapped up in pedantic details. Stop doing that and zoom out a bit.

Power comes into a collection of gear from the grid (whether that be a local substation or a very long length of wire: I DON'T CARE!). Power comes out, powers facility (REALLY, it's that simple).

In between those two circuits is a circuit breaker, that upon detecting an abnormal condition, breaks the circuit.

Which is what we saw during the superbowl: Stuff turned off because the circuit was broken because the circuit breaker detected something awry and did what it was designed to do.

Seriously. These aren't very big words.

Re:The TL;DR

[inasity_rules]

Firstly, if you don't care, then what the hell are you doing on slashdot? Why are you even commenting here? Did you miss the title of the page? Secondly I think your understanding is dangerously flawed, and the attidude of "I don't care" is quite frankly pathetic, not to mention dangerous. A circuit breaker never detects *anything*. Ever. It can't. It is just a set of mechanical contacts. I know you don't care but swallow your pride and learn something. Then maybe if you happen to ever find yourself in a substation you won't look like a complete tool. Take your crusade against calling things what they are elsewhere. This is certainly not the place for it. Now I'm going to go drive my car. Or was it an aeroplane? They're all the same thing anyway, modes of transport. Heck they both have wheels and engines. Lets not get pedantic about these things...

Re:The TL;DR

[adolf]

Oh, and for pedantry: I can also say that communication failed because of a PIC card error in the CPP that caused the CIP tray to lock, but nobody cares. It might be the truth, but it's pedantic: If folks try to understand, their eyes glaze over. And almost everyone else just skips that part.

It is much simpler to generalize and say "the radio system failed." Even if is so ambiguous as to alarm every autistic nit on this side of the Mississippi who understands these terms, this would still a correct (though generalized) statement.

Re:The TL;DR

[adolf]

A circuit breaker never detects *anything*. Ever. It can't.

Mine do. They're in the basement, in the panel. Not on a trolley, or on a cart, or in their own building. But they do. Some of them detect overcurrent, some of them also detect ground faults, and some of them additionally detect arc faults. It's not an abnormal panel.

I know you don't care but swallow your pride and learn something.

But you're not trying to teach. You're trying to prove that my terms are wrong. And they're not incorrect terms to begin with.

Which, really, is the whole point that I've been trying to make. Or did you miss that part?

Re:The TL;DR

[inasity_rules]

This isn't the same at all. My goodness you're strange. "The protection tripped the breaker." That makes sense to anybody and everybody. "Protection against what?" "Short circuit." "Oh". End of story. This isn't about being technical or pedantic. It's about effective communication. It communicates everything effectively, and clearly. We're not even using technical terms here. If a layman asks "Why?" We can say, "it was set up wrong." The article is about the reason the breaker tripped. This being slashdot, we're interested in that reason. It is why we come here. Tell me honestly, am I being trolled, because I have a hard time believing you're genuine.

Re:The TL;DR

[inasity_rules]

But in this case, your terms are wrong. Good grief. I give up.

Re:The TL;DR

[adolf]

Yes I am strange, and your UID is strange: It has too many digits. So what?

The "protection relay" is all that has been discussed in any of TFAs that I've seen, without any mention of "breaker". And then someone generalizes and says that it's a "circuit breaker," before some EEs come along and say that the "protection relay" is not a circuit breaker, when the fact is that it does just that: It does not have to be in series with the load in order to control that load*.

And then I show up in the middle of all of it and try to simplify terms. And then you tell me how wrong I am for using simple and established English terminology, even though in the broadest of terms such a system would obviously be considered a "circuit breaker."

*Simple, bone-headed electronics: Need to turn things on and off? Use a transistor. Need to turn bigger things on and off? Use that transistor to drive a relay. Need to turn even bigger things on and off? Use a contactor* driven by that relay. Bigger things? Switchgear. Need a fancy way to turn big things off when something goes wrong? Use a "protection relay" in place of or in addition to your usual relay.

Most folks would consider that a pedantic way to describe a simple system, even if they don't even know the word "pedantic" but merely understand the concept, because such details aren't really all that important to the general populace. And anyone who understands English could properly describe the system as "an electronic switch."

**: The delineation between "relay" and "contactor" is also pretty vague, such that the terms can be used interchangeably across a fairly wide range of applications.

Good luck flying your car to work.

Re:The TL;DR

[inasity_rules]

My UID is low because I was an anonymous coward for many years. I could not be bothered to sign up.

None of the above invalidates my point, we can be accurate here without being pedantic or unclear to "the general populace," which slashdot isn't targeted at.

Re:The TL;DR

[adolf]

That makes sense to anybody and everybody. "Protection against what?" "Short circuit." "Oh".

These are the same people who have a light fixture that is intermittent, and declare that it is caused by a "short," when in reality they're almost certainly experiencing what would more appropriately described as a "long."

You're really not helping educate anyone at all. Please stop.

Re:The TL;DR

[adolf]

Your UID is low?

You're also a comedian, right?

Re:The TL;DR

[inasity_rules]

You're really not helping educate anyone at all. Please stop.

In this case, sadly this is true, though I don't believe I am the problem here. Never-the-less I shall leave it at that.

Re:The TL;DR

[inasity_rules]

*Not low. Typo. Sorry. On my off days I'm a comedian, but I'm not quiting my day job.

Re:The TL;DR

[adolf]

But in this case, your terms are wrong.

Whatever you say.

Re:The TL;DR

[inasity_rules]

If you insist..

Just becuase you can't show me a breaker with a protection relay in all that. So, yes your terms, in this instance, are very wrong.

Re:The TL;DR

[adolf]

What was it that I said about scale? I can't be bothered to look; perhaps you remember.

Re:The TL;DR

[inasity_rules]

And at this scale, your terminology is definitely and clearly wrong. What is not to get?

Re:The TL;DR

[inasity_rules]

I think I may have a way to get the concept through to you. On my one site, I have a very complex system controlled by a PLC. The PLC can(amoung many other functions) remotely close and trip most of the breakers in the substation, but is physically distant from the substation. The PLC will trip a breaker based on many factors including a physical safety interlock. What you're trying to tell me is amounts to "the PLC is part of the breaker". The issue is, the fundamental philosophy of a household breaker does not scale up to the devices mentioned in TFA, or the ones I work with. They are fundamentally different. A breaker is "just a switch", but the intelligence that controls the breaker is not part of that switch (with very good reason). It is therefore incorrect, in the context to refer to the entire system as a breaker, since the entire system may not even be contained in the panel you are trying to, think of as a "breaker." My PLC is in no way part of the breaker, and can not be considered so even though it controls the breaker. It also measures temperature and runs VSDs and various other things not related to the breaker. The concept is different. The over-simplification is dangerous, and annoying because I have had customers who think like you do, and as a result have very unrealistic and dangerous expectations of what things are or can do. That sort of thinking costs me time and could cost someone else their life. Perhaps this is why I haven't given up trying to get you to see what I'm saying.

Re:The TL;DR

[Anonymous Coward]

Low voltage, high current air circuit breakers (ACB's), tend to have integrated protection relays and CTs.

Anything above 1000 volts (MV or HV depending on your country) tends to have a separate protection relay, and separate CT's. often because the CTs are hooked to multiple protection relays for different purposes and /or for metering as well.

Lead electrical engineer for a large international contractor, Currently installing over 80 x MV boards and LV boards in 17 subs, with over 200mva installed power...pro-tip, pay peanuts and you get monkeys.

CYA

[msauve]

Yet, the manufacturer of the trip relay says "Based on the onsite testing, we have determined that if higher settings had been applied, the equipment would not have disconnected the power..." Based on Entergy's incorrect initial claims that "it wasn't us," I tend to think they're not being honest.

Re:CYA

[TX_Sparky]

Such devices do have settings for permissible FLA (full load amperes), time to trip, and (on most) many others. Very easily, the problem could have been not the device, but how it was set up.

Re:CYA

[sjames]

"What the hell's your problem? Just shove a penny in it so we can watch the game!"

Re:CYA

[thegarbz]

Manufacturer of a over current trip relay says if you raise the high current trip point to above the amount of current you were drawing it wouldn't trip?

Say it ain't so!

Re:CYA

[Anonymous Coward]

so, put a penny in the fuse box, so-to-speak, and everything would've been ok?

Re:CYA

[Anonymous Coward]

"it wasn't us".

Incompetence isn't a valid excuse, right? Color me surprised that new kit wasn't properly installed and calibrated.

Re:CYA

[bill_mcgonigle]

I smell BS.

Scenario: this part 'failed', the relay was opened, and the power went out. Subsequently the power was restored.

Must be true: Either the relay was reset or the relay was bypassed.

Known fact: The HID lamps they have take 20 minutes for a full reset cycle, and power was out for 35 minutes. Lights could have come back on as soon as 20 minutes into the outage.

Conclusion: the faulty part was identified and handled just under 35 minutes into the power outage.

This all seems reasonable. But...

Now, a week later, they've 'discovered' why the power went out by "tracing the source with testing"? No. What they have is a dispute as to whether the protective relays were properly set. We know both relays were both set to the same settings but the manufacturer is saying that setting was too low and the energy company is dancing around that specific question, but insisting it's not their fault.

The reason for them not to admit to an error seems to be that New Orleans is bidding on the 2018 Superb Owl as well.

Re:CYA

[Anonymous Coward]

I have worked for Entergy, and worked extensively with S&C.
S&C has always been forthcoming, even when it wasn't to their advantage.
Entergy, has not. They will provide tidbits of information that are true, that would lead someone to believe that this was someone else's fault (such as this relay didn't act exactly like another identical relay). But they won't release the full truth (such as, but the difference between the two relays was within tolerance and had nothing to do with the failure).

mob boss make a bad bet?

[Anonymous Coward]

Maybe some mob boss made a wicked bad bet and wanted to give the game a bit of a momentum shift, so he called his "buddies" at the electric union..."tell your boys to make it look like an accident"

Come on, if all those EU soccer games can be rigged, why not the NFL?

Re:mob boss make a bad bet?

[Anonymous Coward]

Sounds somewhat plausible. A long break gives the team that's behind a chance to adjust. Teams that are doing well are not going to change anything. I wonder if there are any statistics to validate the idea that a break like that favors the team that's behind. We all know there are "half-time adjustments", but the outage was right after half-time.

You could argue that the defense was tired; but the come-back was characterized by more SF offense, not better defense. So much for that.

FWIW, neither me nor a friend wanted to bet against SF so we bet on the absolute value of the spread. When they were getting demolished, I could console myself with winning the bet since I had taken the "large spread" position. A narrow SF victory would have been good even though I would lose the bet. I got the worst of both worlds--no hometown victory, and I lost the spread bet.

Re:mob boss make a bad bet?

[KingAlanI]

Commentators talked about the momentum shift, and then that happened.

Yeah, if the spread was Baltimore by 3 or less, this would affect the betting outcome but not the game outcome, which makes sense as a lower-impact way to 'fix' it. However, most spreads I saw were San Francisco by 4. Maybe the attempt to 'fix' it went wrong, maybe it was about some side bet.

Re:mob boss make a bad bet?

[KingAlanI]

*if the spread was Baltimore by 3 or more*

Did someone has his cheeseburger?

[Anonymous Coward]

Well has you?

Re:Did someone has his cheeseburger?

[sabri]

Yes, apologies, English is not my first language. I only scored 115 on my TOEFL test.

Re:Did someone has his cheeseburger?

[Anonymous Coward]

I'm guessing French is your first language.

Re:Did someone has his cheeseburger?

[AK Marc]

Seems TOEFL is for UK and Australia mainly. So what do you care?

Seems like system failures

[matty619]

Are frequently caused by the devices installed to prevent them. Quite ironical.

Re:Seems like system failures

[the eric conspiracy]

That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.

I think the turning point was when journaling file systems came to Linux.

Re:Seems like system failures

[ericloewe]

Ironic. And it's not irony.

Re:Seems like system failures

[Anonymous Coward]

You are quite right. Irony is raining on your wedding day, or perhaps winning a free ride when you've already paid. Maybe even running into a no smoking sign on your cigarette break, or ten thousand spoons when all you need is a knife?

Re:Seems like system failures

[MightyYar]

True, but there is a failure and then there is a FAILURE. Lights going out... that's an oops. Trunk line overheating and starting a fire during the Superbowl... that's worse. Transformer exploding during Superbowl... that's worse, too. So, yeah, the system failed - and maybe putting the circuit breaker in-line makes a problem more likely. But it almost certainly makes the failure less severe.

Re:Seems like system failures

[Anonymous Coward]

I was having more failures caused by the UPS's than if I didn't have them in my system.

If that's the case, you should have stopped buying shit equipment instead of ditching them altogether.

Re:Seems like system failures

[poofmeisterp]

That's why I stopped using UPS's on my home computers. I was having more failures caused by the UPS's than if I didn't have them in my system.

I think the turning point was when journaling file systems came to Linux.

I think it was a turning point when journaling file systems came to be -PERIOD-.

Re:Seems like system failures

[ahabswhale]

I've never experienced any problems with my UPSs and they have definitely saved me from some unnecessary shutdowns and reboots. On a side note, I never install the included software.

Re:Seems like system failures

[Anonymous Coward]

You know what's ironic?

Hematite.

Re:Seems like system failures

[Anonymous Coward]

The device is an overcurrent relay. Its purpose is to prevent a fire in downstream equipment due to a current higher than that equipment can handle. It did its job just fine.

Re:Seems like system failures

[antdude]

So what happens when you have a power outage? :P

Re:Seems like system failures

[Anonymous Coward]

Nope, those are all just coincidences, just like everything else in that song. That is what the true irony of the song title is.

Re:Seems like system failures

[Anonymous Coward]

Irony - 5. an outcome of events contrary to what was, or might have been, expected. (http://dictionary.reference.com)

So yes, it's fucking ironic.

Re:Seems like system failures

[AK Marc]

I've had that argument more times than I can count. Getting one resilient device is more reliable than two devices. Because the cluster/HSRP fails more often than the hardware does.

Re:Seems like system failures

[AK Marc]

Putting in a protection device to increase reliability actually decreases reliability, and you don't find that ironic? I don't think that word means what you think it means.

Re:Seems like system failures

[toddestan]

Likewise, I find that the cheap-ass RAID controllers built into motherboards tend to cause more data loss than just running JBOD and taking your chances. Though nowadays there is software RAID on Linux, which is good enough for a lot of things.

Re:Seems like system failures

[Anonymous Coward]

The second one.

Re:Seems like system failures

[sdsucks]

Wow, what kind of UPS's were you using? I don't recall ever having a failure caused by a UPS.

Re:Seems like system failures

[sdsucks]

I guess if your replacing one SPOF with another, it better be more reliable. ;)

Re:Seems like system failures

[dbIII]

I've had plenty with APC and plenty more with a cheap and nasty Chinese brand that has probably given up in shame.
Have enough of them and the high chance of failure stacks up so you start seeing a few. Have them for long enough and the batteries will die and won't always die gracefully. A frequent battery replacement schedule instead of waiting for a warning could get you around the last one. The largest problem I had was when there were minor power fluctuations that would make lights flicker and make the UPS's panic and shutdown instantly - thus leaving everything up that was not on a UPS while the "protected" stuff suddenly lost power. While that was due to unusual loading on the power network (I suspect it was a nearly crane regenerating as it rapidly lowered a load) it happened quite a couple of dozen times one year and we couldn't find a way to filter it out so eventually nearly everything was migrated off UPS. UPS, when it works, gives you time to write to a filesystem but that's no longer as important with things like batteries in controller cards and far more stable filesystems. Expecting it to tide you over a power outage is far too optimistic - that's what a generator with a big fuel tank is for.

Re:Seems like system failures

[greg1104]

It is possible to buy a UPS for around $40, and all of these items are crap. CyberPower, DirectUPS, Tripp-Lite, Opti-UPS, Minuteman, Powercom...any of these can end up becoming the least reliable component to a Linux system.

Even with APC, who sells reliable units if you spend enough, you're looking at a few hundred dollars for one that is usefully reliable. And even there you have to be careful, do your research, and test to be sure. The biggest issue right now is how power is generated, which determines whether they will support active PFC power supplies. The models from APC that say "stepped approximation to a sinewave" for example are nothing but trouble. You want "pure sine wave". The Smart-UPS PCW-SUA1000 has the right design at $370. It is depressing how bad their cheaper models are nowadays, and how much you can spend on a unit that's still junk.

Re:Seems like system failures

[drinkypoo]

If you're not trying to play big badass games you can build a low-power PC and power it with something like a PicoPSU, that one will do 160W sustained. Then you can build your own UPS trivially using a car battery and charger. Put the computer against an external wall and run power through it to deal with venting issues. It just sucks to have the power go out when you're in the middle of something. Or, of course, use a cheap inverter. I would have done this already but both my battery chargers are manual, they never stop charging. I suppose it would be fairly easy to build a circuit to do monitoring and to switch the charger using an Arduino or MSP.

Re:Seems like system failures

[ericloewe]

http://en.wiktionary.org/wiki/irony

What you consider irony is not actually irony, as you can see in 4. If we started calling dogs "cats", it wouldn't make them cats, would it?

Re: Seems like system failures

[Anonymous Coward]

was it an overcurrent relay? I didn't see it stated anywhere. There are other schemes to protect cable.

Re:Seems like system failures

[the eric conspiracy]

The computer turns off.

The only difference is that it's less graceful that if you have a UPS.

Re:Seems like system failures

[the eric conspiracy]

I'm very sparing with RAID as well. Maybe the balance would tilt in favor of UPS use if I used RAID more often.

Re:Seems like system failures

[antdude]

So you care not for data losses you were working on then. :( Are you sure all UPSes do that? I know one of mine did because it was very old.

Re:Seems like system failures

[AK Marc]

You can never eliminate a single point of failure. You can move it, but there's almost always a single point of falure left, usually the protocol/application that combines a box that's a single point of failure into a cluster that's a single point of failure. But as long as the lines are meshed, people are happy. Actual uptime is irrelevant.

Re:Seems like system failures

[AK Marc]

How long did you search to find one that had exactly what you wanted? The first hit searching on the word gave me http://dictionary.reference.com/browse/irony, which the first definition meets the definition used here.

Irony includes when the actual is the opposite of the intended. But then these discussions always come down to word nazis like yourself being believers in proscriptive language, not descriptive, and English is officially proscriptive (even if that is an oxymoron).

Re:Seems like system failures

[Anonymous Coward]

I think what you've described is not a single point of failure. You need two failures - the box and the protocol/application to have a system failure.

But design is a point of failure. That's why in the most resilient redundant system, you have your redundant power supplies be of different models. But there is one design you just can't make redundant - the design of the redundant system.

Re:Seems like system failures

[AK Marc]

You need two failures - the box and the protocol/application to have a system failure.

So you are telling me that if your RAID card fails and writes bad data to all 5 discs in your array, they will not have corrupt data because they are a Redundant Array of Inexpensive Discs. You are telling me that if you botch the config of VRRP, your network will not go down, because you must have at least one box fail to have a failure.

An interesting theory, but reality has proven you wrong a few million times.

Re:Seems like system failures

[the eric conspiracy]

I very rarely have data losses while working on a computer.

Back when I first started out I had the experience of using some quite dodgy software so I got into the habit of saving things quite frequently. It's served me well over the years.

Re:Seems like system failures

[antdude]

How about online stuff? That part sucks.

Re:Seems like system failures

[dbIII]

I suppose it would be fairly easy to build a circuit to do monitoring and to switch the charger using an Arduino or MSP

I'm sure you could do something better than what APC uses with such a thing since there is a lot more to it than their monitoring hardware.

Re:Seems like system failures

[drinkypoo]

I'm sure you could do something better than what APC uses with such a thing since there is a lot more to it than their monitoring hardware.

Are you telling me they don't use a microcontroller? I doubt that, although I suppose it's technically possible.

Re:Seems like system failures

[dbIII]

Their design has not changed for a long time and there are not many conditions it watches for - hence my problem with the things cutting out while everything else stays on. Some models and other brands may have a microcontroller and be able to better handle a wide range of conditions. The main thing is if it's your own you can modify it when such problems occur.
I mostly gave up on them since a full blackout stops people working in the office buildings anyway while a power glitch means I get dozens of calls about a server on UPS being down. Recent filesystems and hardware can handle pulling the plug far better than they used to.

Superbowl 47?

[oodaloop]

In combat sports where people get hurt for the amusement of spectators, we use Roman numerals.

Re:Superbowl 47?

[RivenAleem]

But your average Superbowl viewer will just ask "What's an Extra Large Vii?", quickly followed by "Does that come in a Supersize?"

Bill Maher

[Dr. Tom]

"Just when you thought the NFL couldn't get any blacker"

"God punishes Beyonce for acting like a stripper."

-- Bill Maher

Re:Bill Maher

[DigiShaman]

Bull shit! Bill Maher doesn't believe in God.

Is the maintenance engineer related to...

[140Mandak262Jamuna]

There was this engineer who was in charge of the production studio that did the live broadcast of the presidential debate between Jimmy Carter and Gerald Ford. I wonder if this stadium maintenance engineer is that guy's son or something...

Re:Is the maintenance engineer related to...

[fotoguzzi]

Could you elaborate? Was there a lighting incident in 1976 that only you remember?

Re:Is the maintenance engineer related to...

[Anonymous Coward]

The sound was off for about 30 minutes.

Re:Is the maintenance engineer related to...

[tipo159]

Could you elaborate? Was there a lighting incident in 1976 that only you remember?

The audio went out. http://www.youtube.com/watch?v=DrP5ZM0otP8

OOO WEE OOO

[uncoveror]

It was not any piece of technology that caused that outage, it was vengeful spirits!

FALSEHOOD !! HOWZ IT FEEL TO BE THIRD-WORLD US ??

[Anonymous Coward]

We expect shit like that here !! Now you and your WallStreet criminals running your country can eat your shit !!

12 minutes after the halftime show???

[Anonymous Coward]

As a technical troubleshooter I can not believe how they try to say it was not caused by the grand surge of electricity needed.
Maybe the device saw the usage as abnormal and just did it's job a little bit late.
Just admit it, you didn't think it would draw that much amps/be a problem with stage over-hype.

Re:12 minutes after the halftime show???

[poofmeisterp]

As a technical troubleshooter I can not believe how they try to say it was not caused by the grand surge of electricity needed.
Maybe the device saw the usage as abnormal and just did it's job a little bit late.
Just admit it, you didn't think it would draw that much amps/be a problem with stage over-hype.

First thing I did after seeing it was check solar activity. Once I saw that it was nominal and not CLOSE to a major power fluctuation event, I determined it was caused by Human Error. Don't fucking care what the error was - people will fix broken shit. It was just an error.

Alphas

[n2rjt]

Reminds me of an episode of the Syfy-channel show Alphas

Re:Alphas

[Mitreya]

Reminds me of an episode of the Syfy-channel show Alphas

Yeah, Superbowl reminds me of SyFy shows too.

Soooo

[wbr1]

After the HT show, Beyone went to her dressing room, switched on her super-vibe 6000, and popped a breaker, right?

Irony

[Gnaget]

If you have trouble figuring out what is ironic and what isn't: This is ironic.

Re:Irony

[Anonymous Coward]

What is "Irony"? (from wikianswers)

Irony is a verbal or situational context involving outcomes that are either unexpected, unanticipated, or actually the opposite of what they should be. A statement may be intended to mean the opposite what it normally means, or an event can occur that is unexpected. Sometimes it can be so unexpected that it's funny. For example: a man steps around a mud puddle to avoid getting his shoes wet, but steps on a broken sprinkler line which sprays him head to toe. Another example would be a thief who steals a car, only to be caught because the car belongs to the police commissioner. The 3 different types of irony are: verbal - similar to sarcasm, but not necessarily insulting. For example, an overachiever who passed a test says, "I bombed the test." situational - when a surprising and unexpected event occurs. For example, a professional swimmer almost drowns. dramatic - when the character of a story doesn't know something but the reader/viewer does, and acts in a way that is obviously ill-advised. For example, in a horror movie, a person wanders into an abandoned house. We can predict that ghosts will start playing tricks on that person, but he/she seems unaware of the ghosts.

http://wiki.answers.com/Q/What_is_irony

Re:Irony

[geminidomino]

I don't think Alanis Morisette nor too many of her fans read slashdot, but thanks anyway. :)

Comment removed

[account_deleted]

Comment removed based on user account deletion

Dude looks like a LADY

[Anonymous Coward]

LINUX and cheap ass stuff are like birds of a feather

Working just fine

[OzPeter]

From TFA

The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.

If the device tripped out because the load it saw exceeded its settings, then I'd say that the device was functioning perfectly fine at the Superbowl.

Now were those setting suitable for Beyonce's half time show? That would be the question to ask.

Re:Working just fine

[Anonymous Coward]

I think the actual question is the other direction, was Beyonce's half-time show suitable for the distribution system at the facility.

Re:Working just fine

[Anonymous Coward]

I think the actual question is the other direction, was Beyonce's half-time show suitable for the distribution system at the facility.

The Beyonce halftime show was 'off the grid', running totally off of generators. So, nothing to do with the stadium power problems.

Raise your hand if you wanted it to be "Windows"

[erroneus]

I'll start by raising mine. Seriously. It's just funny to me now.

SCADA

[Anonymous Coward]

Who wants to bet it is networkable and therefore exploitable? I'm looking at you, China.

Ravens can see in the dark dept?

[Shag]

I did not know that. I thought you needed a Superb Owl to do that.

Another dead rumor

[Tablizer]

So it wasn't Beyonce's blow-dryer after all?

NFL = No Fucking Lights

[Tablizer]

nuf sed

Phrasing

[Anonymous Coward]

Never use defective and protective in the same phrase. Avoid it with a triple-check.

JJ

Re:Explanation

[Anonymous Coward]

Oops, I thought it was some kind of dish. Thanks!

Re: Explanation

[boundary]

Thanks. I assume it's for crown green bowls, or something similar?

Next Year...

[Anonymous Coward]

The NFL just announced that next year, the Superbowl will be played at a Motel 6, because they'll leave the lights on for you.

Who gets fired vs. who gets bonuses

[girlinatrainingbra]

re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels, so the firing most often will happen to those at the lower level who executed the commands, including putting in crappy materials that were ordered when the higher-ups want to save money. At least that seems to be the way of the USA; Japan's older way would have those responsible all the way up to the chair/CEO stepping up and taking blame and getting out of the way or resigning. Over here, the standard is to blame someone else.
.
"The buck stops here" for Eisenhower; but notice how Obama stayed out of the way when Hillary Clinton tried to initially take the blame for Benghazi, but ultimately tried to tap-dance her way out of all responsibility when it actually came time for the congressional hearings.

Re:Who gets fired vs. who gets bonuses

[Anonymous Coward]

re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.

hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels, so the firing most often will happen to those at the lower level who executed the commands, including putting in crappy materials that were ordered when the higher-ups want to save money. At least that seems to be the way of the USA; Japan's older way would have those responsible all the way up to the chair/CEO stepping up and taking blame and getting out of the way or resigning. Over here, the standard is to blame someone else.
.

"The buck stops here" for Eisenhower; but notice how Obama stayed out of the way when Hillary Clinton tried to initially take the blame for Benghazi, but ultimately tried to tap-dance her way out of all responsibility when it actually came time for the congressional hearings.

You must have been watching/listening to the fox news take on things, because Hillary crucified congress during the attempted inquisition and treated them like the weasels they were bitch-stomped their rhetoric.

Re:Who gets fired vs. who gets bonuses

[Runaway1956]

What you say is pretty much true. Sadly, that didn't happen because Shrillary is that good - it happened because congress are such pussified pretenders. Few congressmen seem to have any clue what life is like outside the United States. They are to busy sucking the corporate teats that keep them in power to pay attention to the real world.

Re:Who gets fired vs. who gets bonuses

[Hognoxious]

re: Presumably the person that receives the big end-of-year bonus when everything goes well?
.
hahahaha! No, the guys/gals near the top get the bonuses when everything goes well. Scapegoats exist at the lower levels

Well thanks for pointing that out, Miss tic-tac tits.

If you hadn't been here we'd have been grossly misinformed about how capitalism works! Shame on you, isorox, for lying to us so!

Re:Who gets fired vs. who gets bonuses

[magic maverick]

Just a heads up, you're an asshole...

Oh, and I think I might have worked out why you can't find any women who actually like you. (Not even your mother likes you. She just pretends.)

Re:Who gets fired vs. who gets bonuses

[Hognoxious]

Not even your mother likes you. She just pretends.

You do know that ouija boards are bunk, right?

Re:Who gets fired vs. who gets bonuses

[Anonymous Coward]

Just a heads up. You're still an asshole. In fact, if you were not you'd probably have killed yourself by now. Do the world a favor and do it. But you have decided that you will live as long as you can. To just screw things up for the rest of us. Asshole.

I hope you die in a horrific fire screaming in agony, and then you can go and burn some more in hell.

Process Management Failure

[Anonymous Coward]

This was a process management failure. The new protective relay was installed just before the superbowl, so the first real test of the new installation was... the superbowl itself.

Just like you never change a working system just before the client demo, this was a "Don't fuck with what isn't broken" situation.

Re:Process Management Failure

[thebigmacd]

BS

The relay device wasn't put online until December 21. Between then and the Super Bowl, the device functioned properly during three major events -- the New Orleans Bowl, a Saints-Panthers NFL game, and the Sugar Bowl -- Entergy said.

TFA

[Anonymous Coward]

You've got to be kidding me, the guy they quote as an electrical engineering professor, I presume to add an air of validity and weight to the fluff, is grossly incorrect in the facts about protective relays. Either he doesn't know wtf he's talking about, or he needs to get out of his tower and out into the real world every now and again.

Firstly, as large as a truck? Breakers and reclosers can be very large indeed, but the protective relay is a small computerized device installed in the DOOR of an MCC or switchgear lineup. Most of them are about the size of a toaster. They take in readings from instrumentation located in different places around the gear they are protecting such as voltage, current, phasing, temperature, etc. They perform calculations to determine things like phase imbalance (all large systems are polyphase), ground currents, power factor and the like, and then based on those calculations determine whether to command action from other devices in the gear, such as breakers.

Secondly, as to his assertion that they are notoriously unreliable, he is also ridiculously incorrect. I work in industrial process controls, and have overseen the installation of, and personally setup/programmed literally hundreds of these devices in my career, and have yet to have any experiences that would cause me to believe that the devices themselves are dodgy.

The problem really is that setting the proper parameters is difficult, and it's both a task that many (perhaps most) EEs are not cut out for, and at the same time a balance among many tradeoffs between safety, efficiency and uptime. That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

tl;dr Programming protective relays correctly is hard work, and as in all types of engineering, a tradeoff between many factors.

Re:TFA

[geoskd]

That the electric utility is called before a city council meeting to "answer for" a power outage at a football game is, frankly, laughable.

When its an event with that kind of money involved, you bet they get called in to explain it. Power outages are just not common in the U.S. and are normally only due to human error, or acts of nature. Devices that can fail should be introduced into systems in ways that provide redundancy. This is done for the sake of continued up-time, and safety. Not having sufficient redundancy can easily be referred to as human error. Saying that the task is "hard work" is an unacceptable excuse. If the system design was insufficient, then it should not have been used. If it was sufficient but was installed wrong, then the installer is at fault. Cutting costs is not an acceptable reason for a system failure. If the proper equipment and installation cost more than there was budget for, then the issue should have been escalated until it could be resolved by adding additional resources, or scrapping the project. Endangering a $100 million operation for the sake of saving $50,000 on an electrical installation is cause for an inquest, and given the severity, likely someone will loose their job as a result.

-=Geoskd

Re:TFA

[Anonymous Coward]

How much redundancy do you want though? A complete redundant set of transmission lines and all the associated equipment to the stadium? Where are you going to put those lines? What do they cost? The last I knew, high voltage transmission lines were about $1million/mile in a rural area, and that was five years ago. I have no idea what they would cost in an urban area. If the football game is so important, maybe they should have their own power generation like hospitals do.

Re:TFA

[geoskd]

How much redundancy do you want though? A complete redundant set of transmission lines and all the associated equipment to the stadium? Where are you going to put those lines? What do they cost? The last I knew, high voltage transmission lines were about $1million/mile in a rural area, and that was five years ago. I have no idea what they would cost in an urban area. If the football game is so important, maybe they should have their own power generation like hospitals do.

Something the size of a stadium might have redundant power supplies, or it might not, but no single piece of equipment should be capable of shutting down a supply line unless there really is danger. The offending piece of equipment should have had a failover / backup unit if needed. In fact it probably had such equipment, but it wasn't triggered because the problem was not equipment failure, but rather improper installation. No amount of redundancy can fully compensate for improper installations. There are just too many creative idiots out there. That having been said, the "idiot" here should be sent packing. When you're dealing with power levels that can cause major catastrophes, you really don't want someone with a track record of failure. Failures can be lethal in that business.

-=Geoskd

Re:TFA

[Anonymous Coward]

> Saying that the task is "hard work" is an unacceptable excuse.

That isn't usually why failures occur in privately owned equipment. Typically management doesn't want to spend a single cent more to make something more reliable. They know they will have failures and blame the engineers so they don't give a damn. In this case, it is the typical example of management being pound foolish and blaming engineers for their own mistakes.

The last job I was fired from was because the power supply in the production pay system quit and was down for a couple of hours which stopped all of the women using the sewing machines. The downtime cost the company almost $50k. My request for a replica db server for $1k plus a half of a day to setup was turned down a few months before. I was fired because we didn't have the replica server. As the plant manager said, I knew we needed it so it was my fault I didn't force him to approve the request to buy it. I also wasn't paid unemployment because the state agreed that even though I didn't have the budget to maintain a reliable system, it was my responsibility, and not management's, to keep the production pay system running. In the end, the engineers always get screwed.

Re:TFA

[Anonymous Coward]

This was a piece of safety equipment. It's *intended* to be a single point of failure.

Also, if you only have one power feed there's very little value -- and a lot of costs -- in having multiple current paths.

Re:TFA

[Anonymous Coward]

One person cannot be culpable for the failure of a test program to verify this equipment would function as intended. Testing is baked into the engineered procedures to design and install this system, and their work controls dictate that they be reviewed and approved. It's obvious you know little regarding electrical distribution and protection, in addition to not rtfa.

Re:TFA

[Anonymous Coward]

or the journalist on a deadline to file didn't do enough homework understand a word the guy said and wrote what he thought he heard instead, horribly misquoting the guy in doing so?

I guess you've never been quoted in the paper before. It's kinda scary how badly the can get it wrong.

Re:TFA

[geoskd]

One person cannot be culpable for the failure of a test program to verify this equipment would function as intended. Testing is baked into the engineered procedures to design and install this system, and their work controls dictate that they be reviewed and approved. It's obvious you know little regarding electrical distribution and protection, in addition to not rtfa.

He or she is culpable if they put the wrong values in for the operating parameters. If this had gone the other way and caused a fire or worse, then there would be criminal negligence charges. Engineers get paid to do the job correctly. Where safety is at sake, they shouldn't be putting their name on something unless it has been properly verified. I can understand an engineer wanting to be conservative in these values so that a problem will tend to be a false positive rather than a failed safety, but it doesn't change the fact that there is a narrow "correct" range for these operating values that will allow the device to operate correctly without causing false positives. The fact that the values were not within that range indicates that the responsible engineer did not know what that range was, and consequently had not done his/her job.

-=Geoskd

Well, theres yer problem.

[funwithBSD]

You were running Windows on your Superdome!

The true cause

[Anonymous Coward]

and I thought it was Beyonces singing that caused the power outage.

Re:Explanation

[Anonymous Coward]

I may dislike any tax exemption, but it does have its uses. These sporting events bring in a lot of money and also spend a lot of money. They are paying taxes in dividends instead of up front.

Re:Explanation

[KingAlanI]

yeah, some people thought it was suspicious because stopping Baltimore's momentum helped enable that San Francisco rally/near-comeback.

Irony!

[Jaden42]

Someone should inform Jewel that this is an appropriate example of irony!

Re:Irony!

[wikdwarlock]

Ok.

I like Jewel's music.

I like Alanis' music.

People will judge, they always do.

Both artists write good lyrics, and have some pretty infamously bad examples of using the wrong word at the wrong time.

So, to set the record straight:

Alanis: Sang "Isn't it ironic?" when, in fact, few, if any of the scenarios she mentioned were irony (implied meaning in opposition to the literal meaning)

Jewel: Sang "...with such casualty", when she actually mean casualness (i.e. not death, but with little regard)

So there, that's settled.

Wrong mediocre 90s musician.

[Anonymous Coward]

You're thinking of Alanis Morissette...

Re:Wrong mediocre 90s musician.

[Jaden42]

Ugh you are right. Apologies. My want to get that joke out there trumped fact checking. Do I get points if I am still laughing inside though?

Sensationalist Journalism?

[phizi0n]

"Power company Entergy New Orleans says the Super Bowl blackout was caused by device designed to prevent power outages."

Isn't the point of the protective relay to CAUSE power outages when the load is too high in order to prevent damage to equipment and fires from the line carrying more load than it should.

Re:Sensationalist Journalism?

[Celeritas 5k]

I had this thought as well-- "preventing power outages" is only a function of this relay in the sense that it's difficult to supply power to a burned-down building.

Re:Sensationalist Journalism?

[Anonymous Coward]

Only when it is needed. In this case it wasn't needed. The cable and transformer were carrying less than their rated load.

This merits a Scotty quote

[Sol Rosinberg]

"The more they overthink the plumbing, the easier it is to stop up the drain."

who cares?

[Anonymous Coward]

why is this in the feed? spamish.

Dear

[nimbius]

Mr. Leonard, Its me, jim. you might not know me very well, im not exactly C-level so i never really met you. I just wanted to ask, hows that Relay sub-subcontractor thing working out for you? Me? oh ive been pretty successful since the termination with my own engineering consulting firm. We work on switchgears, relays, you name it!

Hope your team won the superbowl, Jim Ex master relay engineer, Entergy INC Local Union affiliated.

Typical

[Anonymous Coward]

As usual, more downtime is caused by the equipment meant to protect against problems than the actual problems they protect against. In the past three years, the cage we have at Level 3 has lost power on average every three months. Their equipment made by Eaton (formerly Cutler-Hammer) caused most of the downtime. The most recent three power losses were caused by the automatic transfer switch disconnecting the data center from both the generator and battery power because the utility power from Seattle P&L was too far out of sync from 60 hz. In all three cases we were not running on utility power, but Eaton made the decision that because utility power was less than third-world standards that it should disconnect all three power sources (utility, battery, and generator) in order to cause downtime. Actual power problems only caused single downtime in those three years, but Eaton has decided to cause power outages a dozen times. As usual, the crooked vendors try to create a need for an expensive service agreement by making their equipment so unreliable. Only once did Level 3 pay for breaking their SLA because they engage in finger pointing rather than engineering.

Adding complexity does not improve reliability

[gweihir]

... in general. It is just the same as with security: If you do not design it in from the beginning, it does not really help and ha s a good change of making things worse.

As it looks like the circuit-breaker type device was configured wrongly, that holds even more: The more components that need a setting, the higher the probability somebody messes up or has his/her capability for understanding how the system works exceeded.

Re:Adding complexity does not improve reliability

[inasity_rules]

But if you are competent and get the settings right, those relays can prevent huge amounts of damage. Well worth it. I know, I work with them, and if I did not have such devices the one site I worked at would have burned down when the capacitor banks faulted.

Re:Adding complexity does not improve reliability

[gweihir]

I don't dispute that. I am merely saying that they need to be part of the initial design to be most effective and that later additions need to be done very, very carefully.

Cut through the bull

[Anonymous Coward]

Blame for the New Orleans power failure is a hot potato that no one will accept. Someone in the Superdome failed to properly calculate what the load would be. He will probably get a promotion while they shift the blame to "Faulty equipment" and the press will buy it if they repeat that long and loud enough. Has anyone looked to see just how the system was set? I bet that it got reset real fast before anyone came snooping around. Its just like asking where all the Federal money for emergency hurricane supplies went when it didn't go to the superdome.

No good info yet

[Animats]

Even EL&P's coverage doesn't say anything substantive. When we start seeing articles with diagrams of the feeders, maybe we'll know something.

An enclosed stadium in Louisiana in winter shouldn't be anywhere near its electrical load peak. No air conditioning load.

Battle Short!!!

[Anonymous Coward]

Protection circuits are not necessary during the big game...besides wires make perfectly good fuses.

Slow electrons?

[Hognoxious]

Now were those setting suitable for Beyonce's half time show? That would be the question to ask.

Apparently not, I was able to see and hear it 100%.

You do know that the outage occurred after the second half kicked off, right?

Furthermore..

[Anonymous Coward]

Why can't it be fixed in 30 seconds. The assholes responsible
for this moronic shit need to be charged the super bowl
advertising rate for a 40 minute spot.

Re:Furthermore..

[Anonymous Coward]

oh, that was just cuz with lights at that power level, you need to let them cool off before firing them up again, like a video projector bulb

What is this?

[Anonymous Coward]

A SCADA device driven by Windows XP SP1 and administered
from Chittagong?

In another news,

[Anonymous Coward]

In another news, two unlucky women were shot by police officers who are "designed" to protect them.

This was in no way a utility grid hack??

[Anonymous Coward]

Is it possible this was a hack/attack on the electrical? Better to say faulty device than admit, we got attacked, yes?