BP Gulf of Mexico Rig Lacked Alarm Systems

DMandPenfold writes "BP's monitoring IT systems on the failed Deepwater Horizon oil rig relied too heavily on engineers following complex data for long periods of time, instead of providing automatic warning alerts. That is a key verdict of the Oil Spill Commission, the authority tasked by President Barack Obama to investigate the Gulf of Mexico disaster."

Comments

As opposed to...

[toejam13]

Three Mile Island, where the complaint was that there were too many alarms going off.

Re:As opposed to...

[Gruturo]

Three Mile Island, where the complaint was that there were too many alarms going off.

Yeah, surprisingly alarms have to be neither missing nor useless (by being irrelevant, hard to understand, going off for the wrong reasons, presenting wrong scenario, not correlating causes etc etc etc).

Who'd have thunk it.

Re:As opposed to...

[ColdWetDog]

Truly amazing, indeed. Too lazy to look it up, but earlier reports had shown that Transocean (the rig owners, not BP like the stupid article mentions) had shut down many automatic warning systems because of too many false positives.

It's not like we've never seen this sort of thing before ...

"You are about to do something."

CANCEL, or ALLOW?

Re:As opposed to...

[omglolbah]

Indeed. Alarm suppression is a complex thing to set up in many cases. I personally work in the business and know how much thought goes into the alarm handling of the plants operating in Norwegian waters.

One example of a "simple" suppression case is that if Controller A goes down, you do not need to tell the operator that ALL signals on this controller is in "bad quality" or out of bounds. What you need to tell them is that the controller is down, and which systems are affected (which they will see on their displays as valves change color or somesuch. Our system uses white asterisks and white color to indicate that something is 'dead')

More complex cases are things like not throwing alarms for low flow rates in pipes where the valves are closed, or not throw electric alarms on equipment set to maintenance mode.

Regardless of all this, there should be an alarm system that has priorities.

Pri 1 alarms are such that they require IMMEDIATE attention. Such as a dangerous triple-high alarm (HHH or 3H) of a tank, pressure or temperature or a controller going down.
Pri 2 would be alarms that could develop into Pri 1 if not handled within a few minutes (H/HH) alarms etc.
Pri 3 would be what we call "pre-alarms". Things that could cause process upset or issues down the line. Like a low flow of coolant even though the temperature of the equipment being cooled hasnt started raising yet. Or a low level in a fuel tank.
Pri 4 we usually assign as maintenance issues. Like two redundant sensors having more than 0.5% deviation between them (But not enough to cause a real alarm). Things that should be looked at but within a day or so.

Being able to filter alarms like this helps immensely during an emergency. This is an old system with a limited number of 'alarm groups' and 'priority levels' but it still works fairly well. Operators can see what happens even with several hundred alarms going off at the same time. On our simulator we did a fun test where we tripped 70% of the plant (about 18000 distinct 'tags' or io points went into Bad quality and several thousand in alarm).
The operators were able to stop the cascade failure and no pipe burst in the simulator :)

Shit -will- hit the fan. It is always nice to be able to filter it so that only the important shit actually hits the wall :p

Re:Why do they even bother?

[tomhudson]

Here's one fact - the regulators screwed up. Blaming it on a lack of alarms is disingenuous at best, corrupt at worst.

1. Regulators Failed to Address Risks in Oil Rig Fail-Safe Device
   http://www.nytimes.com/2010/06/21/us/21blowout.html?_r=1&pagewanted=all
2. Spill report: It could happen again
   'Failure of management' and regulators given blame for disaster
   http://www.chron.com/disp/story.mpl/business/7367856.html
3. Slick Operator
   How British oil giant BP used all the political muscle money can buy to fend off regulators and influence investigations into corporate neglect.
   http://www.newsweek.com/2010/05/07/slick-operator.html

This wasn't a technical failure - it was a failure brought out by greed and corruption. The blow-out was only the symptom, and addressing the symptom isn't going to prevent similar incidents from happening again.

We've seen this before - the mortgage disaster and bank bailouts, the savings and loan disaster, etc.

Start by fixing campaign financing - private donations only, strict annual limit per capita, no 3rd party involvement, etc.

-- Barbara

Re:how much did that cost

[Rob the Bold]

I don't even want to know how much tax payer money was pissed away for that "key verdict" - having worked with quite a few monitoring and alarm systems for years I can tell you that most of the time "automatic alarms" get ignored and in fact can cause worse problems when an actual real alarm does occur because of how the operators tune them out - seems like they completely missed the mark on this - the real problem is most likely where you would expect it, the people running the system - human error I am sure !

You don't even have to ignore the alarm that isn't there. But I don't think the "alert" that we're discussing is the big klaxon/flashing sign reading "OIL LEAK," or an oil pressure light with electrical tape over it. What the article indicates was missing was an automatic method of indicating that a failure was imminent. As far as the cost of determining this: learning from mistakes can be expensive. Not learning from mistakes is likely even more so.

I know BP leased the rig, but come on

[AGMW]

it was Transocean that owned and operated the rig?, so perhaps the story could better be titled:-

Transocean Gulf of Mexico Rig, leased to BP, lacked Alarm Systems

This means they learned nothing

[magus_melchior]

They had this exact problem with Texas City-- they didn't do maintenance on the systems, so a subsystem overfilled with volatile hydrocarbons with no alarms going off at all-- and when one alert sounded at the monitoring area, they ignored it. They didn't invest the (relatively) small cost of installing a flare (to burn off excess), so the excess hydrocarbons spilled out into the open. Cost-cutting and an incredibly cavalier approach to maintenance from the London management generated a fucking fuel-air bomb in Texas.

This is one instance where the Brit management, when they changed to Hayward, should have told their investors to "fuck off-- er, give us a few years" and spend the necessary money to get their facilities up to snuff, or decommission the facilities that are too costly to maintain. Alas, profit motive proved more powerful than basic empathy or responsibility.