BP Gulf of Mexico Rig Lacked Alarm Systems

DMandPenfold writes "BP's monitoring IT systems on the failed Deepwater Horizon oil rig relied too heavily on engineers following complex data for long periods of time, instead of providing automatic warning alerts. That is a key verdict of the Oil Spill Commission, the authority tasked by President Barack Obama to investigate the Gulf of Mexico disaster."

Comments

As opposed to...

[toejam13]

Three Mile Island, where the complaint was that there were too many alarms going off.

Re:As opposed to...

[Gruturo]

Three Mile Island, where the complaint was that there were too many alarms going off.

Yeah, surprisingly alarms have to be neither missing nor useless (by being irrelevant, hard to understand, going off for the wrong reasons, presenting wrong scenario, not correlating causes etc etc etc).

Who'd have thunk it.

Re:As opposed to...

[ColdWetDog]

Truly amazing, indeed. Too lazy to look it up, but earlier reports had shown that Transocean (the rig owners, not BP like the stupid article mentions) had shut down many automatic warning systems because of too many false positives.

It's not like we've never seen this sort of thing before ...

"You are about to do something."

CANCEL, or ALLOW?

Re:As opposed to...

[omglolbah]

Indeed. Alarm suppression is a complex thing to set up in many cases. I personally work in the business and know how much thought goes into the alarm handling of the plants operating in Norwegian waters.

One example of a "simple" suppression case is that if Controller A goes down, you do not need to tell the operator that ALL signals on this controller is in "bad quality" or out of bounds. What you need to tell them is that the controller is down, and which systems are affected (which they will see on their displays as valves change color or somesuch. Our system uses white asterisks and white color to indicate that something is 'dead')

More complex cases are things like not throwing alarms for low flow rates in pipes where the valves are closed, or not throw electric alarms on equipment set to maintenance mode.

Regardless of all this, there should be an alarm system that has priorities.

Pri 1 alarms are such that they require IMMEDIATE attention. Such as a dangerous triple-high alarm (HHH or 3H) of a tank, pressure or temperature or a controller going down.
Pri 2 would be alarms that could develop into Pri 1 if not handled within a few minutes (H/HH) alarms etc.
Pri 3 would be what we call "pre-alarms". Things that could cause process upset or issues down the line. Like a low flow of coolant even though the temperature of the equipment being cooled hasnt started raising yet. Or a low level in a fuel tank.
Pri 4 we usually assign as maintenance issues. Like two redundant sensors having more than 0.5% deviation between them (But not enough to cause a real alarm). Things that should be looked at but within a day or so.

Being able to filter alarms like this helps immensely during an emergency. This is an old system with a limited number of 'alarm groups' and 'priority levels' but it still works fairly well. Operators can see what happens even with several hundred alarms going off at the same time. On our simulator we did a fun test where we tripped 70% of the plant (about 18000 distinct 'tags' or io points went into Bad quality and several thousand in alarm).
The operators were able to stop the cascade failure and no pipe burst in the simulator :)

Shit -will- hit the fan. It is always nice to be able to filter it so that only the important shit actually hits the wall :p

Re:Seems a little unrelated

[tomhudson]

And there was another near-disaster because at one nuke plant, the button you had to press was back-lit by a bulb that, over time, had caused the plastic to expand to the point that the button COULDN'T be pressed - which they found out the hard way.

Things will always fail in weird, unexpected ways - that's why you need humans in the loop.

Re:Why do they even bother?

[tomhudson]

Here's one fact - the regulators screwed up. Blaming it on a lack of alarms is disingenuous at best, corrupt at worst.

1. Regulators Failed to Address Risks in Oil Rig Fail-Safe Device
   http://www.nytimes.com/2010/06/21/us/21blowout.html?_r=1&pagewanted=all
2. Spill report: It could happen again
   'Failure of management' and regulators given blame for disaster
   http://www.chron.com/disp/story.mpl/business/7367856.html
3. Slick Operator
   How British oil giant BP used all the political muscle money can buy to fend off regulators and influence investigations into corporate neglect.
   http://www.newsweek.com/2010/05/07/slick-operator.html

This wasn't a technical failure - it was a failure brought out by greed and corruption. The blow-out was only the symptom, and addressing the symptom isn't going to prevent similar incidents from happening again.

We've seen this before - the mortgage disaster and bank bailouts, the savings and loan disaster, etc.

Start by fixing campaign financing - private donations only, strict annual limit per capita, no 3rd party involvement, etc.

-- Barbara

Nagios

[IceCreamGuy]

Haven't they been on Nagios Exchange recently? check_catastrophe.pl has been out for like 3 years!

check_catastrophy -H blowout-preventer716.haliburton.com -w ANY_LEAKS - c ANY_FRIGGIN_LEAKS

Re:how much did that cost

[hedwards]

I think everyone's familiar of that phenomenon regarding the alarm that cried wolf due to all the car alarms. Rarely do people even turn their head when they hear a car alarm.

Competent professionals don't do that. The problem with car alarms is that they aren't aimed at professionals, competent or otherwise, they're aimed at the general public and the mechanism they use isn't typically going to assure that anything is going on.

Competent professionals like the ones that are supposed to be running rigs should know to check them out every time and not turn the alarm off withotu ascertaining that the alarm is in fact false. Disabling an alarm should only be done when there are adequate contingency plans in place to handle if the condition happened and how they would respond.

I used to work security at a high rise and we'd often times have alarms turned off on portions of the building. It was the only way to ensure that under certain circumstances that work wouldn't cause a false alarm. It was done in a controlled way with plans in place to make sure that there was somebody keeping an eye on it while the work was being done, and that the alarms would be turned back on when they could be.

And every time that building had an alarm go off which wasn't a known cause, it was always investigated promptly. Alarms that go off repeatedly need to be fixed, not disabled.

Re:how much did that cost

[Rob the Bold]

I don't even want to know how much tax payer money was pissed away for that "key verdict" - having worked with quite a few monitoring and alarm systems for years I can tell you that most of the time "automatic alarms" get ignored and in fact can cause worse problems when an actual real alarm does occur because of how the operators tune them out - seems like they completely missed the mark on this - the real problem is most likely where you would expect it, the people running the system - human error I am sure !

You don't even have to ignore the alarm that isn't there. But I don't think the "alert" that we're discussing is the big klaxon/flashing sign reading "OIL LEAK," or an oil pressure light with electrical tape over it. What the article indicates was missing was an automatic method of indicating that a failure was imminent. As far as the cost of determining this: learning from mistakes can be expensive. Not learning from mistakes is likely even more so.

Re:Why do they even bother?

[omglolbah]

Have a peek at the Norwegian sector. We've been doing this shit since the 70s and try damn hard to not have another Alexander Kielland...
http://en.wikipedia.org/wiki/Alexander_L._Kielland_(platform)

The norwegian petroleum oversight is something... The regulators are ruthless when it comes to compliance and better yet... they are not directly controlled by politicians ;)

The cost of one fuckup is too much to allow people to cut corners.

I sure as hell dont in my job... and I do it for a living. When we have the option of doing it right, or doing it fast.. we pick right. Every time. I dont care if the customer is pissed at things being delayed. We do it -right-.

I know BP leased the rig, but come on

[AGMW]

it was Transocean that owned and operated the rig?, so perhaps the story could better be titled:-

Transocean Gulf of Mexico Rig, leased to BP, lacked Alarm Systems

This means they learned nothing

[magus_melchior]

They had this exact problem with Texas City-- they didn't do maintenance on the systems, so a subsystem overfilled with volatile hydrocarbons with no alarms going off at all-- and when one alert sounded at the monitoring area, they ignored it. They didn't invest the (relatively) small cost of installing a flare (to burn off excess), so the excess hydrocarbons spilled out into the open. Cost-cutting and an incredibly cavalier approach to maintenance from the London management generated a fucking fuel-air bomb in Texas.

This is one instance where the Brit management, when they changed to Hayward, should have told their investors to "fuck off-- er, give us a few years" and spend the necessary money to get their facilities up to snuff, or decommission the facilities that are too costly to maintain. Alas, profit motive proved more powerful than basic empathy or responsibility.