BP Gulf of Mexico Rig Lacked Alarm Systems

DMandPenfold writes "BP's monitoring IT systems on the failed Deepwater Horizon oil rig relied too heavily on engineers following complex data for long periods of time, instead of providing automatic warning alerts. That is a key verdict of the Oil Spill Commission, the authority tasked by President Barack Obama to investigate the Gulf of Mexico disaster."

Comments

As opposed to...

[toejam13]

Three Mile Island, where the complaint was that there were too many alarms going off.

Re:As opposed to...

[Anonymous Coward]

Let's not forget that TMI was contained. The final safeguards system worked. In the BP spill it didn't. And the nuclear industry learned its lessons after TMI (as well as the NRC).

The oil industry needs to learn its lessons in the same way or else be regulated to the same extent as the nuclear industry. There really ought to be a NRC-like organization regulatory agency that ensures the safety of these rigs.

Re:As opposed to...

[omglolbah]

Indeed. Alarm suppression is a complex thing to set up in many cases. I personally work in the business and know how much thought goes into the alarm handling of the plants operating in Norwegian waters.

One example of a "simple" suppression case is that if Controller A goes down, you do not need to tell the operator that ALL signals on this controller is in "bad quality" or out of bounds. What you need to tell them is that the controller is down, and which systems are affected (which they will see on their displays as valves change color or somesuch. Our system uses white asterisks and white color to indicate that something is 'dead')

More complex cases are things like not throwing alarms for low flow rates in pipes where the valves are closed, or not throw electric alarms on equipment set to maintenance mode.

Regardless of all this, there should be an alarm system that has priorities.

Pri 1 alarms are such that they require IMMEDIATE attention. Such as a dangerous triple-high alarm (HHH or 3H) of a tank, pressure or temperature or a controller going down.
Pri 2 would be alarms that could develop into Pri 1 if not handled within a few minutes (H/HH) alarms etc.
Pri 3 would be what we call "pre-alarms". Things that could cause process upset or issues down the line. Like a low flow of coolant even though the temperature of the equipment being cooled hasnt started raising yet. Or a low level in a fuel tank.
Pri 4 we usually assign as maintenance issues. Like two redundant sensors having more than 0.5% deviation between them (But not enough to cause a real alarm). Things that should be looked at but within a day or so.

Being able to filter alarms like this helps immensely during an emergency. This is an old system with a limited number of 'alarm groups' and 'priority levels' but it still works fairly well. Operators can see what happens even with several hundred alarms going off at the same time. On our simulator we did a fun test where we tripped 70% of the plant (about 18000 distinct 'tags' or io points went into Bad quality and several thousand in alarm).
The operators were able to stop the cascade failure and no pipe burst in the simulator :)

Shit -will- hit the fan. It is always nice to be able to filter it so that only the important shit actually hits the wall :p

Re:Seems a little unrelated

[tomhudson]

And there was another near-disaster because at one nuke plant, the button you had to press was back-lit by a bulb that, over time, had caused the plastic to expand to the point that the button COULDN'T be pressed - which they found out the hard way.

Things will always fail in weird, unexpected ways - that's why you need humans in the loop.

I know BP leased the rig, but come on

[AGMW]

it was Transocean that owned and operated the rig?, so perhaps the story could better be titled:-

Transocean Gulf of Mexico Rig, leased to BP, lacked Alarm Systems

This means they learned nothing

[magus_melchior]

They had this exact problem with Texas City-- they didn't do maintenance on the systems, so a subsystem overfilled with volatile hydrocarbons with no alarms going off at all-- and when one alert sounded at the monitoring area, they ignored it. They didn't invest the (relatively) small cost of installing a flare (to burn off excess), so the excess hydrocarbons spilled out into the open. Cost-cutting and an incredibly cavalier approach to maintenance from the London management generated a fucking fuel-air bomb in Texas.

This is one instance where the Brit management, when they changed to Hayward, should have told their investors to "fuck off-- er, give us a few years" and spend the necessary money to get their facilities up to snuff, or decommission the facilities that are too costly to maintain. Alas, profit motive proved more powerful than basic empathy or responsibility.