Slashdot Mirror
BP Gulf of Mexico Rig Lacked Alarm Systems
DMandPenfold writes "BP's monitoring IT systems on the failed Deepwater Horizon oil rig relied too heavily on engineers following complex data for long periods of time, instead of providing automatic warning alerts. That is a key verdict of the Oil Spill Commission, the authority tasked by President Barack Obama to investigate the Gulf of Mexico disaster."
Three Mile Island, where the complaint was that there were too many alarms going off.
I mean, IT is always the irresponsible bad guy, right? It couldn't be someone else told them not to do it because it took too long, or was a waste of money, or...
Just another whitewash...
For justice, we must go to Don Corleone
Hm...lack of alarms...leading to a catastrophic engineering failure...where have I heard this story before...
Palm trees and 8
I don't even want to know how much tax payer money was pissed away for that "key verdict" - having worked with quite a few monitoring and alarm systems for years I can tell you that most of the time "automatic alarms" get ignored and in fact can cause worse problems when an actual real alarm does occur because of how the operators tune them out - seems like they completely missed the mark on this - the real problem is most likely where you would expect it, the people running the system - human error I am sure !
Things will always fail in weird, unexpected ways - that's why you need humans in the loop.
Is common practice everywhere "why buy a 5 dollar alarm when we can force some engineer to watch figures for days on end?" Gosh people hate engineers for no reason.
Where is the mod rating for "scary"? Also,
I wonder if the US government would go after it quite so much ? There does seem an attempt to play up the blame on BP and not the part played by Halliburton & others.
the power grid fault had a race bug that was fixed but the software update was not yet installed on that system.
As well the lack of tree timing and under trained people working the grid who did not know that other alarms where telling them.
When will we get a governing body that can punish or apply fines for this and enforce those fines or punishments...seriously, we need to evolve with these types of companies that spit all over international laws (or lack of)
like
1) someone have alarm systems available but noone wants to buy them.
2) and they saw the disaster as a good opportunity to sell more of them
3) and announcing that deepwater horizon lacked them sounds like a good business plan
4) just to guarantee that they will have customers for longer period of time
5) government is going to make them mandatory for any such operations
6)
7) profit
Haven't they been on Nagios Exchange recently? check_catastrophe.pl has been out for like 3 years!
check_catastrophy -H blowout-preventer716.haliburton.com -w ANY_LEAKS - c ANY_FRIGGIN_LEAKS
Lots of educated engineers, and this probably could have been fixed with a daemonized perl script that could send a trap to an snmp monitor if conditions got beyond a certain point. Or something like that. I'm sure they had more complex monitoring software, but obviously missed something simple along the way.
BP's monitoring IT systems on the failed Deepwater Horizon oil rig relied too heavily on engineers following complex data for long periods of time, instead of providing automatic warning alerts.
So, in other words, let's replace engineers who are on the spot and have some feel for what is going on with software that might not know what to do when something bad happens, and is dependent upon settings provided by people who apparently weren't able to recognize the signs of disaster until it was too late anyways. Regardless, I have the feeling there were plenty of alarm systems involved in this disaster, and I'll wager that the relevant ones were either incorrectly programmed or were turned off because they were inconvenient.
The higher the technology, the sharper that two-edged sword.
Actually, there were BPs in a redundant configuration but when the control was lost the main failed to operate and the backup's batteries were in too poor condition to work. As with most disasters there were a myriad of contributing factors. After looking at numerous reports (everyone is certainly trying to make sure their investigations are public) it looks like:
1. Familiarity breeds contempt. Alarms shut down or ignored partly because of annoyance and partly because incorrect conclusions were made about the state that the well was in, leading to a dangerous situation and disastrous consequences. Not unlike pilots in poor visibility conditions who ignore their instruments and distrust them leading to controlled flight into terrain.
2. Money trumps safety. There was tremendous corporate pressure to bring the well in. In the oil production world, almost everything is done by contract with petroleum producer owning and operating very little of what is going on. Rigs, crews, services are all contracted to do certain jobs and the competition is fierce. No one wants to be the company that could not do the task or who were late getting it done. Consider: if some different decisions were made and the well was brought in safely but say two or three months late and with several million more dollars spent, we would have never heard about anything and some of the well contractors, including individuals such as the rig boss, contract engineers, may have been looking for work elsewhere.
I'm interested to see if anything changes after all of the investigations, a la airline safety after a TSB investigation.
er... I meant "Blowout Preventers" for "BPs". Sorry for the confusion with British Petroleum.
I don't have a source. But CNN has coverage that engineers warned that the blowout preventers were going to leak, and BP ignored them. This is a corporate failure, as much as it is a technical one.
Does it seem a little wrong to call it an 'IT system'? Control system, SCADA, or embedded system maybe, but IT?
Transocean Gulf of Mexico Rig, leased to BP, lacked Alarm Systems
Eclectic beats from Leeds, UK
handmadehands.co.uk
They had this exact problem with Texas City-- they didn't do maintenance on the systems, so a subsystem overfilled with volatile hydrocarbons with no alarms going off at all-- and when one alert sounded at the monitoring area, they ignored it. They didn't invest the (relatively) small cost of installing a flare (to burn off excess), so the excess hydrocarbons spilled out into the open. Cost-cutting and an incredibly cavalier approach to maintenance from the London management generated a fucking fuel-air bomb in Texas.
This is one instance where the Brit management, when they changed to Hayward, should have told their investors to "fuck off-- er, give us a few years" and spend the necessary money to get their facilities up to snuff, or decommission the facilities that are too costly to maintain. Alas, profit motive proved more powerful than basic empathy or responsibility.
"We are Microsoft. You shall be assimilated. Competition is futile."
er... I meant "Blowout Preventers" for "BPs". Sorry for the confusion with British Petroleum.
Who are, of course, no longer actually called British Petroleum but just "BP", since the merger with American Oil (Amerco).
Eclectic beats from Leeds, UK
handmadehands.co.uk
Would that be the merged entity of British Petroleum and Amoco which is called BP?
sigh... so many "sheldon" moments
I meant, BP as in Blowout Preventers, as opposed to BP as in the company formerly known as British Petroleum.
Cheers
Here's a buzzword for you that is applicable in this situation: Crowdsourcing Put the internal paperwork the government has received on an easily accessible website (or wait for Wikileaks to do it). This spill received enough attention and affected enough lives that the general public would be interested to see what is going on and do there own investigations of the facts. There are plenty of professionals in manufacturing design and control systems design that would be more than willing to give some of their own time to read up on some of the facts. I would personally like to see the cause and effect tables and their alarm database for the rigs control system/SIS system. I would rather see for myself than take a journalist's word for it.
It was the Gentilly reactor in Quebec. Made the local papers (pre-internet).