BP Gulf of Mexico Rig Lacked Alarm Systems

As opposed to... by toejam13 · 2011-01-08 05:58 · Score: 4, Interesting

Three Mile Island, where the complaint was that there were too many alarms going off.

Re:As opposed to... by Gruturo · 2011-01-08 06:03 · Score: 4, Insightful

Three Mile Island, where the complaint was that there were too many alarms going off.
Yeah, surprisingly alarms have to be neither missing nor useless (by being irrelevant, hard to understand, going off for the wrong reasons, presenting wrong scenario, not correlating causes etc etc etc).
Who'd have thunk it.

--

Vacuum cleaners suck. Kings rule.
Re:As opposed to... by ColdWetDog · 2011-01-08 06:21 · Score: 4, Insightful

Truly amazing, indeed. Too lazy to look it up, but earlier reports had shown that Transocean (the rig owners, not BP like the stupid article mentions) had shut down many automatic warning systems because of too many false positives.

It's not like we've never seen this sort of thing before ...

"You are about to do something."

CANCEL, or ALLOW?

--
Faster! Faster! Faster would be better!
Re:As opposed to... by Anonymous Coward · 2011-01-08 06:33 · Score: 0, Interesting

Let's not forget that TMI was contained. The final safeguards system worked. In the BP spill it didn't. And the nuclear industry learned its lessons after TMI (as well as the NRC).
The oil industry needs to learn its lessons in the same way or else be regulated to the same extent as the nuclear industry. There really ought to be a NRC-like organization regulatory agency that ensures the safety of these rigs.
Re:As opposed to... by countertrolling · 2011-01-08 07:00 · Score: 1, Flamebait

In the BP spill it didn't.
Yes it did. Information, leakage of which would have caused much more damage to the company, was contained very effectively. Now, as far as they are concerned the matter is closed.

--
For justice, we must go to Don Corleone
Re:As opposed to... by noidentity · 2011-01-08 07:21 · Score: 1

Right. The monitoring systems should summarize things to a point that human operators can reliably understand the situation with a reasonable amount of detail, not too much or too little. If they don't do this, then they are badly designed.
Re:As opposed to... by $RANDOMLUSER · 2011-01-08 08:01 · Score: 1

When Three Mile Island happened (1979), 4Mhz Z80s were state-of-the-art. Industrial control systems have come a long, long, long way since then.

--
No folly is more costly than the folly of intolerant idealism. - Winston Churchill
Re:As opposed to... by DerekLyons · 2011-01-08 08:29 · Score: 1

I was thinking much the same thing... There probably is a happy medium, but it's going to be really really hard to hit.

Sitting a monitoring console hour after hour, day after day, is very tiring and wearing. So systems that monitor for trends and alert the operator are very valuable for cutting through that. But on the flip side, it becomes very easy to depend more and more on the automated systems and less and less on knowledge of the system, environment, and equipment. TAANSTAFL.

Disclaimer: Unlike many Slashdotters, I've actually sat roughly this kind of console, though mine was attached to strategic missiles rather than an oil well. And since I was aboard an SSBN rather than an oil rig, I've done the isolation bit as well.

Though the usual Slashdot chorus of Monday Morning Quarterbacks (read: "I'm a computer geek and thus an expert on everything, regardless of experience") will soon chime in, I'll state from actual experience that being such an operator isn't nearly as easy as you think it is.
Re:As opposed to... by ritcereal · 2011-01-08 08:38 · Score: 1

If you're responsible for keeping a system up and you have the ability to trigger alarms then you should! If you're generating false positive alarms then the solution is not to disregard or disable the alarms! You should be investigating every single alarm, documenting it and then fixing the reason why it happened. Alarms can be caused by a misconfiguration of the alarm system which is just as critical as a real alarm. If your fire alarm goes off every day would you fix it or would you start disregarding the alarm? Even more important, if you kept calling 911 screaming fire, would you expect the Firefighters to show up every time? I certainly would, however I wouldn't expect them to be happy about it. The moral of the story is, if your job includes alarms you should take full responsibility for them. Turning them off or ignoring them isn't being responsible no matter who you try to pass the blame to...it still is on your shoulders for not fixing the issue. Excuses don't mean shit at the end of the day. Three Mile Island was a disaster and the BP Oil Spill has caused an incredible amount of damage that we may never fully comprehend. In either case, alarms are critical to the safety of not only the employee's but the environment, citizens, the economy, and the company. Excuses that there were too many false positives just means that people needed to fix the false positives instead of ignoring or disabling them!
Re:As opposed to... by timeOday · 2011-01-08 08:38 · Score: 0

Easier said than done. Always triggering the right alarm - and only the right alarm - amounts to creating a system that somehow knows exactly how to handle any situation, no matter how complex. Otherwise it will certainly trigger alarms for secondary problems without knowing they have a common cause, or trigger alarms for problems that would normally be serious enough to warrant attention, but don't rise to that level due to dire circumstances.
And if you could make a perfect detector like that, you'd hardly need to alert people. You'd just automate the whole thing.
Re:As opposed to... by omglolbah · 2011-01-08 08:49 · Score: 5, Interesting

Indeed. Alarm suppression is a complex thing to set up in many cases. I personally work in the business and know how much thought goes into the alarm handling of the plants operating in Norwegian waters.
One example of a "simple" suppression case is that if Controller A goes down, you do not need to tell the operator that ALL signals on this controller is in "bad quality" or out of bounds. What you need to tell them is that the controller is down, and which systems are affected (which they will see on their displays as valves change color or somesuch. Our system uses white asterisks and white color to indicate that something is 'dead')
More complex cases are things like not throwing alarms for low flow rates in pipes where the valves are closed, or not throw electric alarms on equipment set to maintenance mode.
Regardless of all this, there should be an alarm system that has priorities.
Pri 1 alarms are such that they require IMMEDIATE attention. Such as a dangerous triple-high alarm (HHH or 3H) of a tank, pressure or temperature or a controller going down.
Pri 2 would be alarms that could develop into Pri 1 if not handled within a few minutes (H/HH) alarms etc.
Pri 3 would be what we call "pre-alarms". Things that could cause process upset or issues down the line. Like a low flow of coolant even though the temperature of the equipment being cooled hasnt started raising yet. Or a low level in a fuel tank.
Pri 4 we usually assign as maintenance issues. Like two redundant sensors having more than 0.5% deviation between them (But not enough to cause a real alarm). Things that should be looked at but within a day or so.
Being able to filter alarms like this helps immensely during an emergency. This is an old system with a limited number of 'alarm groups' and 'priority levels' but it still works fairly well. Operators can see what happens even with several hundred alarms going off at the same time. On our simulator we did a fun test where we tripped 70% of the plant (about 18000 distinct 'tags' or io points went into Bad quality and several thousand in alarm).
The operators were able to stop the cascade failure and no pipe burst in the simulator :)
Shit -will- hit the fan. It is always nice to be able to filter it so that only the important shit actually hits the wall :p
Re:As opposed to... by turbidostato · 2011-01-08 11:59 · Score: 2

"Easier said than done."
Of course, or there wouldn't be bussiness around it.
"Always triggering the right alarm - and only the right alarm - amounts to creating a system that somehow knows exactly how to handle any situation, no matter how complex."
Wrong. It amounts to getting rid of false asumptions or trying to sell a solution as the magic snake oil that will end all and every problem. Triggering the right alarm and only the right alarm is as easy as:
1) Known situation: manage automatically
2) Unknown situation: rise "human needed" alarm
3) Gather information among the "human needed" risen alarms to extract patterns so more and more situations can be managed by 1).
It only needs the guts from some manager to say "no, sir, we don't know that much about managing this kinds of situations, so it will need a lot of clever, well paid engineers that you will need to listen to as God Himself speech" instead of "Of course it'll be done, sir, in order for you to collect the bonuses".
Re:As opposed to... by turbidostato · 2011-01-08 12:14 · Score: 2

"Excuses that there were too many false positives just means that people needed to fix the false positives instead of ignoring or disabling them!"
While I'm with your overall message, you seem to forget that for this to work, bonuses and penalties need to be aligned; when they are not, things like this are expected to happen.
I.E: I certainly should care about each and every rised alarm, and I'm even told to do so. *But* I'm not payed to take care of rising alarms as soon as I can but to accomplish a different task (like bring up to production an oil rig) by the earliest date, while taking care of rising alarms gets in the middle. Since they both tasks are ireconcillable which one do you think a sensible person (or even a manager) should expect to suffer?
Re:As opposed to... by sirsnork · 2011-01-08 14:00 · Score: 1

Not to mention, you don't even have to have it being "live" until no. 2 items are into single digits per month. Just run it next to the existing systems and have someone monitor it until you're down to very low numbers for no 2

--

Normal people worry me!
Re:As opposed to... by arivanov · 2011-01-09 10:43 · Score: 1

Wrong analogy. BT is a UK company.
Read job adverts for this class of UK company IT architects on jobserve. They are _VERY_ explicit that the job of the architect is only to shop-n-ship. There is no allowance to collect reqs for a made-to-order job or spec-out an in-house system. If it is not supported by an off the shelf package it will not be. Period. The "We are not software developers" mantra taken to its ultimate limit.
My educated guess that the hodgepodge of systems delivered by 3 subcontractors for the drill had not off-the-shelf package supporting it and from there on it was policy.

--
Baker's Law: Misery no longer loves company. Nowadays it insists on it
http://www.sigsegv.cx/

Of course it is IT's fault by Anonymous Coward · 2011-01-08 06:00 · Score: 1

I mean, IT is always the irresponsible bad guy, right? It couldn't be someone else told them not to do it because it took too long, or was a waste of money, or...

Re:Of course it is IT's fault by nomadic · 2011-01-08 06:48 · Score: 1

I mean, IT is always the irresponsible bad guy, right?

On slashdot IT is never the bad guy. It's always some mythical manager who must have ordered them to do what they do. Why can nobody here ever believe a programmer/engineer/IT guy was incompetent?
Re:Of course it is IT's fault by hedwards · 2011-01-08 07:06 · Score: 1

Because even in that case, the problem originated higher up.

Why do they even bother? by countertrolling · 2011-01-08 06:08 · Score: 1

Just another whitewash...

--
For justice, we must go to Don Corleone

Re:Why do they even bother? by Anonymous Coward · 2011-01-08 06:15 · Score: 0

if you insist on "counter trolling" than why not back up your statement with facts, or hell, even elaborate on it some more to say why you feel that way... until then your just trolling yourself..... the only way to have a responsible sensible discussion of the issues at hand is with hard facts supporting the claims.... Until then your argument is just as dumb and unfounded as climate change deniers, ghost hunters, paranormal experts, and other such non-sense... FACTS.... cite raw data from a reliable source.... how did you come to the conclusion this is a whitewash? why should we believe you? if you dont feel that this information is important, than please dont share your opinion.
Re:Why do they even bother? by tomhudson · 2011-01-08 06:38 · Score: 5, Informative
Here's one fact - the regulators screwed up. Blaming it on a lack of alarms is disingenuous at best, corrupt at worst.
1. Regulators Failed to Address Risks in Oil Rig Fail-Safe Device
  http://www.nytimes.com/2010/06/21/us/21blowout.html?_r=1&pagewanted=all
2. Spill report: It could happen again
  'Failure of management' and regulators given blame for disaster
  http://www.chron.com/disp/story.mpl/business/7367856.html
3. Slick Operator
  How British oil giant BP used all the political muscle money can buy to fend off regulators and influence investigations into corporate neglect.
  http://www.newsweek.com/2010/05/07/slick-operator.html
This wasn't a technical failure - it was a failure brought out by greed and corruption. The blow-out was only the symptom, and addressing the symptom isn't going to prevent similar incidents from happening again.
We've seen this before - the mortgage disaster and bank bailouts, the savings and loan disaster, etc.
Start by fixing campaign financing - private donations only, strict annual limit per capita, no 3rd party involvement, etc.
-- Barbara
Re:Why do they even bother? by countertrolling · 2011-01-08 06:44 · Score: 1

Sorry, do your own googling. And study a little history*. Pick a period, any period. It doesn't matter. Over and over again the same things happen, damn near play by play, word for word. This here is just another example of the "thin blue line" that permeates.
*particularly, if you're up to it, that of BP, but the same deal applies to all human efforts

--
For justice, we must go to Don Corleone
Re:Why do they even bother? by nomadic · 2011-01-08 06:54 · Score: 2

Exactly; the private sector cannot be trusted to do things safer/more efficiently/better. This is exactly why strong government regulation, especially when it comes to environmental and health issues, is needed.
Re:Why do they even bother? by ScrewMaster · 2011-01-08 07:09 · Score: 1

Sorry, do your own googling. And study a little history*. Pick a period, any period. It doesn't matter. Over and over again the same things happen, damn near play by play, word for word.

You mean like this

--
The higher the technology, the sharper that two-edged sword.
Re:Why do they even bother? by omglolbah · 2011-01-08 08:53 · Score: 3, Insightful

Have a peek at the Norwegian sector. We've been doing this shit since the 70s and try damn hard to not have another Alexander Kielland...
http://en.wikipedia.org/wiki/Alexander_L._Kielland_(platform)
The norwegian petroleum oversight is something... The regulators are ruthless when it comes to compliance and better yet... they are not directly controlled by politicians ;)
The cost of one fuckup is too much to allow people to cut corners.
I sure as hell dont in my job... and I do it for a living. When we have the option of doing it right, or doing it fast.. we pick right. Every time. I dont care if the customer is pissed at things being delayed. We do it -right-.
Re:Why do they even bother? by turbidostato · 2011-01-08 12:22 · Score: 0

"Exactly; the private sector cannot be trusted to do things safer/more efficiently/better."
Quite on the contrary, you can expect the private sector to do things certainly more efficiently and better, once you understand what's the proper definition of "better" within context. In fact, that's all you can trust the private sector to come with.
Regarding "safer", just apply my previous paragraph: to which extent can "safer" be derived of, or translated into, "more efficiently and better within context"? That's the safety level you can expect, no less no more.
Re:Why do they even bother? by rtb61 · 2011-01-08 14:20 · Score: 1

It also appears that the major players involved were already pretty aware of the risk of that particular well, including a vary favourable insurance policy by Transocean (no cost cutting here), buying a oil clean up company by Halliburton and many insiders selling BP stock.
No type of warning systems could have saved this particular rig from a major gas blowout with only one spark required to ignite it. It could have however saved the crew if they could have abandoned the rig prior to ignition.
Greed feeding stupidity, driving too many cost cutting, bonus boosting short cuts, all protected by failing regulatory and inspection standards. In this it is pretty clear that quite a few corporate executives were fully aware of the risks and decided to cash in on them rather than fix the problem. So the investigation should also cover, who was aware of the problems with the rig and what actions did they take based upon that awareness (did they seek to profit or did they seek to minimise harm) and as such as accessories prior to the fact how liable are they for the consequences.

--
Chaos - everything, everywhere, everywhen
Re:Why do they even bother? by mrmeval · 2011-01-09 14:14 · Score: 1

Earlier I saw this and will attribute it to the author with a link http://news.slashdot.org/comments.pl?sid=1942186&cid=34806036
That definition matches this case. The regulatory agency should be shattered and it's personnel dismissed without benefit or pension and disbarred from any government service. Those directly responsible charged with what ever crime is possible.
Then the agency should be remade with some checks in it's ability to wave any regulatory requirement.

--
I'd go on a Vegan diet but the delivery time from Vega is too long. --brownkitty

Seems a little unrelated by Anonymous Coward · 2011-01-08 06:10 · Score: 0

TMI was caused by the mechanical failure of a cooling pump, and user error caused by the myriad of safety systems confusing engineers about just what they have to do.

BP was caused by absolutely no safety systems whatsoever being installed or the ones existing being ignored so BP execs could have a slightly larger christmas bonus.

Re:Seems a little unrelated by tomhudson · 2011-01-08 06:27 · Score: 3, Interesting

And there was another near-disaster because at one nuke plant, the button you had to press was back-lit by a bulb that, over time, had caused the plastic to expand to the point that the button COULDN'T be pressed - which they found out the hard way.
Things will always fail in weird, unexpected ways - that's why you need humans in the loop.
Re:Seems a little unrelated by dogsbreath · 2011-01-08 07:21 · Score: 2

Actually, there were BPs in a redundant configuration but when the control was lost the main failed to operate and the backup's batteries were in too poor condition to work. As with most disasters there were a myriad of contributing factors. After looking at numerous reports (everyone is certainly trying to make sure their investigations are public) it looks like:
1. Familiarity breeds contempt. Alarms shut down or ignored partly because of annoyance and partly because incorrect conclusions were made about the state that the well was in, leading to a dangerous situation and disastrous consequences. Not unlike pilots in poor visibility conditions who ignore their instruments and distrust them leading to controlled flight into terrain.
2. Money trumps safety. There was tremendous corporate pressure to bring the well in. In the oil production world, almost everything is done by contract with petroleum producer owning and operating very little of what is going on. Rigs, crews, services are all contracted to do certain jobs and the competition is fierce. No one wants to be the company that could not do the task or who were late getting it done. Consider: if some different decisions were made and the well was brought in safely but say two or three months late and with several million more dollars spent, we would have never heard about anything and some of the well contractors, including individuals such as the rig boss, contract engineers, may have been looking for work elsewhere.
I'm interested to see if anything changes after all of the investigations, a la airline safety after a TSB investigation.
Re:Seems a little unrelated by dogsbreath · 2011-01-08 07:24 · Score: 1

er... I meant "Blowout Preventers" for "BPs". Sorry for the confusion with British Petroleum.
Re:Seems a little unrelated by AGMW · 2011-01-08 22:11 · Score: 1

er... I meant "Blowout Preventers" for "BPs". Sorry for the confusion with British Petroleum.
Who are, of course, no longer actually called British Petroleum but just "BP", since the merger with American Oil (Amerco).

--
Eclectic beats from Leeds, UK
handmadehands.co.uk
Re:Seems a little unrelated by Peil · 2011-01-09 00:24 · Score: 1

Would that be the merged entity of British Petroleum and Amoco which is called BP?
Re:Seems a little unrelated by dogsbreath · 2011-01-09 07:08 · Score: 1

sigh... so many "sheldon" moments
I meant, BP as in Blowout Preventers, as opposed to BP as in the company formerly known as British Petroleum.
Cheers
Re:Seems a little unrelated by Anonymous Coward · 2011-01-10 00:46 · Score: 0

Source? Because this sounds like FUD, at least by western nuclear standards. Any push button is part of human operations. Humans are in the loop to keep a plant operating, not to keep a plant safe. That means that failure to push the button could lead to an unplanned shutdown, but not to a meltdown.
Or to reword your conclusion: Humans will always fail in weird, unexpected ways - that's why you remove humans from the loop.
Re:Seems a little unrelated by tomhudson · 2011-01-10 02:10 · Score: 1

It was the Gentilly reactor in Quebec. Made the local papers (pre-internet).

Cost cutting by Anonymous Coward · 2011-01-08 06:17 · Score: 0

I wouldn't be surprised if BP's CIO boasted about saving millions USD by consolidating data centers and retiring hundreds of "redundant" applications before the blast occurred, some of which might have saved everyone's bacon down there.

Sounds familiar... by betterunixthanunix · 2011-01-08 06:23 · Score: 1

Hm...lack of alarms...leading to a catastrophic engineering failure...where have I heard this story before...

--
Palm trees and 8

Re:Sounds familiar... by ae1294 · 2011-01-08 06:27 · Score: 1

Hm...lack of alarms...leading to a catastrophic engineering failure...where have I heard this story before...
Ummm the Internet?

how much did that cost by Dolphinzilla · 2011-01-08 06:24 · Score: 1

I don't even want to know how much tax payer money was pissed away for that "key verdict" - having worked with quite a few monitoring and alarm systems for years I can tell you that most of the time "automatic alarms" get ignored and in fact can cause worse problems when an actual real alarm does occur because of how the operators tune them out - seems like they completely missed the mark on this - the real problem is most likely where you would expect it, the people running the system - human error I am sure !

Re:how much did that cost by DrugCheese · 2011-01-08 06:52 · Score: 1

I don't even want to know how much tax payer money was pissed away for that "key verdict" - having worked with quite a few monitoring and alarm systems for years I can tell you that most of the time "automatic alarms" get ignored and in fact can cause worse problems when an actual real alarm does occur because of how the operators tune them out - seems like they completely missed the mark on this - the real problem is most likely where you would expect it, the people running the system - human error I am sure !
I think everyone's familiar of that phenomenon regarding the alarm that cried wolf due to all the car alarms. Rarely do people even turn their head when they hear a car alarm.
I think I'm gonna make a "Let's blame IT!" t-shirt cause it's pretty popular theme. Seems to me that the hardware for detecting the problems was there, but the software required "the right person to be looking at the right data at the right time" which sounds vaguely like "the software requires training". If the data output is coming at you like a Matrix display and it is way too complicated to train people to reliably do their job then you upgrade the software. You can't put blame on inanimate objects for things like this. If a tree falls on your house, you can blame the tree, the wind, your god - but you can't blame a machine for not being built right. Especially not when a human has already admitted to stepping "away from his safety monitor for a brief cigarette break, and that at this moment he missed the complex warning signs on screen."
So the software/hardware is at fault for not reminding the guy audibly to do his job instead of smoke a cigarette. Waste of taxpayer sweat.

--
*DrugCheese rants*
Re:how much did that cost by hedwards · 2011-01-08 07:12 · Score: 3, Insightful

I think everyone's familiar of that phenomenon regarding the alarm that cried wolf due to all the car alarms. Rarely do people even turn their head when they hear a car alarm.
Competent professionals don't do that. The problem with car alarms is that they aren't aimed at professionals, competent or otherwise, they're aimed at the general public and the mechanism they use isn't typically going to assure that anything is going on.

Competent professionals like the ones that are supposed to be running rigs should know to check them out every time and not turn the alarm off withotu ascertaining that the alarm is in fact false. Disabling an alarm should only be done when there are adequate contingency plans in place to handle if the condition happened and how they would respond.

I used to work security at a high rise and we'd often times have alarms turned off on portions of the building. It was the only way to ensure that under certain circumstances that work wouldn't cause a false alarm. It was done in a controlled way with plans in place to make sure that there was somebody keeping an eye on it while the work was being done, and that the alarms would be turned back on when they could be.

And every time that building had an alarm go off which wasn't a known cause, it was always investigated promptly. Alarms that go off repeatedly need to be fixed, not disabled.
Re:how much did that cost by Rob+the+Bold · 2011-01-08 07:12 · Score: 4, Insightful

I don't even want to know how much tax payer money was pissed away for that "key verdict" - having worked with quite a few monitoring and alarm systems for years I can tell you that most of the time "automatic alarms" get ignored and in fact can cause worse problems when an actual real alarm does occur because of how the operators tune them out - seems like they completely missed the mark on this - the real problem is most likely where you would expect it, the people running the system - human error I am sure !
You don't even have to ignore the alarm that isn't there. But I don't think the "alert" that we're discussing is the big klaxon/flashing sign reading "OIL LEAK," or an oil pressure light with electrical tape over it. What the article indicates was missing was an automatic method of indicating that a failure was imminent. As far as the cost of determining this: learning from mistakes can be expensive. Not learning from mistakes is likely even more so.

--
I am not a crackpot.
Re:how much did that cost by DMiax · 2011-01-08 07:34 · Score: 1

It seems to me that the verdict states that the probability and possible damages of a human error were too high, due to poor planning of safety features. I don't think it is really much different from what you say.
Re:how much did that cost by thegarbz · 2011-01-09 22:59 · Score: 2

The problem is who is the competent professional who is working on alarms?

Is it the maintenance team who is backlogged with bullshit alarms that go off under normal process conditions because someone decided that it would work to prevent some disaster which may occur?
Is it the process / technical team who decided yet another alarm will be cheaper than re-designing the process to meet the safety guidelines?
Is it the console operator who has gone mental at the alarm going off constantly in the middle of the night and has requested the bypass?
Is it the control engineer who has approved the bypass for the same reasons without a process safety review?

Ask different people on what they do with alarms and you'll get different answers even within the same discipline. We have two process safety engineers at our refinery with two distinctly differing opinions. The one thinks advanced warning is god and requests a process alarm be put on everything, and that every instrument becomes a layer of protection. The other wishes that this was 1960 where signals were pneumatic and adding an alarm to the operations console cost a frigging fortune, because back then we had only sane and highly critical alarms.

The former is winning in my opinion. An alarm goes off in our control room every 2 minutes. There is a list of standing alarms for each area on each console operator's screen, and some alarms even go unacknowledged. Many more are bypassed. But all for what? In reality when something goes wrong the operator screen is flooded with priority 1 critical alarms and the operator can acknowledge maybe 1 or 2 before they just start playing on instinct and training to bring things under control.

If you get to the stage where you are relying on an alarm you have lost. Relying on operator intervention for process safety is the absolute last resort.

Apparently abusing engineers by The+Hatchet · 2011-01-08 06:35 · Score: 1

Is common practice everywhere "why buy a 5 dollar alarm when we can force some engineer to watch figures for days on end?" Gosh people hate engineers for no reason.

--
Where is the mod rating for "scary"? Also, ...

Re:Apparently abusing engineers by the_fat_kid · 2011-01-08 07:15 · Score: 1

do you mean "why buy a $5 alarm when we could pay an engineer thousands of dollars a year to do the same thing?"
I have a phrase that you should practice: "would you like fries with that?"
and "paper or plastic, sir?"
very good now again, in Chinese.
gosh people PAY engineers for no reason...

--
-- Sig under construction...
Re:Apparently abusing engineers by omglolbah · 2011-01-08 08:55 · Score: 2

Unfortunately, a single alarm configuration on a "tag" could cost anywhere from 10k to 100k dollars.
The configuration isnt all that hard or time consuming but the testing of the system after the modification is brutal. At least here where it has to be certified to be allowed into operation ;)
Re:Apparently abusing engineers by Anonymous Coward · 2011-01-08 10:10 · Score: 0

But I thought engineers liked watching figures for days on end. That's all I ever see them doing in movies.
Re:Apparently abusing engineers by The+Hatchet · 2011-01-08 17:34 · Score: 1

If it costs that much, you are doing it wrong. A good engineering team should be able to make something work very well for only a few hundred to a few thousand dollars.

--
Where is the mod rating for "scary"? Also, ...
Re:Apparently abusing engineers by omglolbah · 2011-01-09 12:35 · Score: 2

Doing the change: 3-4 hours of work.
Organizing the update to the controller in the field?
- Requires a look into what could be influenced by the change
- Requires in some cases an 'offline' load of the controller which can only be done at a time of a maintenance downtime (once a year at most, sometimes every 2-4 years)
Documentation:
- Documentation of what functionality changes for operators
- Update of system configuration diagrams
- Update of various tag info in the plant documentation system
Install:
- A job package must be written detailing every change made to the system.
- A test package must be written with a full test suite to check that nothing broke during the change. People make mistakes and this is important.
Now... How much will all this cost?
When I'm working on jobs like this the company I work for charges about 170 bucks an hour...
4*3 hours (The change, verification and signoff, various overhead)
5*2 hours (Field work, included travel time etc, x2 for 2 people)
8*3 hours (documentation, x3 due to document controllers, various overhead)
6*2 hours (job/test package)
5*5 hours (testing)
83 hours, 170 bucks an hour, 14110 USD.
This is a fairly average estimate of what something would cost on -our- side of a very small change. If hardware is involved it rapidly skyrockets in cost.
In addition there is a myriad of people that need to check and verify the change on the -other- side of the fence. Namely the owner and/or operator of the plant.
All these time-consuming road-blocks put in place are barriers against making changes that could breach safety. They look arcane and silly to quite a lot of people but they are there for a reason.
Most of the accidents where I work happen when someone do a quick tiny change. One that "wont cause any issues" except that it turns out it does.
To see why small changes can have huge impacts have a look at this book: http://www.amazon.com/What-Went-Wrong-Histories-Disasters/dp/0884150275
I realize it would be horribly boring reading for anyone not interested in it :p

If BP were a US company ... by Alain+Williams · 2011-01-08 06:42 · Score: 1

I wonder if the US government would go after it quite so much ? There does seem an attempt to play up the blame on BP and not the part played by Halliburton & others.

Re:If BP were a US company ... by Anonymous Coward · 2011-01-08 07:00 · Score: 1

Two names: Exxon Valdez.
There was a huge shit storm when they fucked up
So to answer your question, yes.
BP were the boys in charge and when it comes down to it, it was up to them to keep Haliburton et al. in line, so it was there fault. And it was also the regulators fault for dropping the ball and letting a big corp make them their bitches; which is usually the case with all US Government agencies.
Re:If BP were a US company ... by Anonymous Coward · 2011-01-08 09:21 · Score: 0

Just remind me how much the fine for the Exxon Valdex was again and when they paid it?
If we're talking about companies crapping up on foreign soil (and let's also not forget that BP is listed on the US stock exchange and has more employees and pension recipients in the US than in the UK) how about we look at Union Carbide..................
Re:If BP were a US company ... by AGMW · 2011-01-08 22:19 · Score: 1

... BP were the boys in charge and when it comes down to it, it was up to them to keep Haliburton et al. in line, so it was there responsibility. ...
Fixed that for you ...

--
Eclectic beats from Leeds, UK
handmadehands.co.uk

the power grid fault had a race bug that was fixed by Joe+The+Dragon · 2011-01-08 06:43 · Score: 1

the power grid fault had a race bug that was fixed but the software update was not yet installed on that system.

As well the lack of tree timing and under trained people working the grid who did not know that other alarms where telling them.

When r they getting theirs? by hesaigo999ca · 2011-01-08 06:45 · Score: 2

When will we get a governing body that can punish or apply fines for this and enforce those fines or punishments...seriously, we need to evolve with these types of companies that spit all over international laws (or lack of)

Re:When r they getting theirs? by ScrewMaster · 2011-01-08 07:19 · Score: 2

When will we get a governing body that can punish or apply fines for this and enforce those fines or punishments
Two words: regulatory capture.

--
The higher the technology, the sharper that two-edged sword.
Re:When r they getting theirs? by stewski · 2011-01-08 09:13 · Score: 1

I wonder when such investigations will occur in areas where Americans aren't affected? How is the behaviour of companies such as Exxon in the Niger delta being tracked, oh wait it isn't. Still that doesn't matter, because it doesn't affect fat American business men!
Re:When r they getting theirs? by ScrewMaster · 2011-01-08 09:57 · Score: 1

I wonder when such investigations will occur in areas where Americans aren't affected? How is the behaviour of companies such as Exxon in the Niger delta being tracked, oh wait it isn't. Still that doesn't matter, because it doesn't affect fat American business men!
That's just silly. If a foreign corporation is allowed to do business in your country, it is your government that should perform due diligence and make sure that said corporation is obeying local regulations. If it doesn't, then it should take appropriate action, whatever that might be.

--
The higher the technology, the sharper that two-edged sword.
Re:When r they getting theirs? by stewski · 2011-01-08 10:35 · Score: 1

So if an individual (which a corporation is legally termed) behaves objectionably abroad it is no business of the government from which the individual came from? Don't get me wrong "when in Rome" and all that is fine. But how would the US government react to a US corporation working in north Korea on weapons development, I mean all the work would be obey local regulations...
Your suggestion suggests a level of naivety that I would categorise as in-genuine; to the point of drawing parallels to three monkeys covering their eyes, ears and mouth....
Re:When r they getting theirs? by hesaigo999ca · 2011-01-11 02:29 · Score: 1

So many good points, I would hate to bring it to an end, but I believe that there should be a one track international sanction that needs to be followed in matters that affect environment in such a way that it could affect other nations indirectly (like this spill)....and that governing body should be forceable enough to make all think twice, (like the US bypassing the nato sanction not to invade, sort of like we heard you but dont care and will still do this....) can you imagine if they could actually come up with one that strong....if china , india, us, brits and french etc... all joined together to form a body that actually has that power, you could enforce such rules...you dont follow the rules, then we take it over and you lose it for good....becomes property of xxx organization...
who would bp oil go running to complain, "hey they stole our platform"....the people you want to complain to are the ones enforcing it....a nice catch 22 if you ask me.

Stoopid americans by Anonymous Coward · 2011-01-08 06:52 · Score: 0

The post is wrong. The rig did not belong to BP, it belonged to transocean. BPs oil, transoceans rig

This newsstory sounds... by tp_xyzzy · 2011-01-08 06:52 · Score: 1

like
1) someone have alarm systems available but noone wants to buy them.
2) and they saw the disaster as a good opportunity to sell more of them
3) and announcing that deepwater horizon lacked them sounds like a good business plan
4) just to guarantee that they will have customers for longer period of time
5) government is going to make them mandatory for any such operations
6)
7) profit

Nagios by IceCreamGuy · 2011-01-08 07:02 · Score: 3, Funny

Haven't they been on Nagios Exchange recently? check_catastrophe.pl has been out for like 3 years!

check_catastrophy -H blowout-preventer716.haliburton.com -w ANY_LEAKS - c ANY_FRIGGIN_LEAKS

A perl script? by digsbo · 2011-01-08 07:02 · Score: 1

Lots of educated engineers, and this probably could have been fixed with a daemonized perl script that could send a trap to an snmp monitor if conditions got beyond a certain point. Or something like that. I'm sure they had more complex monitoring software, but obviously missed something simple along the way.

Re:A perl script? by hedwards · 2011-01-08 07:16 · Score: 1

Bad idea, the issue wasn't that the alarms were broken so much as they were ignored for going off too frequently. And rather than address the issue of the frequent occurrences they opted to shut them off. It's unlikely that you're going to solve that by programming around that. Programming around it is more or less the same thing as turning the alarms off or ignoring them.
Re:A perl script? by digsbo · 2011-01-08 08:13 · Score: 1

Operator: "Disk alarm - disk is at 80% capacity."
Manager: "Increase the threshold to 90%."
Re:A perl script? by omglolbah · 2011-01-08 08:58 · Score: 2

Operator: "I cant do that, that has to be run through the PCDA office and certified by the technical staff first."
Manager: "Ok, I'll submit the paperwork"
PCDA: "This is a bad idea, lets fix it instead..."
Or something like that is how it goes here :p
If it even passes the manager. Most of the time the technical staff handles the alarms without telling any 'manager'. The operator responsible for the shift has authority over the day to day operation without any manager interference.
You cant operate if non-techies have more control than the techies over tech questions. It has been tried and abandoned ;)

Automation not always "better" ... by ScrewMaster · 2011-01-08 07:18 · Score: 1

BP's monitoring IT systems on the failed Deepwater Horizon oil rig relied too heavily on engineers following complex data for long periods of time, instead of providing automatic warning alerts.

So, in other words, let's replace engineers who are on the spot and have some feel for what is going on with software that might not know what to do when something bad happens, and is dependent upon settings provided by people who apparently weren't able to recognize the signs of disaster until it was too late anyways. Regardless, I have the feeling there were plenty of alarm systems involved in this disaster, and I'll wager that the relevant ones were either incorrectly programmed or were turned off because they were inconvenient.

--
The higher the technology, the sharper that two-edged sword.

Re:Automation not always "better" ... by confused+one · 2011-01-08 08:33 · Score: 1

Don't replace the engineer. Give them tools that enhance their ability to see impending problems and predict the output of the system in it's current state. However, even given the best tools, if someone chooses to ignore the warnings and over-ride the automation then "accidents" will happen.
Re:Automation not always "better" ... by ScrewMaster · 2011-01-08 08:37 · Score: 1

Don't replace the engineer
I wasn't saying that, but it looks like that report is just another example of blaming the technical people for systemic failures of management.

Typical. Absolutely typical.

--
The higher the technology, the sharper that two-edged sword.
Re:Automation not always "better" ... by omglolbah · 2011-01-08 09:04 · Score: 1, Insightful

It all comes down to redundant barriers. A B C 1 ->-0-->--| | | | 0 2 ->-0 | | 3 ->-0-->--0-->--| | | 0 A, B and C are various barriers. A = Automation (automatic shutdown on severe alarms etc) B = Procedures (Check X before doing Y) C = Operator Training As you can see here an accident can only happen if -all- the barriers fail. One is enough to stop the incident. That is the theory anyway. We dont want to replace anyone but we -do- want to add more barriers! :)
Re:Automation not always "better" ... by turbidostato · 2011-01-08 12:32 · Score: 1

" We dont want to replace anyone but we -do- want to add more barriers!"
Who is "we"? For all that matters, the manager is not part of "we": all he wants is his bonuses.
Re:Automation not always "better" ... by omglolbah · 2011-01-08 17:08 · Score: 1

I'm sadly not allowed to disclose the company name due to an NDA, but it is one of the largest in norther europe.
At the particular company where I work we fucking HATE the shoddy work and failed procedures of this disaster. It makes us all look like asshats.
The people in charge of the technical things here are actually not the people who are trying to get bonuses. The government oversight on the security of such sites and rigs is so strong as to be borderline anal. And personally I am fine with that. I would hate if some fuckup in our code caused a disaster.
Re:Automation not always "better" ... by turbidostato · 2011-01-12 10:41 · Score: 1

"The people in charge of the technical things here are actually not the people who are trying to get bonuses."
That's why I asked for your definition of "we". Of course the engineers dislike appearing like asshats.
"The government oversight on the security of such sites"
So being a representative democracy, I'd say goverment is the kind of "we" to be in control in managing such externalities instead of "we", the high managers that get the bonuses.
Of course, your government is one of those damn communist ones, with their damn socialized social security, their damn socialized education and their damn regulations agains "we", poor international corporations, so you must be wrong, somehow you must be wrong.

corporate failure by Anonymous Coward · 2011-01-08 07:51 · Score: 1

I don't have a source. But CNN has coverage that engineers warned that the blowout preventers were going to leak, and BP ignored them. This is a corporate failure, as much as it is a technical one.

Re:corporate failure by omglolbah · 2011-01-08 09:05 · Score: 1

Yep.
Both are bad.. Together they are absolutely cataclysmic.
Complete failure of barriers here. Have a gander at my other comment about the idea behind those barriers.
http://news.slashdot.org/comments.pl?sid=1942186&cid=34807134
Re:corporate failure by AGMW · 2011-01-08 22:25 · Score: 1

I don't have a source. But CNN has coverage that engineers warned that the blowout preventers were going to leak, and BP ignored them. This is a corporate failure, as much as it is a technical one.
I certainly saw engineers from Transocean, or was it Halliburton, saying something like that. Luckily we can obviously trust those engineers because they (and the company they work for) has nothing to gain from saying it.
Of course, it could be argued that if those engineers, who presumably worked for Transocean (who owned and operated the rig) knew there was a problem and did nothing about it then they, and the company they work for, are left holding the smoking gun!
Unless we allow the "ve vere only following orders" defence these days!

--
Eclectic beats from Leeds, UK
handmadehands.co.uk

Why call it IT? by cherry-blossom · 2011-01-08 09:05 · Score: 1

Does it seem a little wrong to call it an 'IT system'? Control system, SCADA, or embedded system maybe, but IT?

Re:Why call it IT? by turbidostato · 2011-01-08 12:34 · Score: 1

Does it seem a little wrong to call it an 'IT system'? Control system, SCADA, or embedded system maybe, but "IT?"
Was not Information moving around? Was not that Information moving around by Technical means?
Automatic control systems are IT, Supervisory Control And Data Acquisition systems are IT, signaling embedded systems are IT.

I know BP leased the rig, but come on by AGMW · 2011-01-08 09:31 · Score: 4, Interesting

it was Transocean that owned and operated the rig?, so perhaps the story could better be titled:-

Transocean Gulf of Mexico Rig, leased to BP, lacked Alarm Systems

--
Eclectic beats from Leeds, UK
handmadehands.co.uk

Re:I know BP leased the rig, but come on by geekbrad · 2011-01-09 00:20 · Score: 1

Technically, as in "what does the paperwork say", of course, you're right. Though the Deepwater Horizon had drilled under lease to BP since it was built - before Transocean was even involved. Your headline makes it sound like BP just borrowed a screwdriver from them, rather than having had exclusive use of this rig since inception.

This means they learned nothing by magus_melchior · 2011-01-08 09:47 · Score: 4, Interesting

They had this exact problem with Texas City-- they didn't do maintenance on the systems, so a subsystem overfilled with volatile hydrocarbons with no alarms going off at all-- and when one alert sounded at the monitoring area, they ignored it. They didn't invest the (relatively) small cost of installing a flare (to burn off excess), so the excess hydrocarbons spilled out into the open. Cost-cutting and an incredibly cavalier approach to maintenance from the London management generated a fucking fuel-air bomb in Texas.

This is one instance where the Brit management, when they changed to Hayward, should have told their investors to "fuck off-- er, give us a few years" and spend the necessary money to get their facilities up to snuff, or decommission the facilities that are too costly to maintain. Alas, profit motive proved more powerful than basic empathy or responsibility.

--
"We are Microsoft. You shall be assimilated. Competition is futile."

Re:This means they learned nothing by thegarbz · 2011-01-09 23:16 · Score: 1

Different problem different situation. An alarm is only valued if it's actioned. The problems at Texas were as you put it cavalier approaches, but not to maintenance, to everything. Someone's too man enough to follow the instructions and instead rely on instinct to guide them. What use is a high level alarm on a fractionation tower when operators will routinely and against procedure start up the unit with the level instrument overfilled. When starting up an average unit there can literally be hundreds of alarms sounding throughout the day as alarms are typically set for stable operating conditions rather than startup / shutdown. The vast majority of them get ignored because making noise is just what the alarm panels do when a unit is started up.

Hate to break it to you but any partially complicated refinery is a fuel-air bomb. Literally some units are pure oxygen sitting right underneath hydrocarbon above auto-ignition point with only a differential pressure preventing one leaking into the other. The question is always what process safety initiatives are designed in the unit. I know one refinery where process reversals such as the above are common place throughout the year. I know another refinery with an emergency shutdown system that puts nuclear reactors to shame which will in the event of an issue trigger a fully automated shutdown of the unit completely overruling operator control.

By the way, as someone who has invested in the said company, perhaps you should take a look at their books before you act like you know what decisions went on. There was never a shortage of money for safety related projects post the Texas city incident.

This isn't even close to coming as interesting... by Anonymous Coward · 2011-01-08 13:54 · Score: 0

...as this.
http://phoenixrisingfromthegulf.wordpress.com/

the fuck? by Anonymous Coward · 2011-01-09 07:29 · Score: 0

Okay super cool fucking story, bro. If there are no alarms then it means the engineers are the ones responsible. If there were suppose to be alarms but weren't, then it's BP's fault for not making sure they were using the rig correctly.

Make the investigation public by pandymen · 2011-01-09 10:28 · Score: 1

Here's a buzzword for you that is applicable in this situation: Crowdsourcing Put the internal paperwork the government has received on an easily accessible website (or wait for Wikileaks to do it). This spill received enough attention and affected enough lives that the general public would be interested to see what is going on and do there own investigations of the facts. There are plenty of professionals in manufacturing design and control systems design that would be more than willing to give some of their own time to read up on some of the facts. I would personally like to see the cause and effect tables and their alarm database for the rigs control system/SIS system. I would rather see for myself than take a journalist's word for it.

Slashdot Mirror

BP Gulf of Mexico Rig Lacked Alarm Systems

92 comments