BSOD Issues On Deepwater Horizon

ctdownunder passes along this excerpt from a NY Times article about a rig worker's testimony concerning the April 20 accident at the Deepwater Horizon well: "The emergency alarm on the Deepwater Horizon was not fully activated on the day the oil rig caught fire and exploded, triggering the massive spill in the Gulf of Mexico, a rig worker on Friday told a government panel investigating the accident. ... On Friday, Mr. Williams added several new details about the equipment on the vessel, testifying that another Transocean official turned a critical system for removing dangerous gas from the drilling shack to 'bypass mode.' When he questioned that decision, Mr. Williams said, he was reprimanded. ... Problems existed from the beginning of drilling the well, Mr. Williams said. For months, the computer system had been locking up, producing what the crew deemed the 'blue screen of death.' 'It would just turn blue,' he said. 'You’d have no data coming through.' Replacement hardware had been ordered but not yet installed by the time of the disaster, he said." The article doesn't mention whether it was specifically a Windows BSOD, or just an error screen that happened to be blue.

Comments

They didn't fix a lot of things

[18_Rabbit]

For example, they KNEW that the BOP (blowout preventer) was not functioning correctly. one of the 2 control systems was out, and they had been bringing up pieces of the rubber seal in the test fluid. They were cutting corners on their cut corners. You'd think this would serve as exhibit A to silence all the "GOVERNMENT R BAD, CORPORATIONS R GOOD" nutcases in the USA today, but unfortunately it does not seem to have had that effect.

Re:They didn't fix a lot of things

[Anonymous Coward]

the government's responses to national crises like this one should also tell you that those "GOVERNMENT IS GOOD, DOWN WITH CORPS" nutcases in the usa should also be silenced.

How about down with self-serving bureaucracy? you know, the kind that insulates its ideology from reality so much that everyone else is left holding the resulting inevitable calamity.

Re:They didn't fix a lot of things

[Attila Dimedici]

You mean because the regulators did such a wonderful job at enforcing the regulations that were already in place that we should create new regulations?
I am never a fan of government regulations, but when there are problems with an industry we can discuss possible government regulations of that industry. However, I am always opposed to new regulations to address a problem that appears to have happened largely because exisitng regulations were not being followed. If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

Re:They didn't fix a lot of things

[amorsen]

If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

The regulators were tasked to check that the companies followed the procedures for checking their own operations. This kind of twice-removed oversight is becoming increasingly common in lots of places, because it saves money for the government (popular with voters) as well as being popular in the private sector (for obvious reasons).

It works great as long as companies are overall honest and all their problems are caused by simple negligence. It doesn't work so well in the face of outright fraud.

Re:They didn't fix a lot of things

[T.E.D.]

The regulators were tasked to check that the companies followed the procedures for checking their own operations. This kind of twice-removed oversight is becoming increasingly common in lots of places, because it saves money for the government (popular with voters) as well as being popular in the private sector (for obvious reasons). It works great as long as companies are overall honest and all their problems are caused by simple negligence. It doesn't work so well in the face of outright fraud.

It doesn't work period. Anybody who understands economics (as "fiscal conservatives" claim they do) should understand that you can't expect entities to act contrary to the incentives around them out of a sense of civic duty or something. When you set up a system where there are tremendous financial incentives to cheat, the chances are getting caught are almost nil, and even then the punishments will be laughable compared to the money saved by cheating, it would be insane to not expect things like this to happen.

The only way to prevent reoccurances is to change the system. That will require changing the regulations.

Re:They didn't fix a lot of things

[camperdave]

I am never a fan of government regulations

Government regulations are what keep you from dying every time you make toast, plug in the kettle, or turn on the TV. They keep you safe on the roads. They stop your house from falling in, from toxic chemicals being found in your food, and thousands upon thousands of other hazards that every day life throws at you.

Re:They didn't fix a lot of things

[Chris Burke]

However, I am always opposed to new regulations to address a problem that appears to have happened largely because exisitng regulations were not being followed. If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

Because the lack of regulations, and the lack of regulatory enforcement, are closely related phenomenon. Both are the consequence of a government who, like you, viewed government regulation as a bad thing, and felt that industry was best left to itself. So they relaxed the regulations. And while they didn't succeed in getting rid of all regulations, that same philosophy carried over into the hiring and management of the regulatory body -- that the regulations they were supposed to be enforcing were not important, and industry should be given every benefit of the doubt that they were doing the right thing regardless of the letter of the law. Fundamentally, the the enforcers of the regulations didn't think the regulations should be enforced, and so they didn't.

In other words, it was the anti-regulation philosophy that caused the regulations to not be enforced.

You say that the problem was caused by the lack of regulation. But that presumes that the oil companies would not perform proper maintenance and safety procedures unless forced to. It presumes that the default case in the absence of regulation would be that BP shirked their responsibility and allowed this spill to occur. MMS needed to have prevented the spill which BP would have otherwise caused. Which is an accurate view of reality, but the opposite of the anti-regulation philosophy.

So there are two ways in which the anti-regulation philosophy falls short. Blaming the lack of regulatory enforcement for the spill is a perfect example of how.

And as to why anyone would think new regulations would be enforced? I think they would be, provided the enactment of new regulations -- which suggests a belief that regulations are important just like the repealing of regulations suggests a belief that they are not -- is coincident with a housecleaning of MMS and the hiring of people who are not of the anti-regulation philosophy and a director-level-on-down belief that yes, these regulations are important.

Do I think this certainly will happen? Not at all. The firing of the MMS director is just the start of a long road I'm not sure they're going to walk down. However, arguing that because regulations we demonstrably need to make industry do the right thing may not be sufficiently enforced, is not a reason to not have the regulations! It's an argument to press the government to focus on making sure their agents do enforce them.

If you're anti-regulation, latching onto the failure of MMS to keep BP as evidence of your cause is the last thing you want to do.

Besides, compared to regular inspections of safety equipment and so on, simply regulating the need for relief wells to be pre-drilled in case all the other safety regulations aren't followed would be quite likely to succeed. Your general dislike of regulation does not outweigh the need for simple improvements like this.

Re:They didn't fix a lot of things

[godefroi]

More of the same, man, more of the same. It's all the same crap, packaged up in different colors and with a different label on top.

How much did they save?

[GrumblyStuff]

I mean, the whole rig's cost is in the hundreds of millions (Wiki says $560 mil but google link said $350 mil). The whole disaster is in the tens of billions, ain't it?

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.

Makes the $500,000 a day lease look like pennies.

Re:How much did they save?

[jd2112]

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.Makes the $500,000 a day lease look like pennies.

In the corporate word, the important thing is to save money no matter how mutch extra it costs.

Re:How much did they save?

[CoffeeDog]

I hate to be a cynic but if you take the cost savings on cutting safety corners across all their operations (rigs, refineries, etc) for the time the company has been operating them, I bet they still came out on top and BP wouldn't change a damn thing about how they operate short of some regulatory body (lol MMS) forcing them to.

Re:How much did they save?

[GrumblyStuff]

I am not an geologist or a drilling rig operator or a CEO of a multibillion dollar company (IANAGOADROOACOAMDC) but I would at least ask that they have their equipment in working order. Given the state of BP's stock, I'm sure the stockholders would appreciate that, too (at least in hindsight anyway).

Computers should be working. BOP should be fully functional. They should have disaster plans from the getgo instead of trying to think of stuff on the fly or using failed tactics from the 30 year old Ixtoc spill just to look busy.

To use a car analogy, I'm not asking you to walk. I'm asking you to have working brakes, lights, windshield and windshield wipers, seatbelts, to not drive recklessly, and to have the physical capacity to drive.

Cutting corners is the name of the game

[plopez]

Cutting corners is the corporate way. I have seen so much "Mickey Mouse" stuff at places I've worked it disgusts me. Untrained workers, electrical boxes in pools of water, large pumps at refineries held in place by 4 bolts rather than the six bolts which were intended to be used etc. But of course, none of these problems are the CEO (or board members) of BP's fault. They only take credit when things go right. Avoiding responsibility is the name of the game.

Re:Egregious

[quickpick]

The question is why is there not criminal prosecutions for bad engineering that leads to the loss of life? Why is it that only people with guns who kill people get criminal prosecutions?

IMNL but be very, very careful where you are going with this. I submit this as an example: If you built a machine and it happened to be involved in the death of several people a prosecutor could argue that your machine was 'bad engineering' and if they found sufficient evidence that people disagreed with you and were able to convince a jury of this you would end up in jail. Now if you were in a project where everyone was in agreement that it was a good idea then he could potentially still argue collusion. I'd imagine that you would have 'tolerances' but even these could possibly be argued as bad engineering, because why would you unleash upon the people a machine that statistically would kill a certain number of people?
If all we do is prosecute failure then no one would be willing to risk their lives to innovate. The only real loss here is if the industry learns nothing and repeats its mistake.

Interesting

[Shulai]

Nobody is bashing Windows so far, yet it seems to be what the editor look for when he wrote the headline. Has Windows improved enough that nobody try to make fun of it anymore, or slashdotters are already older and more mature?

the regulators were the regulated

[circletimessquare]

the regulations don't matter in this case. i'm glad you admit we need some regulations, but the real issue here is regulator==regulated

http://www.washingtonpost.com/wp-dyn/content/article/2010/07/22/AR2010072205133.html?hpid=topnews

His statement came after Rep. Jackie Speier (D-Calif.) asked about a Washington Post article that reported that dozens of former Interior officials had crossed over into the oil industry and that three out of four industry lobbyists had once worked for the federal government.

The rate is more than double the norm in Washington, where industries recruit about 30 percent of their lobbyists from the government, according to data from the nonpartisan Center for Responsive Politics. With more than 600 registered lobbyists, the industry has among the biggest and most powerful contingents in Washington, The Post reported.

the lobbyists, the interior officials, the corporate assholes: all the same people

all the same smoochy same golf hole playing same bar attending backslapping crowd of assholes

that's why we had the disaster in the gulf

you can pass all the regulations you want, it doesn't matter if the ones who are supposed to be policing the industry ARE the industry

That's why capitalism is broken

[copponex]

A company we hired nearly destroyed the Gulf of Mexico... What's that got to do with us?

One our business partners was rating these bonds as AAA when they were worthless, and we were busy making billions passing the bonds off as good investments... What's that got to do with us?

The company we hired to dispose of this toxic waste is just dumping it in a river... What's that got to do with us?

In effect, modern capitalism is a system of mafia thugs and their hired patsies who operate technically within the law, as long as they hire an agent to do their dirty work to take the fall. Any of the real costs can be passed off to the public, either though bailouts or just ruining the commons.

Re:It was Windows NT

[Fookin]

From what I recall, this rig had been in place since 2000 and hadn't been in dry dock since launch. My guess is that they wanted to run a tried and true OS that was compatible with all their systems / sensors / panels, etc and Win2K had just come out. I remember the place I was working at in 2000 was still running NT4 so it doesn't surprise me that they wouldn't have upgraded. I'd love to know why their IT staff didn't send a modern workstation with VMWare Player and a NT4 image installed to run the system. Would have saved a lot of pain and trouble ...

Re:Egregious

[darkmeridian]

Transocean was not an independent contractor. BP directed the Transocean employees on what to do. The Transocean employees sometimes told the BP executives that they were risking a failure of the well, but their concerns were vetoed. Decisions such as the choice of casing, what material to use to plug the well, and whether or not to try to fix the safety equipment were all made by BP and executed by Transocean.

Re:Why didn't they fix it?

[jackalope]

You've obviously never worked at a Fortune 500 company. The simplest things take weeks, if not months, to requisition. Then add submitting a labor request to have it installed. Add in a helicopter trip to get the tech and the equipment to the rig and you've got massive delays. Seems like they need expedited procedures for life-critical safety systems. (Sorry that was obvious wasn't it).

Re:BSOD

[NotBornYesterday]

Investigations of most disasters reveal not a single cause, but a combination of factors which lead to the disaster itself. Often, the absence of any one of those root causes may have avoided the disaster, or at least mitigated it to some degree. While I would not minimize the importance of the other factors which have already been acknowledged as key causes, identifying all the possible causes is critical to avoiding future repetition of the problem. If a computer-controlled alarm system was so faulty that its operators shut it down rather than endure its false alarms, we should give it due consideration as a potential contributing factor.

Re:Why didn't they fix it?

[Ex-MislTech]

In most multi million dollar operations some kind of redundancy is built in.

I am thinking there was a laptop or other desktop in a non critical role
that could have stepped into the role.

I think bad management plus ppl making excuses to not fix the problem
are often at the root of the issue.

We have a "not my problem" cutlure that spends more time looking for
excuses to deflect something than just fixing it and moving on.

I have been doing this for over 25 years so I can speak with some experience.

there is no functioning government in haiti

[circletimessquare]

its a libertarian utopia

every abuse by government you decry, does not happen in haiti. instead, those same abuses, and a hundred thousand worse abuses, happen on every street corner, by thugs and mafia instead

fact, whether you realize it or not: make government small, and a power vacuum will exist that will be filled by entities that are not accountable to you. being not accountable to you, there is no recourse when they abuse you. that really is the truth. someday you will wake up and realize and stop working so damn hard to destroy this country