BSOD Issues On Deepwater Horizon

ctdownunder passes along this excerpt from a NY Times article about a rig worker's testimony concerning the April 20 accident at the Deepwater Horizon well: "The emergency alarm on the Deepwater Horizon was not fully activated on the day the oil rig caught fire and exploded, triggering the massive spill in the Gulf of Mexico, a rig worker on Friday told a government panel investigating the accident. ... On Friday, Mr. Williams added several new details about the equipment on the vessel, testifying that another Transocean official turned a critical system for removing dangerous gas from the drilling shack to 'bypass mode.' When he questioned that decision, Mr. Williams said, he was reprimanded. ... Problems existed from the beginning of drilling the well, Mr. Williams said. For months, the computer system had been locking up, producing what the crew deemed the 'blue screen of death.' 'It would just turn blue,' he said. 'You’d have no data coming through.' Replacement hardware had been ordered but not yet installed by the time of the disaster, he said." The article doesn't mention whether it was specifically a Windows BSOD, or just an error screen that happened to be blue.

Comments

BSOD

[DWMorse]

A Blue Screen of Death by a computer yields a Black Screen of Death on an ocean. Interesting. Kill all humans, anyone?

Re:BSOD

[pitchpipe]

So in this case BSOD is not a metaphor!

Re:BSOD

[Anonymous Coward]

There is no evidence that BSODs contributed to this disaster. What is know to have contributed is the cheap cement job, plugged pressure sensors on the blowout preventer, possible damage to the blowout preventer during drilling (rubber fragments observed), and using seawater instead of drilling mud. None of these were automated.

Re:BSOD

[GrumblyStuff]

To be fair, the cheap cement job was what BP ordered. I think it was two plugs instead of three and they skipped the final (and expensive) inspection.

That said, Halliburton still needs to answer for all the shit it's pulled in Iraq.

Re:BSOD

[NotBornYesterday]

Investigations of most disasters reveal not a single cause, but a combination of factors which lead to the disaster itself. Often, the absence of any one of those root causes may have avoided the disaster, or at least mitigated it to some degree. While I would not minimize the importance of the other factors which have already been acknowledged as key causes, identifying all the possible causes is critical to avoiding future repetition of the problem. If a computer-controlled alarm system was so faulty that its operators shut it down rather than endure its false alarms, we should give it due consideration as a potential contributing factor.

Re:BSOD

[iamavirus]

Based on reading the article + other news sources, the alarm system wasn't disaster preventive. It was a gas (danger) detector, and may have prevented zero / some / all fatalities.

Re:BSOD

[GrumblyStuff]

Halliburton did the cementing job. http://online.wsj.com/article/SB10001424052748703572504575214593564769072.html

http://www.huffingtonpost.com/2010/05/10/halliburton-deepwater-hor_n_570733.html

They didn't fix a lot of things

[18_Rabbit]

For example, they KNEW that the BOP (blowout preventer) was not functioning correctly. one of the 2 control systems was out, and they had been bringing up pieces of the rubber seal in the test fluid. They were cutting corners on their cut corners. You'd think this would serve as exhibit A to silence all the "GOVERNMENT R BAD, CORPORATIONS R GOOD" nutcases in the USA today, but unfortunately it does not seem to have had that effect.

Re:They didn't fix a lot of things

[Anonymous Coward]

the government's responses to national crises like this one should also tell you that those "GOVERNMENT IS GOOD, DOWN WITH CORPS" nutcases in the usa should also be silenced.

How about down with self-serving bureaucracy? you know, the kind that insulates its ideology from reality so much that everyone else is left holding the resulting inevitable calamity.

Re:They didn't fix a lot of things

[Attila Dimedici]

You mean because the regulators did such a wonderful job at enforcing the regulations that were already in place that we should create new regulations?
I am never a fan of government regulations, but when there are problems with an industry we can discuss possible government regulations of that industry. However, I am always opposed to new regulations to address a problem that appears to have happened largely because exisitng regulations were not being followed. If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

Re:They didn't fix a lot of things

[amorsen]

If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

The regulators were tasked to check that the companies followed the procedures for checking their own operations. This kind of twice-removed oversight is becoming increasingly common in lots of places, because it saves money for the government (popular with voters) as well as being popular in the private sector (for obvious reasons).

It works great as long as companies are overall honest and all their problems are caused by simple negligence. It doesn't work so well in the face of outright fraud.

Re:They didn't fix a lot of things

[spun]

GOVERNMENT R BAD, CORPORATIONS R GOOD
However:
BP screwed up bad
BP is a corporation
Therefore, the initial statement must be false, even if governments R BAD also.

Proof by Reductio ad absurdum. No fallacy. Thanks for playing, and here's a copy of our home game, 'Logic for Dummies.'

Re:They didn't fix a lot of things

[T.E.D.]

The regulators were tasked to check that the companies followed the procedures for checking their own operations. This kind of twice-removed oversight is becoming increasingly common in lots of places, because it saves money for the government (popular with voters) as well as being popular in the private sector (for obvious reasons). It works great as long as companies are overall honest and all their problems are caused by simple negligence. It doesn't work so well in the face of outright fraud.

It doesn't work period. Anybody who understands economics (as "fiscal conservatives" claim they do) should understand that you can't expect entities to act contrary to the incentives around them out of a sense of civic duty or something. When you set up a system where there are tremendous financial incentives to cheat, the chances are getting caught are almost nil, and even then the punishments will be laughable compared to the money saved by cheating, it would be insane to not expect things like this to happen.

The only way to prevent reoccurances is to change the system. That will require changing the regulations.

Re:They didn't fix a lot of things

[fyoder]

Then we need new regulations regulating regulators. And I know, you're thinking, but who will regulate the regulators of the regulators? There will be regulators for the regulators of the regulators as well. It will be regulators all the way to the bottom.

The real answer is to stop regarding corporations as 'persons' and go back to regarding them as what they are, associations, and ones which can be disbanded when they screw up big time. A corporation who, through its negligence, causes a major environmental disaster doesn't get to continue to exist.

Granted, that's unenforceable outside of a particular nation state, but it would certainly reduce share holder value if several countries, including the US, regarded it as outlaw and forbade it to do business.

Or if we're going to continue to regard them as persons, what sort of a punishment would a human person get for gross criminal negligence? What would be the corporate equivalent?

Because when it comes right down to it, regulation is better than no regulation, but ultimately can't be counted on, because there are minimal consequences for failure to comply, and because of lax enforcement in the first place.

The first rule for corporations should be that if they screw up big time, they cease to exist. But anything that draconian has to be preceded by defining corporations in law as non-persons. Sadly, given US Supreme Court rulings on the issue, it might take a constitutional amendment.

Re:They didn't fix a lot of things

[camperdave]

I am never a fan of government regulations

Government regulations are what keep you from dying every time you make toast, plug in the kettle, or turn on the TV. They keep you safe on the roads. They stop your house from falling in, from toxic chemicals being found in your food, and thousands upon thousands of other hazards that every day life throws at you.

Re:They didn't fix a lot of things

[Chris Burke]

However, I am always opposed to new regulations to address a problem that appears to have happened largely because exisitng regulations were not being followed. If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

Because the lack of regulations, and the lack of regulatory enforcement, are closely related phenomenon. Both are the consequence of a government who, like you, viewed government regulation as a bad thing, and felt that industry was best left to itself. So they relaxed the regulations. And while they didn't succeed in getting rid of all regulations, that same philosophy carried over into the hiring and management of the regulatory body -- that the regulations they were supposed to be enforcing were not important, and industry should be given every benefit of the doubt that they were doing the right thing regardless of the letter of the law. Fundamentally, the the enforcers of the regulations didn't think the regulations should be enforced, and so they didn't.

In other words, it was the anti-regulation philosophy that caused the regulations to not be enforced.

You say that the problem was caused by the lack of regulation. But that presumes that the oil companies would not perform proper maintenance and safety procedures unless forced to. It presumes that the default case in the absence of regulation would be that BP shirked their responsibility and allowed this spill to occur. MMS needed to have prevented the spill which BP would have otherwise caused. Which is an accurate view of reality, but the opposite of the anti-regulation philosophy.

So there are two ways in which the anti-regulation philosophy falls short. Blaming the lack of regulatory enforcement for the spill is a perfect example of how.

And as to why anyone would think new regulations would be enforced? I think they would be, provided the enactment of new regulations -- which suggests a belief that regulations are important just like the repealing of regulations suggests a belief that they are not -- is coincident with a housecleaning of MMS and the hiring of people who are not of the anti-regulation philosophy and a director-level-on-down belief that yes, these regulations are important.

Do I think this certainly will happen? Not at all. The firing of the MMS director is just the start of a long road I'm not sure they're going to walk down. However, arguing that because regulations we demonstrably need to make industry do the right thing may not be sufficiently enforced, is not a reason to not have the regulations! It's an argument to press the government to focus on making sure their agents do enforce them.

If you're anti-regulation, latching onto the failure of MMS to keep BP as evidence of your cause is the last thing you want to do.

Besides, compared to regular inspections of safety equipment and so on, simply regulating the need for relief wells to be pre-drilled in case all the other safety regulations aren't followed would be quite likely to succeed. Your general dislike of regulation does not outweigh the need for simple improvements like this.

Re:They didn't fix a lot of things

[godefroi]

More of the same, man, more of the same. It's all the same crap, packaged up in different colors and with a different label on top.

Re:They didn't fix a lot of things

[Vancorps]

Out of curiosity, where are you getting your information? You mean the last time there was a spill in the Gulf?

The oil spill off the Australian coast in August last year would seem to counter your claim entirely. Their procedures were lax and probably are still lax because those procedures are expensive. I'm not sure why people are so quick to dismiss the Gulf spill as a series of calamitous events when there is a ton of evidence to the contrary. Maybe people just want cheap gas too much and are willing to ignore all evidence so they don't have to face any consequences.

The sad reality is that oil is spilled everyday, Shell spills every year as much oil as the Exxon Valdez disaster in Alaska. Of course they spill it in Nigeria so no one cares.

Egregious

[eclectro]

There are faulty engineering and management decisions every step of the way when producing this well. This is not the first disaster for BP that ended in the loss of life. The question is why is there not criminal prosecutions for bad engineering that leads to the loss of life? Why is it that only people with guns who kill people get criminal prosecutions?

Re:Egregious

[quickpick]

The question is why is there not criminal prosecutions for bad engineering that leads to the loss of life? Why is it that only people with guns who kill people get criminal prosecutions?

IMNL but be very, very careful where you are going with this. I submit this as an example: If you built a machine and it happened to be involved in the death of several people a prosecutor could argue that your machine was 'bad engineering' and if they found sufficient evidence that people disagreed with you and were able to convince a jury of this you would end up in jail. Now if you were in a project where everyone was in agreement that it was a good idea then he could potentially still argue collusion. I'd imagine that you would have 'tolerances' but even these could possibly be argued as bad engineering, because why would you unleash upon the people a machine that statistically would kill a certain number of people?
If all we do is prosecute failure then no one would be willing to risk their lives to innovate. The only real loss here is if the industry learns nothing and repeats its mistake.

Re:Egregious

[Andy Dodd]

From most of what I've read, the subcontractors in question (Halliburton and Transocean) were doing the work, but BP had full control over the operations.

The flow was something like this:
Halliburton or Transocean: That's a bad idea, we don't recommend that.
BP: Do it anyway.
H/T: OK...

Although the question is at what point H/T should have said, "Hell no!"

Re:Egregious

[Anonymous Coward]

I found this episode of 60 minutes quite interesting:

http://www.cbsnews.com/video/watch/?id=6490509n&tag=api

Apparently, BP was putting on a lot of pressure to do things quickly, since they were running behind schedule and it was costing them money.

Specifically, on the day of the accident, there was an argument between representatives of Transocean and BP on how to close the well (in preparation for later exploitation by another ship). Transocean was in favor the slower, safer procedure. BP wanted things to be done more quickly. They did it the BP way, which was the point when the accident happened. So, according to this report, there were BP emplyes on the Deepwater Horizont, and they influenced the procedures by pressuring their subcontractors.

According to the report, several other things had to happen as well in order for things to go wrong so badly, but I would not so easily let BP of the hook.

Re:Egregious

[Achromatic1978]

Seriously, stop using Terry Childs as a posterboy for "Wronged Geek". He was an obstinate, self-serving asshole.

Protip for all you people saying "They could have reconfigured the routers, etc." (on Childs refusal to hand over passwords) - not so much.

Why? Because Childs had either disabled serial consoles, disabled password recovery, or configured devices to -never- save configuration, only to run in RAM.

Well, shit, you say, restore the config from backups. Guess what, SF owned no backups of the configuration files, or network maps. The only configuration files Childs kept were on his personal laptop, encrypted with a key known only to him, and configured such that his laptop was the only device capable of updating configs. Network maps? Same. Sitting on his personal laptop. Nowhere else.

The guy viewed SF's network as his personal playground, and believed no-one else worthy to take the reins of it - guess what, he had no authority to decide that, and when he got nicely obstinate about it, he crossed a fairly clear line in the sand.

Stop the martyred geek defending valiantly our security creed. It bares little resemblance to reality.

Re:Egregious

[shadowofwind]

What's it got to do with BP? The rig was owned and operated by a company called Transocean.

This is a common legal and accounting ploy: subcontract everything to other companies, then you're not responsible for anything, even though you're in charge of everything.

I recently worked for a company, run incidentally by the spouse of a BP chief executive, that sells a medical product for applications that the product can not legally be sold for (in the US). Its way around this is to create three companies, one for engineering, one for distribution, and one for marketing. That way, the parent company claims that its selling nothing illegally because it distributes nothing, but only provides information. And the distributor claims that it does not target its product for the illegal applications, since it merely distributes. And the engineering company evades FDA engineering process requirements by saying that it merely distributes the product made by the engineering company, which ignores the regulations because it is ostensibly not subject to regulation since it is not the distributor, and it doesn't have a distribution operation that can be shut down. But all three companies are essentially the same company, run by the same people. The 'ethic' involved is that if you haven't yet been sued successfully, or shut down by regulators, then its all good.

At least Halliburton and Transocean have a separate existence from BP. But BP is still responsible.

Re:Egregious

[darkmeridian]

Transocean was not an independent contractor. BP directed the Transocean employees on what to do. The Transocean employees sometimes told the BP executives that they were risking a failure of the well, but their concerns were vetoed. Decisions such as the choice of casing, what material to use to plug the well, and whether or not to try to fix the safety equipment were all made by BP and executed by Transocean.

How much did they save?

[GrumblyStuff]

I mean, the whole rig's cost is in the hundreds of millions (Wiki says $560 mil but google link said $350 mil). The whole disaster is in the tens of billions, ain't it?

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.

Makes the $500,000 a day lease look like pennies.

Re:How much did they save?

[jd2112]

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.Makes the $500,000 a day lease look like pennies.

In the corporate word, the important thing is to save money no matter how mutch extra it costs.

Re:How much did they save?

[CoffeeDog]

I hate to be a cynic but if you take the cost savings on cutting safety corners across all their operations (rigs, refineries, etc) for the time the company has been operating them, I bet they still came out on top and BP wouldn't change a damn thing about how they operate short of some regulatory body (lol MMS) forcing them to.

Re:How much did they save?

[GrumblyStuff]

I am not an geologist or a drilling rig operator or a CEO of a multibillion dollar company (IANAGOADROOACOAMDC) but I would at least ask that they have their equipment in working order. Given the state of BP's stock, I'm sure the stockholders would appreciate that, too (at least in hindsight anyway).

Computers should be working. BOP should be fully functional. They should have disaster plans from the getgo instead of trying to think of stuff on the fly or using failed tactics from the 30 year old Ixtoc spill just to look busy.

To use a car analogy, I'm not asking you to walk. I'm asking you to have working brakes, lights, windshield and windshield wipers, seatbelts, to not drive recklessly, and to have the physical capacity to drive.

Re:How much did they save?

[Facegarden]

I mean, the whole rig's cost is in the hundreds of millions (Wiki says $560 mil but google link said $350 mil). The whole disaster is in the tens of billions, ain't it?

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.

Makes the $500,000 a day lease look like pennies.

They normally do spare no cost keeping these things going.

My company sells some sensors to oil rig people, and the way it works is that they have a limited equipment budget, but an unlimited repair budget. Yeah. Unlimited.

They are smart enough to buy some spares, but when something critical breaks and they don't have a backup, they will spare no expense to get it fixed. They've had something break on them in the middle of the night, so they put it on a helicopter, flew it to the mainland, and paid our partner in texas to drive 2 hours to meet them with his repair truck, fix it and drive home, and then flew it back. At like 3am.

Which is impressive, and makes sense given how much money these things are worth and how much they cost.

Which makes me wonder why their computers weren't fixed sooner.

The problem is, when our sensors break there's no "bypass mode", so they *have* to be fixed or they can't do anything.

With the computers able to be bypassed, people can ignore it until it becomes a problem.

With whats at stake here, critical safety systems should *not* have a "bypass" mode, I would think.

These people also understand when a mechanical tool is broken, it needs to be fixed. Computers are somehow very "mysterious", so there is a lack of understanding that could be a problem too.
-Taylor

Re:Safety List

[Anonymous Coward]

Yes; steaks cost massive amounts of money, but what does that have to do with what's at stake?

Re:Safety List

[Anonymous Coward]

life and limb [...] at steak

Hannibal, is that you?

Cutting corners is the name of the game

[plopez]

Cutting corners is the corporate way. I have seen so much "Mickey Mouse" stuff at places I've worked it disgusts me. Untrained workers, electrical boxes in pools of water, large pumps at refineries held in place by 4 bolts rather than the six bolts which were intended to be used etc. But of course, none of these problems are the CEO (or board members) of BP's fault. They only take credit when things go right. Avoiding responsibility is the name of the game.

Re:Don't throw Bill under the bus

[ch-chuck]

Not only that, but buried somewhere deep in the multi page EULA that nobody reads is the clause, "company shall not be held in indemnity for losses to person or environmental damage from deepwater oilwells bursting into flame due to defects in software, other than the cost of replacing defective media".

Interesting

[Shulai]

Nobody is bashing Windows so far, yet it seems to be what the editor look for when he wrote the headline. Has Windows improved enough that nobody try to make fun of it anymore, or slashdotters are already older and more mature?

the regulators were the regulated

[circletimessquare]

the regulations don't matter in this case. i'm glad you admit we need some regulations, but the real issue here is regulator==regulated

http://www.washingtonpost.com/wp-dyn/content/article/2010/07/22/AR2010072205133.html?hpid=topnews

His statement came after Rep. Jackie Speier (D-Calif.) asked about a Washington Post article that reported that dozens of former Interior officials had crossed over into the oil industry and that three out of four industry lobbyists had once worked for the federal government.

The rate is more than double the norm in Washington, where industries recruit about 30 percent of their lobbyists from the government, according to data from the nonpartisan Center for Responsive Politics. With more than 600 registered lobbyists, the industry has among the biggest and most powerful contingents in Washington, The Post reported.

the lobbyists, the interior officials, the corporate assholes: all the same people

all the same smoochy same golf hole playing same bar attending backslapping crowd of assholes

that's why we had the disaster in the gulf

you can pass all the regulations you want, it doesn't matter if the ones who are supposed to be policing the industry ARE the industry

It was Windows NT

[Fookin]

I was watching the testimony and he stated that it was a Windows NT system and was constantly giving a BSOD. They had replaced and reimaged the HDD over and over but it still kept happening. There were new servers, workstations, etc standing by and waiting to be installed, but another problem creeped in. They were waiting for another ship to figure out a way to run the old software on the new machines. Once that other ship could get it working and document it, they would then do the replacement on their end. I'm guessing it was a Windows NT 4 workstation.

Re:It was Windows NT

[Fookin]

From what I recall, this rig had been in place since 2000 and hadn't been in dry dock since launch. My guess is that they wanted to run a tried and true OS that was compatible with all their systems / sensors / panels, etc and Win2K had just come out. I remember the place I was working at in 2000 was still running NT4 so it doesn't surprise me that they wouldn't have upgraded. I'd love to know why their IT staff didn't send a modern workstation with VMWare Player and a NT4 image installed to run the system. Would have saved a lot of pain and trouble ...

That's why capitalism is broken

[copponex]

A company we hired nearly destroyed the Gulf of Mexico... What's that got to do with us?

One our business partners was rating these bonds as AAA when they were worthless, and we were busy making billions passing the bonds off as good investments... What's that got to do with us?

The company we hired to dispose of this toxic waste is just dumping it in a river... What's that got to do with us?

In effect, modern capitalism is a system of mafia thugs and their hired patsies who operate technically within the law, as long as they hire an agent to do their dirty work to take the fall. Any of the real costs can be passed off to the public, either though bailouts or just ruining the commons.

Re:That's why capitalism is broken

[Rich0]

Agreed. In fact, as long as the big company has no liability it just turns into one big race for the bottom.

Suppose you run a reputable oil-rig operations company. You'd like to have people outsource their rigs to you. You believe in safety and the environment, so you take all kinds of steps to avoid something like the BP disaster. What happens? Well, you go out of business. You have to compete against other companies that cut corners. Companies like BP don't care about the safety of your workers or the environment, since that is on you. Your competitors charge less, and that is all they care about. Sure, it isn't sustainable, but most small companies aren't sustainable. The company running the rig can pay out dividends while the money is there, and then fold when the lawsuits hit. Indeed, if you didn't run a disreputable company your shareholders would probably fire you (low dividends compared to peers) and replace you with somebody who would mismanage it.

In other industries there is a clear assignment of responsibility, which cannot be outsourced. You can hire somebody else to do the work, but not to assume the liability. If Bayer sells a bottle of tainted aspirin, then they're liable even if they bought bad pills from a supplier. The only thing they're not liable for is what happens to the package after they sell it to the warehouse/store, although they are required to put the pills in tamper-evident packaging.

Indeed, in many industries liability is personal. That's why certified engineers have to sign off on bridges - they are personally responsible for the design (but not necessarily the implementation). I think the EU does something similar for drugs.

Re:Don't throw Bill under the bus

[Critical Facilities]

If it turns out these crashes are Windows blue screens, the media will jump all over Microsoft

Well, before all the Microsoft Haters pile on, according to this the Control System in place was something called Cameron Multiplex Control System, which I've also seen referred to as Cameron MUX and CAMITROL. I am not pretending to be an expert in these things, just thought I'd share what little Googling turned up.

In short, it looks pretty unlikely that there's going to be a red hot poker headed toward Redmond over this.

Re:Why didn't they fix it?

[jackalope]

You've obviously never worked at a Fortune 500 company. The simplest things take weeks, if not months, to requisition. Then add submitting a labor request to have it installed. Add in a helicopter trip to get the tech and the equipment to the rig and you've got massive delays. Seems like they need expedited procedures for life-critical safety systems. (Sorry that was obvious wasn't it).

Re:Why didn't they fix it?

[Ex-MislTech]

In most multi million dollar operations some kind of redundancy is built in.

I am thinking there was a laptop or other desktop in a non critical role
that could have stepped into the role.

I think bad management plus ppl making excuses to not fix the problem
are often at the root of the issue.

We have a "not my problem" cutlure that spends more time looking for
excuses to deflect something than just fixing it and moving on.

I have been doing this for over 25 years so I can speak with some experience.

there is no functioning government in haiti

[circletimessquare]

its a libertarian utopia

every abuse by government you decry, does not happen in haiti. instead, those same abuses, and a hundred thousand worse abuses, happen on every street corner, by thugs and mafia instead

fact, whether you realize it or not: make government small, and a power vacuum will exist that will be filled by entities that are not accountable to you. being not accountable to you, there is no recourse when they abuse you. that really is the truth. someday you will wake up and realize and stop working so damn hard to destroy this country