BSOD Issues On Deepwater Horizon

ctdownunder passes along this excerpt from a NY Times article about a rig worker's testimony concerning the April 20 accident at the Deepwater Horizon well: "The emergency alarm on the Deepwater Horizon was not fully activated on the day the oil rig caught fire and exploded, triggering the massive spill in the Gulf of Mexico, a rig worker on Friday told a government panel investigating the accident. ... On Friday, Mr. Williams added several new details about the equipment on the vessel, testifying that another Transocean official turned a critical system for removing dangerous gas from the drilling shack to 'bypass mode.' When he questioned that decision, Mr. Williams said, he was reprimanded. ... Problems existed from the beginning of drilling the well, Mr. Williams said. For months, the computer system had been locking up, producing what the crew deemed the 'blue screen of death.' 'It would just turn blue,' he said. 'You’d have no data coming through.' Replacement hardware had been ordered but not yet installed by the time of the disaster, he said." The article doesn't mention whether it was specifically a Windows BSOD, or just an error screen that happened to be blue.

Comments

BSOD

[DWMorse]

A Blue Screen of Death by a computer yields a Black Screen of Death on an ocean. Interesting. Kill all humans, anyone?

Re:BSOD

[pitchpipe]

So in this case BSOD is not a metaphor!

Re:BSOD

[Anonymous Coward]

There is no evidence that BSODs contributed to this disaster. What is know to have contributed is the cheap cement job, plugged pressure sensors on the blowout preventer, possible damage to the blowout preventer during drilling (rubber fragments observed), and using seawater instead of drilling mud. None of these were automated.

Re:BSOD

[NotBornYesterday]

Investigations of most disasters reveal not a single cause, but a combination of factors which lead to the disaster itself. Often, the absence of any one of those root causes may have avoided the disaster, or at least mitigated it to some degree. While I would not minimize the importance of the other factors which have already been acknowledged as key causes, identifying all the possible causes is critical to avoiding future repetition of the problem. If a computer-controlled alarm system was so faulty that its operators shut it down rather than endure its false alarms, we should give it due consideration as a potential contributing factor.

Re:BSOD

[GrumblyStuff]

Halliburton did the cementing job. http://online.wsj.com/article/SB10001424052748703572504575214593564769072.html

http://www.huffingtonpost.com/2010/05/10/halliburton-deepwater-hor_n_570733.html

They didn't fix a lot of things

[18_Rabbit]

For example, they KNEW that the BOP (blowout preventer) was not functioning correctly. one of the 2 control systems was out, and they had been bringing up pieces of the rubber seal in the test fluid. They were cutting corners on their cut corners. You'd think this would serve as exhibit A to silence all the "GOVERNMENT R BAD, CORPORATIONS R GOOD" nutcases in the USA today, but unfortunately it does not seem to have had that effect.

Re:They didn't fix a lot of things

[Anonymous Coward]

the government's responses to national crises like this one should also tell you that those "GOVERNMENT IS GOOD, DOWN WITH CORPS" nutcases in the usa should also be silenced.

How about down with self-serving bureaucracy? you know, the kind that insulates its ideology from reality so much that everyone else is left holding the resulting inevitable calamity.

Re:They didn't fix a lot of things

[Attila Dimedici]

You mean because the regulators did such a wonderful job at enforcing the regulations that were already in place that we should create new regulations?
I am never a fan of government regulations, but when there are problems with an industry we can discuss possible government regulations of that industry. However, I am always opposed to new regulations to address a problem that appears to have happened largely because exisitng regulations were not being followed. If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

Re:They didn't fix a lot of things

[amorsen]

If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

The regulators were tasked to check that the companies followed the procedures for checking their own operations. This kind of twice-removed oversight is becoming increasingly common in lots of places, because it saves money for the government (popular with voters) as well as being popular in the private sector (for obvious reasons).

It works great as long as companies are overall honest and all their problems are caused by simple negligence. It doesn't work so well in the face of outright fraud.

Re:They didn't fix a lot of things

[spun]

GOVERNMENT R BAD, CORPORATIONS R GOOD
However:
BP screwed up bad
BP is a corporation
Therefore, the initial statement must be false, even if governments R BAD also.

Proof by Reductio ad absurdum. No fallacy. Thanks for playing, and here's a copy of our home game, 'Logic for Dummies.'

Re:They didn't fix a lot of things

[T.E.D.]

The regulators were tasked to check that the companies followed the procedures for checking their own operations. This kind of twice-removed oversight is becoming increasingly common in lots of places, because it saves money for the government (popular with voters) as well as being popular in the private sector (for obvious reasons). It works great as long as companies are overall honest and all their problems are caused by simple negligence. It doesn't work so well in the face of outright fraud.

It doesn't work period. Anybody who understands economics (as "fiscal conservatives" claim they do) should understand that you can't expect entities to act contrary to the incentives around them out of a sense of civic duty or something. When you set up a system where there are tremendous financial incentives to cheat, the chances are getting caught are almost nil, and even then the punishments will be laughable compared to the money saved by cheating, it would be insane to not expect things like this to happen.

The only way to prevent reoccurances is to change the system. That will require changing the regulations.

Re:They didn't fix a lot of things

[fyoder]

Then we need new regulations regulating regulators. And I know, you're thinking, but who will regulate the regulators of the regulators? There will be regulators for the regulators of the regulators as well. It will be regulators all the way to the bottom.

The real answer is to stop regarding corporations as 'persons' and go back to regarding them as what they are, associations, and ones which can be disbanded when they screw up big time. A corporation who, through its negligence, causes a major environmental disaster doesn't get to continue to exist.

Granted, that's unenforceable outside of a particular nation state, but it would certainly reduce share holder value if several countries, including the US, regarded it as outlaw and forbade it to do business.

Or if we're going to continue to regard them as persons, what sort of a punishment would a human person get for gross criminal negligence? What would be the corporate equivalent?

Because when it comes right down to it, regulation is better than no regulation, but ultimately can't be counted on, because there are minimal consequences for failure to comply, and because of lax enforcement in the first place.

The first rule for corporations should be that if they screw up big time, they cease to exist. But anything that draconian has to be preceded by defining corporations in law as non-persons. Sadly, given US Supreme Court rulings on the issue, it might take a constitutional amendment.

Re:They didn't fix a lot of things

[camperdave]

I am never a fan of government regulations

Government regulations are what keep you from dying every time you make toast, plug in the kettle, or turn on the TV. They keep you safe on the roads. They stop your house from falling in, from toxic chemicals being found in your food, and thousands upon thousands of other hazards that every day life throws at you.

Re:They didn't fix a lot of things

[Chris Burke]

However, I am always opposed to new regulations to address a problem that appears to have happened largely because exisitng regulations were not being followed. If regulators have failed to enforce existing regulations, what makes anyone think they will enforce any new regulations?

Because the lack of regulations, and the lack of regulatory enforcement, are closely related phenomenon. Both are the consequence of a government who, like you, viewed government regulation as a bad thing, and felt that industry was best left to itself. So they relaxed the regulations. And while they didn't succeed in getting rid of all regulations, that same philosophy carried over into the hiring and management of the regulatory body -- that the regulations they were supposed to be enforcing were not important, and industry should be given every benefit of the doubt that they were doing the right thing regardless of the letter of the law. Fundamentally, the the enforcers of the regulations didn't think the regulations should be enforced, and so they didn't.

In other words, it was the anti-regulation philosophy that caused the regulations to not be enforced.

You say that the problem was caused by the lack of regulation. But that presumes that the oil companies would not perform proper maintenance and safety procedures unless forced to. It presumes that the default case in the absence of regulation would be that BP shirked their responsibility and allowed this spill to occur. MMS needed to have prevented the spill which BP would have otherwise caused. Which is an accurate view of reality, but the opposite of the anti-regulation philosophy.

So there are two ways in which the anti-regulation philosophy falls short. Blaming the lack of regulatory enforcement for the spill is a perfect example of how.

And as to why anyone would think new regulations would be enforced? I think they would be, provided the enactment of new regulations -- which suggests a belief that regulations are important just like the repealing of regulations suggests a belief that they are not -- is coincident with a housecleaning of MMS and the hiring of people who are not of the anti-regulation philosophy and a director-level-on-down belief that yes, these regulations are important.

Do I think this certainly will happen? Not at all. The firing of the MMS director is just the start of a long road I'm not sure they're going to walk down. However, arguing that because regulations we demonstrably need to make industry do the right thing may not be sufficiently enforced, is not a reason to not have the regulations! It's an argument to press the government to focus on making sure their agents do enforce them.

If you're anti-regulation, latching onto the failure of MMS to keep BP as evidence of your cause is the last thing you want to do.

Besides, compared to regular inspections of safety equipment and so on, simply regulating the need for relief wells to be pre-drilled in case all the other safety regulations aren't followed would be quite likely to succeed. Your general dislike of regulation does not outweigh the need for simple improvements like this.

Egregious

[eclectro]

There are faulty engineering and management decisions every step of the way when producing this well. This is not the first disaster for BP that ended in the loss of life. The question is why is there not criminal prosecutions for bad engineering that leads to the loss of life? Why is it that only people with guns who kill people get criminal prosecutions?

Re:Egregious

[Andy Dodd]

From most of what I've read, the subcontractors in question (Halliburton and Transocean) were doing the work, but BP had full control over the operations.

The flow was something like this:
Halliburton or Transocean: That's a bad idea, we don't recommend that.
BP: Do it anyway.
H/T: OK...

Although the question is at what point H/T should have said, "Hell no!"

Re:Egregious

[Anonymous Coward]

I found this episode of 60 minutes quite interesting:

http://www.cbsnews.com/video/watch/?id=6490509n&tag=api

Apparently, BP was putting on a lot of pressure to do things quickly, since they were running behind schedule and it was costing them money.

Specifically, on the day of the accident, there was an argument between representatives of Transocean and BP on how to close the well (in preparation for later exploitation by another ship). Transocean was in favor the slower, safer procedure. BP wanted things to be done more quickly. They did it the BP way, which was the point when the accident happened. So, according to this report, there were BP emplyes on the Deepwater Horizont, and they influenced the procedures by pressuring their subcontractors.

According to the report, several other things had to happen as well in order for things to go wrong so badly, but I would not so easily let BP of the hook.

Re:Egregious

[Achromatic1978]

Seriously, stop using Terry Childs as a posterboy for "Wronged Geek". He was an obstinate, self-serving asshole.

Protip for all you people saying "They could have reconfigured the routers, etc." (on Childs refusal to hand over passwords) - not so much.

Why? Because Childs had either disabled serial consoles, disabled password recovery, or configured devices to -never- save configuration, only to run in RAM.

Well, shit, you say, restore the config from backups. Guess what, SF owned no backups of the configuration files, or network maps. The only configuration files Childs kept were on his personal laptop, encrypted with a key known only to him, and configured such that his laptop was the only device capable of updating configs. Network maps? Same. Sitting on his personal laptop. Nowhere else.

The guy viewed SF's network as his personal playground, and believed no-one else worthy to take the reins of it - guess what, he had no authority to decide that, and when he got nicely obstinate about it, he crossed a fairly clear line in the sand.

Stop the martyred geek defending valiantly our security creed. It bares little resemblance to reality.

Re:Egregious

[shadowofwind]

What's it got to do with BP? The rig was owned and operated by a company called Transocean.

This is a common legal and accounting ploy: subcontract everything to other companies, then you're not responsible for anything, even though you're in charge of everything.

I recently worked for a company, run incidentally by the spouse of a BP chief executive, that sells a medical product for applications that the product can not legally be sold for (in the US). Its way around this is to create three companies, one for engineering, one for distribution, and one for marketing. That way, the parent company claims that its selling nothing illegally because it distributes nothing, but only provides information. And the distributor claims that it does not target its product for the illegal applications, since it merely distributes. And the engineering company evades FDA engineering process requirements by saying that it merely distributes the product made by the engineering company, which ignores the regulations because it is ostensibly not subject to regulation since it is not the distributor, and it doesn't have a distribution operation that can be shut down. But all three companies are essentially the same company, run by the same people. The 'ethic' involved is that if you haven't yet been sued successfully, or shut down by regulators, then its all good.

At least Halliburton and Transocean have a separate existence from BP. But BP is still responsible.

How much did they save?

[GrumblyStuff]

I mean, the whole rig's cost is in the hundreds of millions (Wiki says $560 mil but google link said $350 mil). The whole disaster is in the tens of billions, ain't it?

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.

Makes the $500,000 a day lease look like pennies.

Re:How much did they save?

[jd2112]

You'd think they would do anything and spare no cost to keep the fucking thing in working order and floating.Makes the $500,000 a day lease look like pennies.

In the corporate word, the important thing is to save money no matter how mutch extra it costs.

Re:Safety List

[Anonymous Coward]

Yes; steaks cost massive amounts of money, but what does that have to do with what's at stake?

Cutting corners is the name of the game

[plopez]

Cutting corners is the corporate way. I have seen so much "Mickey Mouse" stuff at places I've worked it disgusts me. Untrained workers, electrical boxes in pools of water, large pumps at refineries held in place by 4 bolts rather than the six bolts which were intended to be used etc. But of course, none of these problems are the CEO (or board members) of BP's fault. They only take credit when things go right. Avoiding responsibility is the name of the game.

Interesting

[Shulai]

Nobody is bashing Windows so far, yet it seems to be what the editor look for when he wrote the headline. Has Windows improved enough that nobody try to make fun of it anymore, or slashdotters are already older and more mature?

the regulators were the regulated

[circletimessquare]

the regulations don't matter in this case. i'm glad you admit we need some regulations, but the real issue here is regulator==regulated

http://www.washingtonpost.com/wp-dyn/content/article/2010/07/22/AR2010072205133.html?hpid=topnews

His statement came after Rep. Jackie Speier (D-Calif.) asked about a Washington Post article that reported that dozens of former Interior officials had crossed over into the oil industry and that three out of four industry lobbyists had once worked for the federal government.

The rate is more than double the norm in Washington, where industries recruit about 30 percent of their lobbyists from the government, according to data from the nonpartisan Center for Responsive Politics. With more than 600 registered lobbyists, the industry has among the biggest and most powerful contingents in Washington, The Post reported.

the lobbyists, the interior officials, the corporate assholes: all the same people

all the same smoochy same golf hole playing same bar attending backslapping crowd of assholes

that's why we had the disaster in the gulf

you can pass all the regulations you want, it doesn't matter if the ones who are supposed to be policing the industry ARE the industry

It was Windows NT

[Fookin]

I was watching the testimony and he stated that it was a Windows NT system and was constantly giving a BSOD. They had replaced and reimaged the HDD over and over but it still kept happening. There were new servers, workstations, etc standing by and waiting to be installed, but another problem creeped in. They were waiting for another ship to figure out a way to run the old software on the new machines. Once that other ship could get it working and document it, they would then do the replacement on their end. I'm guessing it was a Windows NT 4 workstation.

That's why capitalism is broken

[copponex]

A company we hired nearly destroyed the Gulf of Mexico... What's that got to do with us?

One our business partners was rating these bonds as AAA when they were worthless, and we were busy making billions passing the bonds off as good investments... What's that got to do with us?

The company we hired to dispose of this toxic waste is just dumping it in a river... What's that got to do with us?

In effect, modern capitalism is a system of mafia thugs and their hired patsies who operate technically within the law, as long as they hire an agent to do their dirty work to take the fall. Any of the real costs can be passed off to the public, either though bailouts or just ruining the commons.

Re:That's why capitalism is broken

[Rich0]

Agreed. In fact, as long as the big company has no liability it just turns into one big race for the bottom.

Suppose you run a reputable oil-rig operations company. You'd like to have people outsource their rigs to you. You believe in safety and the environment, so you take all kinds of steps to avoid something like the BP disaster. What happens? Well, you go out of business. You have to compete against other companies that cut corners. Companies like BP don't care about the safety of your workers or the environment, since that is on you. Your competitors charge less, and that is all they care about. Sure, it isn't sustainable, but most small companies aren't sustainable. The company running the rig can pay out dividends while the money is there, and then fold when the lawsuits hit. Indeed, if you didn't run a disreputable company your shareholders would probably fire you (low dividends compared to peers) and replace you with somebody who would mismanage it.

In other industries there is a clear assignment of responsibility, which cannot be outsourced. You can hire somebody else to do the work, but not to assume the liability. If Bayer sells a bottle of tainted aspirin, then they're liable even if they bought bad pills from a supplier. The only thing they're not liable for is what happens to the package after they sell it to the warehouse/store, although they are required to put the pills in tamper-evident packaging.

Indeed, in many industries liability is personal. That's why certified engineers have to sign off on bridges - they are personally responsible for the design (but not necessarily the implementation). I think the EU does something similar for drugs.