Slashdot Mirror
$600k Fine Over Data Center Death (datacenterdynamics.com)
judgecorp writes: UK contractors Balfour Beatty and Norland have been fined £380,000 ($580k) after an electrician was electrocuted while working on a data center owned by finance firm Morgan Stanley. The fine follows mounting concern that safety is being compromised because of the need for data centers to remain online non-stop. This leads to pressure for contractors to work on live power supplies.
How about a mandatory downtime for the data centre of say, 24 hours?
Hit 'em in the hip pocket - which is what a fine is supposed to do, but rarely, in the case of corporations, achieves its desired affect.
They sentenced me to twenty years of boredom
Isn't it sad that data center downtime is far more expensive than permanent downtime for a human being? This is just absolutely ridiculous and unjust. Someone needs to go to jail for this type of negligence.
The problem being, apparently, that nobody made "the decision". Due to lack of communication, one crew thought that status was A, and the other that it was B. Should you sent to prison the person who allowed live power in an area he thought there was nobody working, or the person who sent people to work where he thought the power was off? Or the bosses in the two different companies involved? Or the bosses in the employing company, a bank, the only place the chains of command met, who though they were employing competent contractors? The problem is that the structures were so confused that, though they didn't realise it, there was no-one in control. Finding someone guilty "beyond reasonable doubt" is almost certainly impossible.
Consciousness is an illusion caused by an excess of self consciousness.
Didn't RTFA but if companies want 24/7 99.9999% up-time. Then they better have paid for the all the stuff to do it safely including line techs trained and certificated for live work. If they cheaped-out they deserve the an even bigger fine. Yes, live work happens and sometimes for no good reason then trying to save a few bucks. But even power companies do live work on their critical infrastructure and even with best tools and training a life is lost here and there.
In my experience, the most likely person to pull a dangerous stunt like working on a live high-voltage feed is someone who's got all the certs and experience in the world and is working on top-flight gear - thus meeting your "paid for the all the stuff to do it safely" requirement - but they get careless "just this once" for whatever reason.
Why?
Because they're certain they know what they're doing, and they're certain the equipment is safe.
But they don't, and it isn't.
From reading TFA it seems he didn't know it was live because there were 2 companies at work without any single person being responsible for coordination.
He didn't sign a contract which said "you indemnify us even if we appoint a moron to direct your work which allows circuits to be energized completely willy nilly without a care of his workers". We don't allow contracts like that to be signed.
The 600k was the fine for non-compliance. You'd get that whether or not someone was killed. (some fines will get a bumper for injury, but not many have a bump for death for some reason)
Ret assured, there will be a multi-million dollar lawsuit filed by the family that will get settled out of court for an "undisclosed amount". (around 4 million is par) The fine was just the wakeup-call for the board to find a scapegoat to be the focus of the PR crucification and actual painful monetary loss for the impending lawsuit. The way things like this usually go, if the press doesn't dig up any real pattern of misconduct, there will probably just be someone issuing a public apology. If they do find a pattern, someone will get the axe.
Unfortunately, these places rarely get a fine unless someone is injured or killed, because nobody knows or cares about the noncomp until it hits the papers. Then the regs look bad if they don't step in and issue a fine like they ought to have done several times in the past to have, y'know, prevented this from happening in the first place.
But regardless of what happens, hopefully there will be changes made. From the looks of it, the tech that got killed was unaware that the wire that got him was energized, due to poor communication from his management, which appears to have been the result of poor communication from upper management and whoever was coordinating the work with the other group that was in charge of the deadly wire. So it's a bit early to be blaming the tech. Heck, he may have opened the box and tested it and found it wasn't connected yet and was safe to leave open, got to work, connecting it to something else, and half an hour later someone in another building lit the box up and the tech never knew what hit him. Things like that can happen when two different groups are working on connected systems and are unaware of each other and not keeping in communication as shared circuits are cut and energized.
I work for the Department of Redundancy Department.
The five nine uptime is not counting planned stops. So you can at a datacenter have a planned stop for a month and still conform to the contract. But it wouldn't make sense.
There is a reason why clustered systems are used - one node goes down another takes over. That's good enough to provide decent uptimes in most cases.
But today with virtual servers it's often one huge single server, and that's a single point of failure system even if the server itself may have built in redundancy there are always something that can fail.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
How much downtime was caused by ensuring the circuit was safe and removing the body?
If zero, SUE THEM INTO OBLIVION. Risking either this contractor unnecessarily (you could have just switched it off) or other workers and emergency workers (because you didn't switch it off after it had demonstrably killed someone).
If some downtime, then why couldn't you have done that to do the work?
Sorry, but I fail to see how the risk of a death and possible short-circuits, joined phases etc. because of working with the terminals live in any way "secures" uptime any more than scheduling proper downtime and having properly redundant systems.
You are just ask likely to bridge the WRONG circuit while working live, or causing a short, which will cause more damage and more downtime than just switching things off to do the work. And you guys have redundant power with UPS that you can bypass to work on the UPS, etc. if necessary? If not, that downtime isn't all that important to you anyway.
There's no excuse for this, hence the court fine. And you've got to be an idiot to knowingly let people work on a live multi-phase system. Hell, even a fused, RCD'd, single-phase can be bad enough.
In working in data centers, I can totally see how this happened. Reading the actual source article, it reads like they had already connected the first circuit, and he got popped while working on the second. I would assume they had shut off both, installed the first PSU, then probably someone turned them BOTH back on instead of just the first one. When he went to connect the second PSU...
These are the kinds of accidents proper "change control" is supposed to stop, it seems no one working there really knew the over-all implementation plan. At our local data center, we have actual licensed electricians for high DC stuff, they know to "never trust always test". Even though we contract all that out too, we try to make sure the people on the site are aware of these things via bright stickers, lock-outs, etc. I have no idea if they have required licensing and training for their "cable jointer" positions in the UK.
"Ret assured, there will be a multi-million dollar lawsuit filed by the family that will get settled out of court for an "undisclosed amount". (around 4 million is par)"
Very unlikely, it was in the UK, we don;t get settlements anywhere near that level
The problem being, apparently, that nobody made "the decision". Due to lack of communication, one crew thought that status was A, and the other that it was B. Should you sent to prison the person who allowed live power in an area he thought there was nobody working, or the person who sent people to work where he thought the power was off?
Did the company have a lockout procedure? If yes, was the procedure followed and if not, why? If the answer to the first is "no" or the second is "employees weren't trained or improperly trained" then that company is liable for the worker's death.
The only thing necessary for evil to triumph is for it to be pitted against a slightly greater evil
The company was certainly liable, and has been fined. The question is, was any person liable, to the extent that the could be imprisoned for manslaughter?
Consciousness is an illusion caused by an excess of self consciousness.
If he was not a competent person then he had no business attempting the work.
The sad truth is, that the incompetent don't realise that they're not competent at the job.
From the press release from the HSE he wasn't working on live equipment.
He was given a work permit by the site operator to route some power but the route was through other equipment that had exposed live connections. The electrician made inadventent contact (with his forehead) with one of these exposed live connections.
Worse, the site operator was aware of this risk, and was aware that whether these exposed terminals were live or not was not under their control but they had a disregard for the risks when issuing the work permit. They got fined £100K for this. (The other company got fined nearly three times as much - presumably for leaving exposed live connections that could accidentally be touched and not marking the area as restricted/dangerous.)
Basically this seems to have been a mega screwup between the management of two companies working on the same site who appear to have had no way to safely coordinate work where the areas of responsibility overlapped.
God said, "div D = rho, div B = 0, curl E = -@B/@t, curl H = J + @D/@t," and there was light.
Infrastructure availability should be 4-5 9's for a tier IV facility-- planned and unplanned downtime. Unfortunately, the project in question appears to be a 2N upgrade, which tend to be the most risky projects if done online. A Tier III or Tier II system cannot be safely upgraded online, especially at 400V. It is marginally more practical to do at 208V, but proper safety procedures are essential. You use insulating blankets to safe off any live parts, gear up in the space suits, etc... and it can be done.
In the US though the fines would be huge. It will eventually lead to either IT failover solutions to remote sites that are 100% reliable, or 3N distribution systems.
Today though everybody thinks they can design/build/operate a data center. On the last note, I know JLL does a better job at mission-critical than CBRE, and they charge more because of it. The added value of designers today is almost nill-- everybody thinks it is easier than it is because it has become a commodity.
Isn't it sad that data center downtime is far more expensive than permanent downtime for a human being?
What's sad about it? Downtime creates downtime for other human lives too. At some point, you have to acknowledge that this is a trade off, a person assumes risk to their own lives in order to make other peoples' lives better or more productive.
The mideast called - the said you were the ideal candidate for their new landmine detection program.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
Except that I am sure that MS is not on the hook for this at all.
That's WHY they use contractors.
Same thing with cell tower owners like Verizon... they own the them but they contract out all the work on them and are not liable for any accidents...
My eyes reflect the stars and a smile lights up my face.
As I read the article the accident was caused by a screw up in communications by the contractor doing the work. They tried to blame it on the data center pressuring them, but the judge apparently didn't accept that argument and fined the contractor. It seems submitter is looking for evil where there is really just incompetence.
Neither do we for the most part, at first the sympathetic jury awards the big-bucks, which is then paid out as say 4 Million over the life of a 30 year structured settlement. Later after the elections are over, and the emotions die down the award is reduced on appeal.
Apocalypse Cancelled, Sorry, No Ticket Refunds
Isn't it sad that data center downtime is far more expensive than permanent downtime for a human being?
What's sad about it? Downtime creates downtime for other human lives too. At some point, you have to acknowledge that this is a trade off, a person assumes risk to their own lives in order to make other peoples' lives better or more productive.
The mideast called - the said you were the ideal candidate for their new landmine detection program.
Because what I wrote was an appropriate if mild response to what you wrote, oh brittle and pathological person.
So tell me, are you willing to have a person die for some database that you find important?
Your post at top tells us you are perfectly fine with people dying so that other peopple can have some file stored in a dat center. Uptime to you is more important than someone else's life.
And the better question is - are you willing to cease existance for some file someone else finds important?
Probably a different answer, isn't it?
Hence I was letting you off easy, and not saying what I think of you, because your sociopathy Is rather disgusting. Not that you give a damn about anyone else but yourself.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
making middle-class living while risking your life for maximum profits of others is pretty grotesque, and should be illegal.
Point to someone who's doing that and we'll see if you know what you're talking about.
Why do you want them to kick off so you aren't inconvenienced?
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.