Slashdot Mirror
$600k Fine Over Data Center Death (datacenterdynamics.com)
judgecorp writes: UK contractors Balfour Beatty and Norland have been fined £380,000 ($580k) after an electrician was electrocuted while working on a data center owned by finance firm Morgan Stanley. The fine follows mounting concern that safety is being compromised because of the need for data centers to remain online non-stop. This leads to pressure for contractors to work on live power supplies.
How about a mandatory downtime for the data centre of say, 24 hours?
Hit 'em in the hip pocket - which is what a fine is supposed to do, but rarely, in the case of corporations, achieves its desired affect.
They sentenced me to twenty years of boredom
Yeah, a more apt punishment could have been to force management to touch a 300V power line.
Isn't it sad that data center downtime is far more expensive than permanent downtime for a human being? This is just absolutely ridiculous and unjust. Someone needs to go to jail for this type of negligence.
The problem being, apparently, that nobody made "the decision". Due to lack of communication, one crew thought that status was A, and the other that it was B. Should you sent to prison the person who allowed live power in an area he thought there was nobody working, or the person who sent people to work where he thought the power was off? Or the bosses in the two different companies involved? Or the bosses in the employing company, a bank, the only place the chains of command met, who though they were employing competent contractors? The problem is that the structures were so confused that, though they didn't realise it, there was no-one in control. Finding someone guilty "beyond reasonable doubt" is almost certainly impossible.
Consciousness is an illusion caused by an excess of self consciousness.
Working live is recognized as an acceptable risk in circumstances where it is not practicable to turn stuff off (UPS's and backup-generators may have made this a logistics nightmare even if you DGAF about uptime.) There are qualifications you can get to prove your competence to work live, and there are some very comprehensive procedures to follow that make live working a reasonably safe undertaking (even for the boys doing live joints on HT cables)
If the guy was a "competent person" in the eyes of the law, then he was perfectly within his rights to take the decision and was fully and personally responsible for any consequences of doing so. If he was not a competent person then he had no business attempting the work.
I'm a UK fully qualified electrician, I know there are plenty of lads who have been "on-the-tools" since they left school and call themselves electricians, but those of us who actually have the relevant bits of paper can enjoy a great deal of professional autonomy and responsibility, and I find this decision insulting.
Didn't RTFA but if companies want 24/7 99.9999% up-time. Then they better have paid for the all the stuff to do it safely including line techs trained and certificated for live work. If they cheaped-out they deserve the an even bigger fine. Yes, live work happens and sometimes for no good reason then trying to save a few bucks. But even power companies do live work on their critical infrastructure and even with best tools and training a life is lost here and there.
In my experience, the most likely person to pull a dangerous stunt like working on a live high-voltage feed is someone who's got all the certs and experience in the world and is working on top-flight gear - thus meeting your "paid for the all the stuff to do it safely" requirement - but they get careless "just this once" for whatever reason.
Why?
Because they're certain they know what they're doing, and they're certain the equipment is safe.
But they don't, and it isn't.
You are an idiot and are COMPLETELY missing the point. The amount is not for just any death. The amount is for enforcing/encouraging or even just allowing a work environment that is unsafe just so that the bottom line doesn't get affected.
That kind of negligence needs to hurt. $600K doesn't even cause a twinge of pain. I am sure the amount of money they were trying to save by not allowing the work to be done safely was a lot more than that.
Even if they don't know what they're doing, they are at least getting hazard pay.
With Load balancing, fail over clusters, hot sites, all the thing that can make part of a site go offline for a while without serious impact.
He didn't sign a contract which said "you indemnify us even if we appoint a moron to direct your work which allows circuits to be energized completely willy nilly without a care of his workers". We don't allow contracts like that to be signed.
The 600k was the fine for non-compliance. You'd get that whether or not someone was killed. (some fines will get a bumper for injury, but not many have a bump for death for some reason)
Ret assured, there will be a multi-million dollar lawsuit filed by the family that will get settled out of court for an "undisclosed amount". (around 4 million is par) The fine was just the wakeup-call for the board to find a scapegoat to be the focus of the PR crucification and actual painful monetary loss for the impending lawsuit. The way things like this usually go, if the press doesn't dig up any real pattern of misconduct, there will probably just be someone issuing a public apology. If they do find a pattern, someone will get the axe.
Unfortunately, these places rarely get a fine unless someone is injured or killed, because nobody knows or cares about the noncomp until it hits the papers. Then the regs look bad if they don't step in and issue a fine like they ought to have done several times in the past to have, y'know, prevented this from happening in the first place.
But regardless of what happens, hopefully there will be changes made. From the looks of it, the tech that got killed was unaware that the wire that got him was energized, due to poor communication from his management, which appears to have been the result of poor communication from upper management and whoever was coordinating the work with the other group that was in charge of the deadly wire. So it's a bit early to be blaming the tech. Heck, he may have opened the box and tested it and found it wasn't connected yet and was safe to leave open, got to work, connecting it to something else, and half an hour later someone in another building lit the box up and the tech never knew what hit him. Things like that can happen when two different groups are working on connected systems and are unaware of each other and not keeping in communication as shared circuits are cut and energized.
I work for the Department of Redundancy Department.
Both those people had people on site at the top of the totem pole. They might not have realized that no one was in control, but they were certainly paid to be.
Pull their certs.
Should be a 600kV line. That would be more stunning.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
The five nine uptime is not counting planned stops. So you can at a datacenter have a planned stop for a month and still conform to the contract. But it wouldn't make sense.
There is a reason why clustered systems are used - one node goes down another takes over. That's good enough to provide decent uptimes in most cases.
But today with virtual servers it's often one huge single server, and that's a single point of failure system even if the server itself may have built in redundancy there are always something that can fail.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
How much downtime was caused by ensuring the circuit was safe and removing the body?
If zero, SUE THEM INTO OBLIVION. Risking either this contractor unnecessarily (you could have just switched it off) or other workers and emergency workers (because you didn't switch it off after it had demonstrably killed someone).
If some downtime, then why couldn't you have done that to do the work?
Sorry, but I fail to see how the risk of a death and possible short-circuits, joined phases etc. because of working with the terminals live in any way "secures" uptime any more than scheduling proper downtime and having properly redundant systems.
You are just ask likely to bridge the WRONG circuit while working live, or causing a short, which will cause more damage and more downtime than just switching things off to do the work. And you guys have redundant power with UPS that you can bypass to work on the UPS, etc. if necessary? If not, that downtime isn't all that important to you anyway.
There's no excuse for this, hence the court fine. And you've got to be an idiot to knowingly let people work on a live multi-phase system. Hell, even a fused, RCD'd, single-phase can be bad enough.
In working in data centers, I can totally see how this happened. Reading the actual source article, it reads like they had already connected the first circuit, and he got popped while working on the second. I would assume they had shut off both, installed the first PSU, then probably someone turned them BOTH back on instead of just the first one. When he went to connect the second PSU...
These are the kinds of accidents proper "change control" is supposed to stop, it seems no one working there really knew the over-all implementation plan. At our local data center, we have actual licensed electricians for high DC stuff, they know to "never trust always test". Even though we contract all that out too, we try to make sure the people on the site are aware of these things via bright stickers, lock-outs, etc. I have no idea if they have required licensing and training for their "cable jointer" positions in the UK.
"never trust always test" would have saved his life.
"Ret assured, there will be a multi-million dollar lawsuit filed by the family that will get settled out of court for an "undisclosed amount". (around 4 million is par)"
Very unlikely, it was in the UK, we don;t get settlements anywhere near that level
The problem being, apparently, that nobody made "the decision". Due to lack of communication, one crew thought that status was A, and the other that it was B. Should you sent to prison the person who allowed live power in an area he thought there was nobody working, or the person who sent people to work where he thought the power was off?
Did the company have a lockout procedure? If yes, was the procedure followed and if not, why? If the answer to the first is "no" or the second is "employees weren't trained or improperly trained" then that company is liable for the worker's death.
The only thing necessary for evil to triumph is for it to be pitted against a slightly greater evil
The company was certainly liable, and has been fined. The question is, was any person liable, to the extent that the could be imprisoned for manslaughter?
Consciousness is an illusion caused by an excess of self consciousness.
600k is nary a blip on MS' radar. That's not a punishment.
Agreed, it's not even a rounding error to those guys.
Just cruising through this digital world at 33 1/3 rpm...
HFT Kills.
Yes, it's sad.
aaaaaaa
Management doesn't understand the difference between telling them it's technically possible to do live maintenance and that it's a challenge like the rest of our technical feats. I feel for the guy and his survivors. I've seen the same pressures play out on my data centers, but thankfully we were able to arrange that type of work as semi-regular full DR test and the place I was working at simply didn't have the same uptime demands when push came to shove. From a safety angle, I can't help but think that moving over the DC voltage for datacenters might be a better option. Some of the big boys are already doing it as we're currently taking AC power, running it through or along side DC power backups, then up to AC for power supplies, then DC for the servers. There's an economy to it, but DC is just safer to work with. That said, there's always going to be an AC/DC bus that's dangerous to work on and live electrical work on it will always take appropriate design ($$) and proper training.
The DC bus is likely at a lower voltage than typical AC circuits, therefore less able to cause electrocution.
It's even sadder than that: The extra cost of hiring people who were competent enough to recognise and mitigate the risks, would've likely been less than $580k.
In these harsh economic times, it's really a race to the bottom.
Industry has already developed safety protocols to address this (LOTO) and three way communications to lower the risk of misunderstandings.
Trolling?
I tend to doubt that the terms of the job stated that there would be bare wires running through the data center. Nobody hired to work as a maintenance person would reasonably expect that such a hazard existed. The idea that he was completely aware of these risks and took the job anyway is speculative nonsense.
Furthermore, if you RTFS, this was a FINE imposed on the company, not the result of a lawsuit by the deceased's family. I'm sure that the UK has workplace safety regulations which prohibit exposed high voltage wires, reenforcing the idea that a worker would not anticipate finding one.
Infrastructure availability should be 4-5 9's for a tier IV facility-- planned and unplanned downtime. Unfortunately, the project in question appears to be a 2N upgrade, which tend to be the most risky projects if done online. A Tier III or Tier II system cannot be safely upgraded online, especially at 400V. It is marginally more practical to do at 208V, but proper safety procedures are essential. You use insulating blankets to safe off any live parts, gear up in the space suits, etc... and it can be done.
In the US though the fines would be huge. It will eventually lead to either IT failover solutions to remote sites that are 100% reliable, or 3N distribution systems.
Today though everybody thinks they can design/build/operate a data center. On the last note, I know JLL does a better job at mission-critical than CBRE, and they charge more because of it. The added value of designers today is almost nill-- everybody thinks it is easier than it is because it has become a commodity.
Isn't it sad that data center downtime is far more expensive than permanent downtime for a human being?
What's sad about it? Downtime creates downtime for other human lives too. At some point, you have to acknowledge that this is a trade off, a person assumes risk to their own lives in order to make other peoples' lives better or more productive.
The mideast called - the said you were the ideal candidate for their new landmine detection program.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
This is no different than any murder investigation and the police should find out who exactly made the decision and make them pay for it.
I feel bad for him, and I hope his family gets a decent settlement*, but have to ask why didn't he check the live line first? When I work on electrical stuff at home, I always check AND DOUBLECHECK that the breakers are off and that no juice is running. And, that's with voltages that won't necessarily kill me. Working with this level of voltage? Holy crap.
* As stated by some in the UK, it's not likely to be a big settlement. Too bad in this case.
I imagine there has been such an investigation. But first, you have to decide what was "the decision" that caused the accident. In aviation they say that an incident is none things going wrong at once, an accident is ten. If five people each believe that one of the other four had put the safety locks in place, and none had, which of the five made the decision which caused the accident?
Consciousness is an illusion caused by an excess of self consciousness.
It's even sadder than that: The extra cost of hiring people who were competent enough to recognise and mitigate the risks, would've likely been less than $580k. In these harsh economic times, it's really a race to the bottom.
But now the Holy Dollar rules everybody's lives
Gotta make a million, doesn't matter who dies
"What the American public doesn't know is what makes them the American public." -Ray Zalinsky (Tommy Boy)
Except that I am sure that MS is not on the hook for this at all.
That's WHY they use contractors.
Same thing with cell tower owners like Verizon... they own the them but they contract out all the work on them and are not liable for any accidents...
My eyes reflect the stars and a smile lights up my face.
600k is nary a blip on MS' radar. That's not a punishment.
Yes, but if TFA is correct and the person who died was not informed of the live circuit status, you can add two zeroes to the end of that number when the wrongful death lawsuit is filed.
As I read the article the accident was caused by a screw up in communications by the contractor doing the work. They tried to blame it on the data center pressuring them, but the judge apparently didn't accept that argument and fined the contractor. It seems submitter is looking for evil where there is really just incompetence.
Neither do we for the most part, at first the sympathetic jury awards the big-bucks, which is then paid out as say 4 Million over the life of a 30 year structured settlement. Later after the elections are over, and the emotions die down the award is reduced on appeal.
Apocalypse Cancelled, Sorry, No Ticket Refunds
Unfortunately, the case appears to be improper Energized Electrical Work (EEW) Google EEW hotwork. There are many cases where EEW is performed. Common applications include linemen replacing cutouts, changing insulators, installing new cutouts for new home construction, etc. EEW hotwork requires special permits and tools and protective clothing.
Death and resulting fines was due to the failure of following proper proceedures for EEW hotwork. LOTO is preferred over EEW, but there are reasons to do EEW. Only those prpoerly trained and follow the permitting process may do EEW.
Google EEW Permit for more info. Many pepole are killed as in some codes 600V and under is considered LOW Voltage, not to be confused with Limited Energy Class 2.
The truth shall set you free!
The UK has victim surcharges for most crimes. Those go directly to the victim and/or those who are victim of the same or similar crimes.
And it's a statutory fine for noncompliance in H&S. Now that it's been established by a court that the companies were negligent, the FAMILY can individually sue on that basis to receive compensation for their particular consequences.
The fact of law for these exact circumstances has been established by a professional body in court. Now the family don't have to pay lawyers to do that part, they can just sue using their precedents on this incident.
More likely, the company or its insurers will now hastily settle such claims out of court to avoid additional legal expense. That won't come cheap, but cheaper than fighting the family with lawyers when courts have already ruled you should never have allowed it to happen.
If anything, this is a case where the government bodies have saved the family lots of unnecessary grief, hassle and expense of their own.
Medium voltage electricians are expected to be able to perform work under such conditions. He fucked up, now he's dead.
What I find sad is that shutting down one of the power sources would cause downtime, who designs a data center with a single power network?
There are power line load balancers for hardware that doesn't allow multiple power input, but most equipment allows for multiple power sources to decrease downtime. Why would this guy need to work on a live circuit?
APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
Not working directly in a data center, what the hell is 2N or 3N?
I am wondering how the hell a datacenter like this could possibly not have redundant power to allow for one of the power sources to go down for maintenance.
APK likes to ask for responses to the same things over and over. Maybe he just likes the responses?
Yes, but if TFA is correct and the person who died was not informed of the live circuit status, you can add two zeroes to the end of that number when the wrongful death lawsuit is filed.
This was not in the USA, so those kind of damages won’t be given out.
making middle-class living while risking your life for the lives and safety of others is at least altruistic and noble.
making middle-class living while risking your life for maximum profits of others is pretty grotesque, and should be illegal.
making middle-class living while risking your life for maximum profits of others is pretty grotesque, and should be illegal.
Point to someone who's doing that and we'll see if you know what you're talking about.
I don't understand why you would need to work on a live rain. Seems to me like a very badly designed system ..
Not really. First of all he wasn't an electrician, he was something called a cable jointer; it doesn't require a license and normally they don't work on live stuff. Secondly, according to TFA he wasn't aware the line was live, which means someone in a supervisory position screwed up.
As to whose fault it is, that depends. If the company in charge of the site has a proper lock out/tag out policy and training program, it's probably some supervisor's fault -- unless the worker went into an area specifically against instructions in which case it's the worker's fault. If the company doesn't have a proper LOTO program (including training for all personnel on site) it's the company's fault.
LOTO may be a PITA, but it's not rocket science.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Any competent, licensed, journeyman or master electrician would take responsibility for his safety in a number of ways: often, by disabling power to a box and applying a padlock so no other worker could reenable that power. It isn't clear from the article what the issue was, but the citation that went with the fine indicates a failure of an extended team to communicate. Some of the people involved knew that the box had live wires, while the victim did not, perhaps? Did the victim have his own electrical tester, or was he a semiskilled helper?
We risk our lives for stupid things all the time. It's a matter of how big the risk is. I got in my car and went to my local coffee shop to grab a cup of coffee earlier today. I risked my life by getting into a steel box with wheels and going out on the road so I could get out of the house and enjoy some coffee. But the risk was known and I found it to be worthwhile.
The problem here isn't that somebody was doing risky electrical work for money. Electrical work is dangerous by its nature, and electrical work for commercial offices is for somebody else's profit by its nature. There's nothing wrong with that on its own. The industry knows what the best practices for safety are, and they know how much risk you're exposed to when you follow those best practices (and it's *never* zero). The problem here was a company not following those best practices and taking on added risk.
Back around 2000, I worked for a company doing power distribution software. One of the EE-type guys there told us that, in Britain, the electrician is supposed to go by what he's told as to whether a line is hot or not. (This was motivation to do our job well, since it kept track of things like lines and what was powered.) In the US, if we screwed up, the electrician was going to take his or her own precautions.
"When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
Isn't it sad that data center downtime is far more expensive than permanent downtime for a human being?
What's sad about it? Downtime creates downtime for other human lives too. At some point, you have to acknowledge that this is a trade off, a person assumes risk to their own lives in order to make other peoples' lives better or more productive.
The mideast called - the said you were the ideal candidate for their new landmine detection program.
Because what I wrote was an appropriate if mild response to what you wrote, oh brittle and pathological person.
So tell me, are you willing to have a person die for some database that you find important?
Your post at top tells us you are perfectly fine with people dying so that other peopple can have some file stored in a dat center. Uptime to you is more important than someone else's life.
And the better question is - are you willing to cease existance for some file someone else finds important?
Probably a different answer, isn't it?
Hence I was letting you off easy, and not saying what I think of you, because your sociopathy Is rather disgusting. Not that you give a damn about anyone else but yourself.
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
making middle-class living while risking your life for maximum profits of others is pretty grotesque, and should be illegal.
Point to someone who's doing that and we'll see if you know what you're talking about.
Why do you want them to kick off so you aren't inconvenienced?
The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
Balanced PDU's are very expensive when compared to standard units.
errr....umm...*whooosh* *whoosh* Is this thing on ?
The company did have a lockout procedure and a "permit to work" procedure. The problem was that there were two different companies working on the same site and they failed to have a proper protocol when one needed the other to disconnect power.
The facility had a single power feed, and the routine maintenance and supervision of the electrical system in the building was outsourced to a building management company. The provision of a second supply and associated switching system was being performed by a specialist electrical contractor.
The time came to disconnect connect the IT loads from the incumbent single-feed switchboard to the dual-feed switchboard during a period of scheduled downtime. The building management contractor issued a "permit to work" to a cable jointer to disconnect a sub-switchboard from the main supply and connect it to the new dual-feed switchboard; the permit certified that the main supply had been turned off and locked out, and would only be unlocked upon return of the permit.
The original plan had been to connect the IT loads first, then connect power to the new switchboard. However, because of a specification error when procuring the switchboard, it had been modified on-site, and following modification required live tested prior to connection of any IT equipment. As a result, it had already been connected to both the existing supply (under the management of the facilities management company), and also to the new supply (which had not yet been handed over, and remained under the control of the installing contractor), before any outgoing cables were connected.
While the building management company had disconnected the building's main supply and locked it off prior to issuing the permit to work on the switchboard, they had failed to contact the contractor handling the 2nd supply and failed to ensure that the 2nd supply was also locked out. As a result, when the cable jointer set to work, the switchboard was still energised by the 2nd supply. Although the connections that the jointer was working on were dead and isolated by a switch in the switchboard, as the switchboard was open, he accidentally contacted the busbars fed from the 2nd supply and was electrocuted.
As stated above, my scenario is it was dead when he started, and someone cut power back in during his work. I'm going on the assumption the tech was following good practices and that management/communication was where the fault occurred, based on the background provided in the article.
But the correct way to address that particular risk is to tie said source wires to ground with heavy cable. That way if some clown does light up the wires while you're working, the worst you're going to get is a bath of copper sparks before a breaker trips somewhere in the line. (unpleasant to be certain, but not lethal)
If I was going to be working in an environment where I was trusting somoene possibly a long distance away to keep their fingers off the knife-switch while I worked, (where driving to another location to install a lockout may not be practical) I think webbing the lines to ground while working on them is a precaution I would be consistently taking. I've read too many stories about clueless idiots blindly switching power back on when they stumble across a switch or breaker that's off.
I work for the Department of Redundancy Department.
Back around 2000, I worked for a company doing power distribution software. One of the EE-type guys there told us that, in Britain, the electrician is supposed to go by what he's told as to whether a line is hot or not. ...
Britain is a conqured country owned by invading generals, and has been since 1066 AD ! 8-)
Get yourself a set of padlocks and tags!
oh, so he did an electricians job, not being one, and killed himself
One would think that the first day of Electrician 101 covers the idea of checking vs assuming.