Slashdot Mirror
Three Mile Island Memories
theodp writes "Thirty years after the partial nuclear core meltdown at Three Mile Island, Robert Cringely describes the terrible TMI user interface, blaming a confluence of bad design decisions — some made by Congress — for making the accident vastly worse. While computers could be used to monitor the reactor, US law prohibited using computers to directly control nuclear power plants — men would do that. So, when the (one) computer noticed a problem, it would set off audible and visual alarms, and send a problem description to a line printer. Simple, except the computer noticed 700 things wrong in the first few minutes of the TMI accident, causing the one audible alarm to ring continuously until it was shut off as useless. The one visual alarm blinked for days, indicating nothing useful. And the print queue was quickly flooded with 700 error reports followed by thousands of updates and corrections, making it almost instantly hours behind. The operators had to guess at what the problem was."
So the problem with Three Mile Island (TMI) was Too Much Information (TMI). But I didn't read the article, as that would have been TMI.
Never has the gravity of an accident (of any kind) been so exaggerated. Before or after.
"The agriculture ministry is not in charge of Gundam" - Japanese ministry official.
Our President at the time, Jimmy Carter, was also a micro-manager and a former nuclear engineer:
U.S. Navy reactor operators, the sort who served under Jimmy Carter in the 1950s,
Is not and never was a nuclear engineer, much less did he command a nuclear sub. He served as an enlisted man on several diesel-electric subs and started, but did not complete, a Naval class in nuclear engineering. He resigned from the Navy (as a lieutenant) before any nuclear subs were commissioned.
The FEMA guys were just plain stupid.
NO U
U.S. Navy reactor operators, the sort who served under Jimmy Carter in the 1950s, were selected primarily for their temperament. ... their Navy job--as at TMI--was to follow the manual. All knowledge was inside the book. So knowing the book was everything. Unfortunately knowing the book isn't the same as knowing the reactor. So knowing the book was everything. Unfortunately knowing the book isn't the same as knowing the reactor.
No. Just fucking no. There's a significant (and necessary) emphasis on following procedures and getting the books out for any planned change to the plant to make sure you're doing things right. But Cringely makes it sound like nuclear operators are just slightly trained mouth-breathers that only know how to look things up in the book and do what it tells them. I can't speak for the civilian training, but the Navy does NOT do things that way.
When something goes wrong, they depend on you having enough internalized knowledge about the plant, its controls, and its indicator systems to work out what's going on and (if necessary) do something about it. Once you've got stuff at least marginally under control, *then* you get the books out to check the applicable procedures to make sure you haven't forgotten something, and to figure out how to recover from whatever happened without causing any more problems.
The Navy puts a lot of effort put into making sure their operators know how and why things work the way they do. They would never have got to the 21st century with the track record they have if all they did was train people to look at the book.
[b.belong('us') for b in bases if b.owner() == 'you']
I don't blame the UI at all. I bleme the belief that the goal of an UI is to lower the required understanding (and thus salary) of the operators.
How the UI worked is irrelevant. Operators who understood what they were doing would have checked what needed to be checked, and taken the precautions the situation warranted, no matter what kind of warnings were lost because of a bad UI.
Alas, the way for an electric company CEO to get big bonuses isn't by spending more money on smart people, but cutting costs which makes the short-term investors happy. So they spend $50k on an idiot-proof interface, and hire an idiot. The problem is that Nature is a whole lot better of churning out interface-proof idiots than programmers are at making idiot-proof interfaces.
It's high time that Western society started valuing knowledge and understanding again, and not just ability to study for requirement tests. Reinstate the journeyman/master system and accredited guilds, and ditch college diplomas as the worthless piece of gilded paper they are.
If you read the article, you'd realize it was a very significant wake up call. Death was narrowly avoided because the reactor containment vessel was over-engineered compared to the typical design. The tragedy is that the lesson the public learned was that nuclear power was too dangerous to use at all, when the reality was that it was poorly designed and mismanaged.
If your user interface lags behind by two hours and the UI is the only way to find out about the extremely complicated and intricate details coming out of a myriad of sensors that are inaccessible to people for safety reasons... I suppose you might be entirely wrong.
In this case, yes, the user interface was necessary for the operators to do their job. Are you going to tell me that submarine operators should rely on their "gut feeling" rather than a measurement of external pressure or depth to determine whether the submarine is safe? These are jobs that can't be done by even the most skilled operator because the information is completely walled off from them for the safety and integrity of the facility.
As far as I can tell, you're advocating that we should hire psychics to determine the safety of the nuclear plant and pay them exorbitantly because spending a single dime on a good interface is wasted money. Sometimes, a $50,000 idiot proof interface is exactly what's called for, rather than intentionally using outdated technology and hoping a printer will provide information fast enough to prevent imminent disaster.
A nuclear plant isn't like a gas plant where you can turn off the tap.
If you have a nuclear reaction that is going out of control, then you have to get it in control. Shutting the plant down would mean you don't have the ability to use things like the control rods to do this.
I wasn't there so I can't say Cringely is wrong about the government regulation of nuclear power, however, I have worked in the semiconductor industry which utilizes some of the deadliest chemicals known to man and their are mandated regulations from various government agencies, EPA, OSHA, etc., that result in the controls, interlocks, and containment systems used to make the industry safe. I'm also pretty sure that the issue in Bhopal was more a lack of regulation than a lack of respect for the dangers. There should have been powerful laws and inspectors to shut down the plant before it killed thousands.
Where we both do agree is on the belief that we can expect more Bhopal and economic melt down events due to bean counter management. Over the past 20 years I've noticed a managerial shift towards a focus on cutting costs and less of a focus on the technology and science behind the manufactured products. In the past two years I've engaged in heated debates with peers and managers over the purpose and focus of engineering resources. Its seems that decision makers are forgetting that the core of a technology based manufacturing corporation is the technology not the cutting of fixed costs by reducing head count, wages, service contracts, etc. Accounting and business management are tools to support the core skills, they are not the core themselves. When accounting and business management undermines the ability of a technology based business to develop and manufacture the core technology of their business you can expect a gradual degradation of the business until it is no longer viable.
Don't let Cringely convince you that he actually knows anything about nuclear power plants--those guys had a whole room full of alarms, gauges, meters, etc., giving them a lot of info about the whole plant.
Shutting down the reactor could probably have been done by the operator within a couple of seconds by flipping a switch. IIRC, though, the automatic safety system shut it down at the beginning of the incident because it detected a situation that warranted it.
[b.belong('us') for b in bases if b.owner() == 'you']
Comment removed based on user account deletion
I think you got it backwards. They didn't want to withold information from humans or remove control from them, so they didn't automate enough and the humans in the loop got swamped with more than they could handle.
Sounds like it was engineered just right. Bean-counters often use "over-engineered" when something is built to withstand the rare but serious malfunctions. Instead, they'd rather things be built to be "good enough" to run fine most of the time. Problem is, a minor issue can become a critical one if you don't build your devices to withstand the rare but serious issues. For example, a failover server setup is 100% overbuilt...until the primary fails.
But it wasn't engineered this way to secure it against a partial meltdown. It was above average for reactor containment vessels actually in use at that time, and the average containment vessel would have failed. The only reason it was able to withstand it was that it happened to be on the final approach path of a former airforce base, and had originally been engineered to withstand a bomber crashing into it.
God, I wish I had mod points for you.
... how much nuclear power is involved with Centralia? Ummmm.... NONE! A natural resource (accidentally ignited by humans) has destroyed a town completely. Personally, I put Centralia on a higher level of "disaster" than I do TMI.
I live about 15 miles away from TMI and I have for 20 years. I've never felt unsafe or felt like I was in danger. People seems to enjoy comparing TMI to being a potential Chernobyl, but there's simply no way that the two can even be compared.
On the other hand, head up to Centralia, PA where the whole town has been demolished because of a fire that has been running through the ignition of a natural, coal vein. A fire ignited some coal, and now the whole town has been abandoned, homes have been razed, there are very few buildings to speak of, there are dangerous leaks of carbon monoxide and other lethal gases, the ground has swelled and cracked from the heat, and this fire is expected to last 250 years.
Now
The Overrated mod is for reversing inappropriate, positive mods, not for voicing disagreement with a post.
As I tire of pointing out and people never tire of not understanding, lack of regulation does not mean free-for-all, might is right or whatever.
An unregulated nuclear industry does not mean plants can pour waste in other people's property. Since governments regulate commons they must either take responsibility to ensure they are not destroyed or privatize them to internalize the externalities.
Send your spendthrift head of state this
I bleme the belief that the goal of an UI is to lower the required understanding (and thus salary) of the operators.
Why do you hate stockholders?
Linux, you magnificent bastard, I read the fucking manual!
Elizabethtown College - about 7 miles from TMI as the crow flies (or the wind blows).
My sister went to school there, and after two and a half semesters there she was diagnosed with thyroid cancer.
Lets rewind... The morning of the accident people reported a metallic taste in the air. Turns out that of the gas released, radioactive iodine accounted for an estimated "8 - 12% of the total gases released, implying a minimum of 1 million iodine curies". See: TMI Accident.
I don't believe that people were not harmed by the radioactive release. In fact, stating that it "had little to no effect on the health of people" is a lie. Any arguments based on that lie are faulty.
I am for nuclear power, and I agree that letting bean counters manage a project like that is the wrong way. I'm in agreement with all of the other people here who think that engineers should be listened to. They are the ones with the knowledge after all...
See, See. UI is important!!!!
I'm a nuclear engineer and I think the use of the term UI for the control room is somewhat 'simplistic'. I personally think a major issue was over design in a certain area (redundant alarms), and lack of safety systems that would prevent the core from melting even with a LOCA in place. It was two hours after the shutdown when the fuel melting began at TMI-2. This was a scenario where the operators couldn't understand what was happening. Now from an operator's perspective (who sits in the operator room) you're not looking at a "UI" in the traditional CS sense. Here is an image of a control room: http://www.ornl.gov/info/ornlreview/v38_1_05/images/a11_controls_full.jpg The events leading up to the disaster started on the secondary side (non-core) leading to a LOCA (Loss of Coolant Accident). For those unfamiliar with the term "secondary side". The secondary side of a Nuclear Power Plant is similar to that of any power generating plant, meaning the secondary side does not contain the reactor core.
Maybe they do understand but don't agree. Privatizing land is no way to protect it from toxic or nuclear waste. Ownership of land is a handy legal contrivance, but let's not take it too far. There is a finite amount of Earth for all the people that have lived, live now, and will ever live. Individuals live relatively briefly and have no right to carelessly dump nuclear waste that will far outlive them, regardless of some piece of paper. Ultimately our right to bury nuclear waste comes from exercising diligence and doing it in a way that won't cause any accidents for a very long time.
Right. If I need a nuclear reactor managed, I'll call you. Good to know the old talent of understanding exactly what the state of a nuclear reactor is by looking at a rock isn't lost. I'm just going to go and plug myself into my other computer now and manipulate it with my mind. Screens and command lines are for pussies, I can feel what it's doing well enough.
Admit it. You post strawman arguments as AC so you get modded Insightful for refuting them, rather than Troll
My sister went to school there, and after two and a half semesters there she was diagnosed with thyroid cancer.
That's nothing. My grandad went to Lourdes and only six months later he got leukemia. I want to know when people are going to take a stand against the unshielded holy radiation that causes such damage to humans.
I'm posting this AC because I just know it's going to be marked troll, and you're going to post something like "she's dead now, you jackass" as though that's relevant to the debate. But if you read the Wikipedia article you linked to, several studies have found no evidence of any increase in death due to TMI, especially compelling with the observations that cancer deaths were highest in the area with lowest fallout, and that the area around TMI has high levels of radon and so high background radiation anyway.
Your sister may have died from cancer (I don't know) and that's certainly a tragedy. Nevertheless, this is not the fault of the nuclear industry, but one of those pieces of shit that happen depressingly regularly in this amoral, godless universe. As a someone who is an engineer or an allied trade (you respect engineers after all, so you must be one) you should know statistics well enough to accept that.
>Operators who understood what they were doing would have checked what needed to be checked
What needed to be checked was the level of water in the core.
There was no instrument for that. I'm not kidding. See _Safeware_, by Nancy Leveson.
(It's a harder problem than it sounds like, if you think about the conditions in the core, but still ...)
The operators, deprived of an accurate picture of what was happening, followed their training, which was to prevent overfilling the cooling system.
The UI failed on functionality, and even if it had succeeded there, ease of use saves time and bandwidth in a crisis.
That control room is very similar (if a bit larger and whiter) to the control rooms in gas plants, oil rigs, and pump/flow stations in oil fields today. The stuff may seem old as heck, but really a lot of that stuff you can't just replace with a fancy new computer. The best you can do in the control room is upgrade to digital displays and consolodate sections a little bit. But that may not even be ideal, because the analog systems will be able to run for a lot longer during a power failure than a digital will, and that's a BIG deal.
One thing you CAN do is send all the information in that control room to a fancy new computer, and then you only need a couple hands-on operators at the plant in case things go very wrong. The rest can be handled by operators sitting in front of a few monitors back at home-base.
I know you didn't really say it, but I'd wager you were thinking it, and you've got to realize that is not a giant computer. It is a giant control room. It's not like you can replace the steering wheel of your car because you've got a new engine.
Security is mostly a superstition... Avoiding danger is no safer in the long run than outright exposure. - Helen Keller
I used to work in the nuclear power plant operator training industry. Believe me, whatever else those operators were, they were not cheap. The CEO could not skimp on salaries and hire idiots. In fact, in a time when $40K was an excellent salary, the training costs per operator was more than $1 million.
On the other hand, there were cultural obstacles. In Europe (Sweden), they hired engineers with masters degrees to become nuclear plant operators. In the USA, they were mostly high school grads who were union members and promoted from running older coal plants. Union politics, not merit decided who got promoted. They were not the best and brightest. Of course in Sweden they also attract the best and brightest to be civil servants. Can you imagine that happening here?
There are always plenty of suggestions as to where society should apply its best and brightest. It is much harder to place the worst and dumbest. Consider the bottom 25%. They have to have jobs. No matter where you assign them, the public will in some way be depending on those jobs being done well. So filling jobs becomes less of a question of rational allocation of resources, but more a matter of attractiveness and recruiting.
A plant operator must stand there and do nothing but monitor year after year, yet react swiftly and accurately in those rare seconds of pure terror, and then have the whole world second guess how well they did it. In addition, they have to do shift work for 24x7 operation. Most people think that it is a hell of an unattractive job. I think that the plant owners do a hell of a job trying to find and retain the best people they can get, and to enrich the jobs to make them less boring. It takes much more than deep pockets to succeed.
So you tell me. You play CEO and tell me how would you convince Google engineers to quit Google and become operators, and how many of the lower quartiles you would assign to invent Google. Convince those bright college students that they don't want to be environmental scientists, but nuclear power plant operators instead.
Fun fact: cows in a field two miles away from Three Mile Island got more radiation from Chernobyl.
If you think you can just "turn off the tap" at a gas plant, you are sorely mistaken. Pressures start to build when you do that, so if you block the gas off in one section, it will build in another. You've got a lot of systems to kill before you can turn off the gas - the source must go first, then at about the same time pumps pushing the gas along (these may be in the same spot, which makes that easier), then you can kill any processing systems along the way, and then you can close the tap.
If you DO have to close the tap first (a pipe failure that is leaking gas, for example) you've got to get a relief valve open and start burning your excess until you can get the gas re-routed or the rest of the plant shut down. If you don't, the plant goes bye-bye big-boom style. Where I work I'm about 1/4-1/2 mile from a flow station, and we're still in the blast radius of a catastrophic failure.
Whereas, as others have pointed out, nuclear power plants do in fact have a 1-button shutoff mechanism that kills the reaction immediately. Then containment is a cinch, if costly.
The problem was the alarm system + the reporting system was poorly designed, and the errors were of such magnituted that it actually looked like less of a problem than it was, and the operators had no way to confirm what the problem was. In a nutshell.
Add to that the fact that, since re-starting one of these systems from a cold start (which is what pushing the little red button to cease the reaction would mean) costs millions of dollars, SOP is to try everything you can to fix the problem FIRST, and then, as an absolute last resort, you kill the reaction (or shut down the gas plant, it's the same reasoning). So if the alarm looks like a malfunction in the alarm system, and not in the process, they are certainly not going to shut down the system until the alarm is fixed or they verify the malfunction in the process.
My question is, where was the redundant alarm system? Shouldn't that be a no-brainer for something with the damage potential of a nuclear plant? I mean, it might not have helped finding the problem, but it might have prompted the decision to shut down much sooner if BOTH alarm systems go out at exactly the same time.
Mind you, this is based entirely on the comments in this thread and the article summary so some of this may have been covered already. Naturally I would never RTFA.
Security is mostly a superstition... Avoiding danger is no safer in the long run than outright exposure. - Helen Keller
This. Most of the US civilian nuclear power industry is, to say the least, heavily influenced by the military nuclear power industry and the cult of personality surrounding Admiral Rickover. If nobody is in control, nobody can be held accountable when the fan hits the shit.
Er, in what way is that "nobody is accountable" attitude reminiscent of the nuclear Navy? They're obsessive when it comes to accountability. Every time I saw any fecal matter hit a rotary device, they were pretty damn rigorous about getting to the bottom of it and finding out who did what.
[b.belong('us') for b in bases if b.owner() == 'you']
Hey!
This is slashdot. Stop using facts to confront ignorant hysteria!
hawk
tmi2> sshutdown -r now
sshutdown: Command not found.
tmi2> halt -c
halt: invalid option: -c
Try `halt --help' for more information.
tmi2> help halt
help: Command not found.
tmi2> shut it down, damnit!
shut: demand not found.
tmi2> assume nuclear defense position
assume: Command not found.
tmi2> stick your head between your legs
stick: command not found.
tmi2> *%&*^&$
[from system: system going down for meltdown NOW!]
[from system: assume nuclear defense positon]
[from system: stick your head between your legs and kiss your ]
***line down***
to adapt a suggestion given by a libertarian acquaintance years ago . . .
Never mind government regulation. Require a half-trillion dollar liability policy. The insurance company will regulate far tighter and more effectively than the government.
hawk, who isn't advocating this, but finds it an interesting proposal
You are aware, of course, that under american and british law, this results in unlimited shareholder liability, aren't you?
Or are you just parroting ignorant claims about how corporate law and liability works.
AFAIK, there has *never* been a time or an anglo-american jurisdiction in which a corporation inadequately capitalized for the business which is entering does not leave its shareholders liable.
But then, I'm just an attorney.
hawk, esq., not offering this as legal advice. If you need that, pay for it, rather than relying on the ignorance posted on slashdot
AIG was an insurance company.
I am telling you that the operators should rely on their understanding of, interest for, and experience with the systems to make a decision what to check. Those who have merely been taught by the book or "given training" are incapable of this. They will be limited by what the book tells them, and fail to make the required decisions.
As they did at Three Mile Island.
The solution isn't to give the drones better training in being drones. It's not giving them a better interface. The solution is to not hire drones.
Thats just wrong. For something as dangerous and deadly as a nuclear reactor, you practically want a monkey to be able to figure out what they need to do.
You DO NOT require someone with a PHD to make the plant safe. You practically want the plant to be idiot proof and scram at the first blush of trouble.
By making it require (rare) operators that understand the plant as a systemic whole, you make them irreplaceable, and from a design for long long long term safety point of view thats just wrong. Over time, understanding of large complex systems at plants degrades, and with a plant lifetime of 20-50 years you will see whole generations change in the lifetime of the plant.
Chuck
Look fella, you just cannot have that requirement, that a person with full understanding of how the plant operates be on site at all times! What happens if the day shift all gets killed on the busride home from the company outing? or if there are say 10 guys who really have an understanding of the plant, and the plant gets bought out by some crap company and they decide to go pump gas for a living...
WHO WILL MAKE SURE THE PLANT IS SAFE THEN?
You have to design for the worst case.
Chuck