Data Storm Caused Nuclear Plant To Shut Down

rs232 writes to let us know that the US House of Representatives Committee on Homeland Security called this week for the Nuclear Regulatory Commission to further investigate the cause of excessive network traffic that shut down an Alabama nuclear plant. Investigators want to know whether the data storm could have been initiated from outside the plant.

Comments

No kidding

[HomelessInLaJolla]

excessive network traffic that shut down an Alabama nuclear plant. It came from all the ACs.

Investigators want to know whether the data storm could have been initiated from outside the plant Using Tor.

Re:No kidding

[Anonymous Coward]

It came from all the ACs.
Conpsiracy

Using Tor.
Theorist

Re:No kidding

[Anonymous Coward]

It came from all the ACs.

Conpsiracy

Hey, just because you were out drinking that night doesn't mean we didn't go on without you!

Re:No kidding

[Caffeinate]

Conspiracy . . . Theorist It's all because of Ronald Reagan, peanuts and baseball!

Re:No kidding

[StarfishOne]

Tor networks are generally not *that* fast.. so causing a data storm is not likely. ;)

Sometimes such connections are sooo slow, it makes users cry. They don't call it onion routing for nothing, eh? ;P

Re: The reason?

[Clockworkalien]

All of the plant employees were looking up Starcraft 2 news.

Shut down?

[Anonymous Coward]

>Investigators want to know whether the data storm could have been initiated from outside the plant.

Do invesigators also want to know how a "data storm" could have caused a nuclear plant to shut down?

Re:Shut down?

[Detritus]

RTFA, bozo.

Re:Shut down?

[Anonymous Coward]

NEVER, asshole!

Re:Shut down?

[Sj0]

It looks like it was a modbus plus network. We're talking a proprietary physical layer on up, specifically designed for PLCs to communicate with one another.

If there was a communications problem and a PLC blinks out of existence on a mission critical system, it's only the safe thing to fail the entire system to prevent damage to people, the environment, and equipment.

Re:Shut down?

[canuck57]

Do invesigators also want to know how a "data storm" could have caused a nuclear plant to shut down?

Not really, it is BS grandstanding for politics like they are doing something when they are not.

For if it were not, the plant would have been shut down by now. They are talking this happened in August, and I would bet the problem still exists.

Re:Shut down?

[Anonymous Coward]

No. That would mean actually doing there job. Instead they look at all the surrounding factors, and not the one that matters.

Has ANYONE come to expect anything else from the current Administration?

As an employee of someone who speaks directly with Congress-critters several times a week, at length, the BULLSHIT they currently have going on in Washington is staggerring. Sadly, I will not tell you about it, lest you would become as disheartened as I.

IT, next to be YRO?

[Anonymous Coward]

I'm missing the "haha" tag so much! Was it probably a pr0n related issue? Or just another case for the **AAs ?

The answer is easy

[tmk]

The employees used YouTube and MySpace

nothing to see, move along.

[SuperBanana]

Some choice quotes, emphasis added:

An investigation into the failure found that the controllers for the pumps locked up following a spike in data traffic -- referred to as a "data storm" in the NRC notice -- on the power plant's internal control system network. The deluge of data was apparently caused by a separate malfunctioning control device, known as a programmable logic controller (PLC).

"Conversations between the Homeland Security Committee staff and the NRC representatives suggest that it is possible that this incident could have come from outside the plant," Committee Chairman Bennie G. Thompson (D-Miss.) and Subcommittee Chairman James R. Langevin (D-RI) stated in the letter. "Unless and until the cause of the excessive network load can be explained, there is no way for either the licensee (power company) or the NRC to know that this was not an external distributed denial-of-service attack."

Wow. Just...wow. As if you needed more proof that this wasn't a hacking attempt:

"The integrated control system (ICS) network is not connected to the network outside the plant, but it is connected to a very large number of controllers and devices in the plant," Johnson said. "You can end up with a lot of information, and it appears to be more than it could handle."

Seriously, how stupid do you have to be to think "OMG, Haxxors?" Answer: work at Homeland inSecurity, or be a Congresscritter. They already figured it out. It was a controller for a specific piece of equipment that flooded the network and triggered a bug in the variable-frequency-drive controllers for pumps.

Re:nothing to see, move along.

[MECC]

Never hire windows admins brandishing the moniker "network admin".

Re:nothing to see, move along.

[A Bugg]

I work at a nuke plant as a system engineer. One of my systems are the reactor recirculation pumps, these type of pumps. I know for a fact there is no way hackers could "data storm" my pumps and there is extreme doubt in my mind that the same thing could happen at Browns Ferry. The pumps digital control system isn't even near any outside network.

However, I will fully put the blame on the PLCs. Those little suckers come in handy but if you don't completely understand every line of code and every instruction they can f_ck you over.

I also love how they say "well if you can't prove it wasn't, then it must have been".

Re:nothing to see, move along.

[bilbus]

My guess is they were using AB PLCs and they started broadcasting. They should have been on VLANs .. thats how we setup our PLC plant networks. Also why is the inside network connected to the internal network .. that should never be done, if you need access you need to vpn in.

Re:nothing to see, move along.

[b0s0z0ku]

Also why is the inside network connected to the internal network .. that should never be done, if you need access you need to vpn in.

Wrong. You shouldn't be able to VPN in -- there should be dedicated machines specifically for the purpose of accessing and monitoring that critical network (assuming that it's really that critical). Once a VPN link is opened, malicious traffic can traverse it the same as any other network. The only totally airtight security is complete physical seperation.

-b.

Re:nothing to see, move along.

[bilbus]

Well, in the real world ..... outside users like system integrators often have to connect in to troubleshoot a problem. Access is controled and monitored.

Re:nothing to see, move along.

[b0s0z0ku]

OK, then there should be a gateway machine/device that can be physically connected or disconnected selectively? Want access to the network? Call first, otherwise you're not getting on. Period.

Re:nothing to see, move along.

[bilbus]

They call us, they enable the account we log in. A person is standing in the work area, making the engineer aware of the location of people in the area. A robot arm starting up with a person in the cage can easly kill. It sounds like the plant network had a plc that went crasy and was broadcasting ... it looks alot like a virus broadcast. The IT department got scared and thought the network got infected and was speeding a virus. IT departments often do as little as posable with the plant network an stick to the corp network. ... they dont like working on this stuff. Outside of computer, network, and cabling the network is maintaned by the system intergrator.

Re:nothing to see, move along.

[Anonymous Coward]

You just have to love Browns Ferry don't you? This is the same plant that had wired its control cabling for two nuclear reactors through the same area. Then they had workers check the air tightness by using candles near their flammable insulation. It wasn't air tight and the flame of a candle was sucked into the insulation. Thus a fire broke out, $100 million of damage occurred, and control was lost of their two nuclear reactors for something around 8 or more hours. Why 8 hours? Because their fire team tried to fight the fire with portable CO2 extinguishers. Yes, for 8 hours. Until the local fire department (which they previously obstructed) put it out with water in 5 minutes. Idiot designers and idiot employees. I'm surprised that plant didn't have a meltdown before TMI. But boiling water reactors are a little harder to destroy.

Re:nothing to see, move along.

[b0s0z0ku]

IT departments often do as little as posable with the plant network an stick to the corp network. ... they dont like working on this stuff.

Certainly easier for them if they have an outside contractor to blame if something fucks up with dangerous equipment. In the current liability climate, I can't blame them too much.

-b.

Re:nothing to see, move along.

[Anonymous Coward]

I also love how they say "well if you can't prove it wasn't, then it must have been".

But god does exist! I can feel him!

Re:nothing to see, move along.

[Anonymous Coward]

For a fact the network recirc pump controllers at Browns Ferry are on a private network...because 3-4 years ago the recirc system engineer tripped both pumps off sitting at his desk in the administration building while playing with the software. Oops.

As far as the "OMG they're gonna melt down because of a packet storm", the real nuclear-safety-keep-the-core-from-melting systems (reactor protection systems, emergency core cooling systems) don't even have any computerized control -- they all rely on simple electrical relays first designed and manufactured 40 years ago to trigger automatic action.

Re:nothing to see, move along.

[jd]

I believe this is the nuke plant that is supposedly using Windows NT to handle SCADA (Supervisory Control and Data Acquisition) functions, and I know it's the plant that has shown gross incompetence in relation to fires (see assorted other postings). Internal systems are not supposed to be connected to external networks. It's unlikely this one was - not because they're smart, but because I'm not certain they'd know how. We can therefore eliminate external causes. Sadly, we have passed from the Age of Englightenment and the Age of Reason into the Age of Paranoia and the Age of Dementia, so that is likely the attribution we can expect from the Department of Homeland Insecurity.

A random fluctuation in internal traffic levels seems equally unlikely. Why? Because it has worked for some time, and I doubt the reactor was doing anything unusual at the time. A true network storm is unlikely - the term exists, but describes an astronomically rare situation. If a network is flooded, it is either near or at capacity. A network storm is when capacity is exceeded in a way that is self-perpetuating. The last time I remember the term being used in a public forum was I think over twelve years ago when a public demonstration of the multibone caused a cascading router flap that shut down a large segment of the Internet backbone due to total gridlock. It wasn't just that nothing else could get through - nothing AT ALL could get through.

What does this leave us? It makes it extremely unlikely that the network traffic per se had anything to do with the shutdown. Much more likely is a cumulative error in the devices involved that merely happened to turn into a fatal bug at roughly the same time as the network spiked. It might be network related, but nobody here can seriously believe it was network caused. Networks may be polled, in which case network traffic that escapes being polled is simply never seen. Network drivers may also be event-driven, but if the interrupt handler is buggy - which would usually mean the handler can be interrupted by itself indefinitely - it's hardly the fault of the network.

In other words, this is a gross programming error that the coders and managers are desperately trying to blame on something - anything - other than their own ineptness. It might merit Scott Adams making a Dilbert cartoon over, but that's it.

Re:nothing to see, move along.

[Anonymous Coward]

Well, what do you expect? This is in Alabama. The people are likely inbred.

Re:nothing to see, move along.

[(negative video)]

A random fluctuation in internal traffic levels seems equally unlikely.

Look up "Poisson distribution". At low packet rates, large rate fluctuations by random chance are the rule. You also have to consider events that can trigger a common packet rate spike, such as a a non-critical subnet being power cycled. Combine this with a device that has an overflowable packet buffer and you have a recipe for inevitable failure.

A true network storm is unlikely - the term exists, but describes an astronomically rare situation. ... A network storm is when capacity is exceeded in a way that is self-perpetuating.

At work we recently had a cheap router near the edge that decided to start echoing broadcast packets. ARP traffic was not pretty, and DHCP got so confused that the Windows clients went all plug-n-play and started making up their own addresses. The core routers automatically detected the repeated packets and decided to go into cycle-breaking mode: automatic rolling network bisection. Unfortunately they had the smarts to find cycles on their own ports but not echoes from a misbehaving device, so that actually made the network more confusing. Eventually IS had to manually bisect the network until the talky node could be found.

In other words, this is a gross programming error that the coders and managers are desperately trying to blame on something - anything - other than their own ineptness.

It's an honest description of the final event that resulted in the system failure.

Re:nothing to see, move along.

[that this is not und]

We need a cite.

You said way too long a story with no attribution.

Re:nothing to see, move along.

[whoever57]

We need a cite.

This isn't Wikipedia. Most posters assume that readers have sufficient IQ to use Google. Perhaps that is a bad assumption in your case.

Re:nothing to see, move along.

[Firethorn]

Having to work with a seperated network myself, I'd have to agree about doing as little as possible with it.

In my case it's for two reasons. One, the disconnected network is considered the critical one, and is far more locked down than the one connected to the internet. Second, the one connected to the internet is the one used 99% of the time.

Anytime we touch a system there's a chance we'll screw it up/break it. Our treatment of the isolated network is pretty much 'don't fix what isn't broken'. It wasn't too long ago that we had a P200 still acting as a PDC on it. It worked, we didn't touch it.

Re:nothing to see, move along.

[Anonymous Coward]

there should be dedicated machines specifically for the purpose of accessing and monitoring that critical network (assuming that it's really that critical). It is a nuclear power plant. Of course it is critical.

Re:nothing to see, move along.

[Bo'Bob'O]

What I'm wondering, is are these even on TCP? Or for that matter even Ethernet? I would have thought this would be something more like RS485 to begin with.

Though I do notice that it says the plant has been closed for a decade until this year. I guess it's best to check all options, but how is that tiny possibility remotely any sort of news when compared to the vastly more likely problem of a bug in a brand new control system, especially when it seems they found exactly what that bug was already?

Re:nothing to see, move along.

[DeadChobi]

Why should the readers have to bear the burden of proof? It's your assertion, you get to show evidence.

Re:nothing to see, move along.

[Anon99]

>However, I will fully put the blame on the PLCs. Those little suckers come in handy but if you don't completely understand every line of code and every instruction they can f_ck you over.

One of my teachers worked as system architect in nuclear plant.
And he taught me the concept of proving critical pieces of software.

Proving means that you know every possible state for your software, that means that you know on every instruction what the register values are and exact memory contents.
Thus you know every possible logical state that your software can enter.

I once did this to one embedded project, that HAD to work, as we were going to build 10000s of units.
Took me about a month to do, but after that there was not single bug found in the product lifetime.

Re:nothing to see, move along.

[Heir Of The Mess]

What I want to know is what the hell are Homeland Security doing trying to protect Nuclear Power plants when there are still 11 yr old girls living in housing projects downloading Britney Spears music off the net FOR FREE!! They should obviously be focussing on funnelling money into the pockets of the people with whom Britney has a contract with.

Re:nothing to see, move along.

[kasperd]

A random fluctuation in internal traffic levels seems equally unlikely. Why? Because it has worked for some time, and I doubt the reactor was doing anything unusual at the time.

This is not about the network being highly loaded with lots of packets comming from all sorts of places. This is about a single device for some reason flooding the network. I have seen the results of units flooding a network with broadcast traffic. I don't consider it highly unlikely for one unit to eventually start doing that because of a design flaw. Somebody should take a closer look on the design of that PLC to see if there is a likely explanation. Maybe a physical defect could have caused it to send a broadcast packet and afterwards think it had not been sent yet and send it again and again. Maybe the explanation is something else. There is no way I can say for sure without having seen the PLC.

Network drivers may also be event-driven, but if the interrupt handler is buggy - which would usually mean the handler can be interrupted by itself indefinitely - it's hardly the fault of the network.

If the handler could interrupt itself, it would probably result in a stack overflow and crash the unit. But that is not the most likely bug to introduce. A more likely and almost as bad problem would be if by the time the interrupt handling ended, it would immediately take another pending interrupt. In that case it would never be processing more than one interrupt at the same time, but yet it would spend all of its CPU time handling interrupts. The unit would appear locked up, but would come back to life shortly after the flooding stops. I have seen the later happen with Linux machines (I don't remember which kernel version, I think 2.4.something). I later repeated the experiment with a Windows ME machine, which also locked up, but didn't come back to life when the network cable was disconnected. This situation was quite easy to test, just loop a cheap 100Mbit/s switch back to itself. It would probably take a 1000Mbit/s network to actually cause this with the last generation of CPUs. I don't know if switches and/or network drivers have been improved to avoid the exact scenario I tested.

In my case this was not a problem, but of course in some critical systems, it can be. I see at least two problems. Units not tested against this scenario, and having redundant units communicate to each other over the same ethernet. Of course just having two ethernets does not solve the problem of one of them being able to take down units. Redundant units protect you against physical defects in one unit, not against design flaws.

Re:nothing to see, move along.

[whoever57]

Why should the readers have to bear the burden of proof? It's your assertion, you get to show evidence.

Gawd, another one.
1. It wasn't my assertion -- I did not make the original post about Browns Ferry. Try reading next time!

2. I just happened to hear an article on PBS about Browns Ferry the day of this post.

3. As I mentioned before, you can confirm it using Google. Here, I'll even show you how to find it using google

4. What is it about "/. is not an encyclopedia" that you don't understand?

There may be many case where one might claim that a post on /. is pure BS, but in the case of the great-grandparent post, the facts are easily confirmed.

Re:nothing to see, move along.

[canuck57]

...Idiot designers and idiot employees. ...

Driven to it by idiot management and idiot politicians.

Going for idiot "employees" and "designers" is going after the effect, not the cause. You want idiots, you hire them. You want "cheap" designs, the designer will design them. Employees and designers are told what to do. If you push back in these roles too much you will not have a job. Mind you, I would include the primary contractor.

Fixing this means fixing management. The best way to do this is to shut down the facility for 6 - 12 months while everything is independently analyzed and fixed at their expense. The board then will fire management to answer to the shareholders. And maybe the shareholders will vote for a more proactive board.

And they can be shut down, I am sure there are clauses of "safety" in the license to operate a reactor that can be used. And site the fact that management doesn't even know what happened, so it could happen again, and again... until some real serious damage results. Just like an aircraft, ground it until fixed.

Re:nothing to see, move along.

[dave1g]

You speak as if Wikipedia lives up to its policy of citing all statements of fact...

Re:nothing to see, move along.

[Anonymous Coward]

People with their heads in the sand enough to systemtically verify a painfully low-level system are less likely to realize that their device operates and interacts in the *real world* which does not have to follow their cleverly verified rules. **Results are all that matters**

The days of the low level custom control codes and custom controllers are numbered. Yes global complexity has its costs but its well worth it concidering the massive economies of scale that exist in the embedded control domain.

If your methodology doesn't scale (registers and memory contexts) then you really need to be thinking about what you are doing wrong rather than the illusion of what your doing right.

Re:nothing to see, move along.

[Anonymous Coward]

I work at a nuke plant as a system engineer.

[...]

I am a senior ME major HOPING to get 40-45k as a starting salary.

http://slashdot.org/comments.pl?sid=95771&cid=8202 136

You fucking liar, in February, you were going to Wash U and today, you're a nuke plant systems engineer?

Re:nothing to see, move along.

[Radworker]

Oh come on now! This has been common knowledge to anyone who cared to find out since it happened back in the 80's. Why do you think they shut Browns Ferry down? I have worked in the industry for 18 years and it still amazes me how little the man on the street knows about the "big green monster". For the record, plant control systems are isolated from the real world. The only way to inject an exploit would be to get it into an embedded system that was eventually installed in a power plant. A little too indirect a route for a terrorist if you ask me. Besides the effected systems wouldn't have caused an accident. In a BWR no flow equals no reactor power. In PWR mind you things would have been a little different. I leave that as an exercise to the reader to research further. I also don't wish to discuss anything that might give insight to someone who might actually try it.

Re:nothing to see, move along.

[b0s0z0ku]

It is a nuclear power plant. Of course it is critical.

Well, the reactor should be critical. If the network controlling it were critical too, I'd be a bit disturbed.

Re:nothing to see, move along.

[Anonymous Coward]

Hahaha, the karmawhore kid is really lame. But very funny indeed. "Hoping to get 40k", hahaha, nuke plant engineer my ass. Good one!

Re:nothing to see, move along.

[DerekLyons]

"The integrated control system (ICS) network is not connected to the network outside the plant, but it is connected to a very large number of controllers and devices in the plant," Johnson said. "You can end up with a lot of information, and it appears to be more than it could handle."
 

Seriously, how stupid do you have to be to think "OMG, Haxxors?" Answer: work at Homeland inSecurity, or be a Congresscritter.

You missed the third option: Someone who actually knows something about real security - rather than believing that the FUD about security constantly spewed on Slashdot constitutes informed discussion about the issues.
 
Even though the SCADA network isn't connected outside of the plant - that does not mean that various nasties couldn't have come in accidentally (or maliciously) across the sneakernet or via malicious physical or [internal] network acess to the PLC that is believed to have caused the failure. It's not nearly so simple as "network not connected to the outside world, hacking ruled out by default".

Re:nothing to see, move along.

[DerekLyons]

A random fluctuation in internal traffic levels seems equally unlikely. Why? Because it has worked for some time, and I doubt the reactor was doing anything unusual at the time. A true network storm is unlikely - the term exists, but describes an astronomically rare situation.

When investigating an accident you cannot ground rule out an occurence that is unlikely or rare - unless you have positive evidence that said unlikely or rare condition did not occur, or positive evidence of another cause. "Unlikely" and "rare" are not synonyms for impossible.
 
 

In other words, this is a gross programming error that the coders and managers are desperately trying to blame on something - anything - other than their own ineptness.

Absent any facts (as opposed to opinions presented as facts), what precisely is your evidence for this conclusion?

Standards!

[26199]

You'd hope that in something as critical as a nuclear power plant the answer would be, very quickly, "no, it didn't come from an external source because that's impossible". Followed by detailed analysis of the logs to determine which internal system screwed up.

That said, the article is a bit sparse on actual technical details, so my derision may be unwarranted.

Re:Standards!

[AudioInfecktion]

As it should be. The point is this. Any of the computer/network equipment that actually runs the plant should not be connected to the outside, period. All normal computers for office work, typing up non-classified reports and reading slashdot should be on a whole seperate network. Idealy, there should be 3 networks since the plant should only have certified equipment connected to it that won't cause what happened here to take place unless something was truly malfunctioning. I'd be a little scared to find windows boxes, and even most unix/linux things connected to the plant network.

Re:Standards!

[mrchaotica]

You'd hope that in something as critical as a nuclear power plant the answer would be, very quickly, "no, it didn't come from an external source because that's impossible".

Actually, power plants have to have a connection to the outside world. Why? Load-balancing for the power grid. If another plant goes down somewhere, this plant needs to know about it so that it can adjust output to compensate. For that, all the plants need to be hooked to a communications grid, which could conceivably be hacked (even though -- I would hope -- it's not connected to the Internet).

Re:Standards!

[legirons]

You'd hope that in something as critical as a nuclear power plant the answer would be, very quickly, "no, it didn't come from an external source because that's impossible

Indeed.

Unfortunately, sometimes our favorite software supplier is involved...

Re:Standards!

[Artifakt]

This actually can be avoided (and AFAIK current designs do). Fast, electronic level response to avoid blackouts and such requires very much less time than changing reactor output would either allow or facilitate anyway, so the direct machine to machine communication links don't really need to go to the power cycle control systems at all. Instead, rapid response grid balancing is done at external switchpoints. For the newer designs, these are outside the whole plant at substations, let alone just outside the core areas. Between these links and reactor control systems, there's supposed to always be an air gap.
        Given that, any hacking would have to include a social engineering element designed to fool the operators into making the wrong decisions. If we include that stipulation, yes, it's quite conceivable. If we postulate someone bridging the air gap, maybe by something as simple as hooking a laptop that also contains a wireless card into the control network, then a non-social engineering attack becomes conceivable, but not really otherwise.
        DOE and NRA doctrine is that adjusting reactor output based solely on a trigger event outside the core instrumentation is supposed to always require a high level human decision. Supervisors are also at least supposed to be trained to the point where they can make these decisions without adding any more response time than a conventional, (i.e. hydroelectric or coal based), plant would need for their human level decision events. (Yes they have them. For example the four TVA dams that supply Alcoa aluminum face a whole series of individual and joint human level decisions every time Alcoa's main furnace system glitches, and these have to include how long Alcoa expects them to need to dump power elsewhere, and for each of them, what options the other three dams are considering).
      The DOE does not legally presume that reactors are even as responsible for balancing the grid as conventional plants, but given how much older a lot of the conventional plants are, it's pretty easy to do much, much better than is strictly required, and it should be noted that, in the last New York blackout all the cascade effects and switching failures happened in 1940's era or earlier fossil fuel plants, and the worst points were 1930's or even 1920's era designs. Still, the rules are that if the conventional plants are failing at load balancing, even if the grid is experiencing severe cascade failures, the nuclear sites will let the whole thing crash rather than take the risks of trying to stabilize the grid by actually modulating their reactions.

Re:Standards!

[Torvaun]

They ran into a potentially dangerous system error, and could not immediately determine the source. If I were running the shop over there, the very first thing I would ask is "Is there any possible way this could be initiated from the outside?" Because I don't want someone to say "Of course not," and go on their merry way. I want someone to be able to explain what happened, why it happened, and what potential circumstances could possibly cause it to happen again. When I ask if it could possibly be started by someone on the outside, I want them to check every path between a line in, and the problem.

Re:Standards!

[dbIII]

Fast, electronic level response to avoid blackouts and such requires very much less time than changing reactor output

It's called hydro - or sometimes even pump storage. Conventional thermal power is cheap but it takes a long time to increase output unless there is already spinning reserve. Non-conventional thermal power still takes time BECAUSE IT IS NOT MAGIC unlike what we are led to believe by those that want to build a few hundred 1950's style plants painted green. Nuclear power possibly would be a mature technology by now if some effort had been put in over the last few decades, but for now it's just a new and expensive way to boil water sold as the peaceful side of the bomb.

Re:Standards!

[Firethorn]

As one of those who would like to see hundreds of new nuke plants, I'll point out that nuclear power has actually slipped slightly below coal for production costs. In addition, new pollution control requirements and the possibility for CO2 sequestration requirements substantially increase the cost and slightly reduce efficiency in coal plants.

I try to always point out that I'd use the nuclear power to replace coal power, which takes nearly as long to cycle up or down, not to try to replace more responsive power systems such as gas-turbine or even hydro. For peak leveling, I'd suggest using the plant's excess power output during non-peak times to produce hydrogen or ethanol or whatever, so you do have that spinning reserve.

Re:Standards!

[notamisfit]

Suddenly, the Navy's whole "no computer technology anywhere in reactor control" mindset makes a whole lot more sense.

Re:Standards!

[dbIII]

As one of those who would like to see hundreds of new nuke plants,

After some R&D and building some prototypes of promising new designs I'd be right with you - but our current best bets are things out of South Africa (pebble bed) and India (accelerated thorium) done on very small buidgets with very small teams and they need more work. The mainstream is just chasing taxpayer supplied pork. If they were after more than a handout they would be putting in some effort - instead they spend orders of magnitaude in PR, advertising and outright bribes than R&D.

As for costs - you can't just conveniently ignore capital costs. If you could hydro, wind, solar etc would win every time even in those places where it would be a stupid idea or where the capital costs are far too large for the return. Nuclear power is a possiblity in those places that have the infrastucture of a weapons program but everywhere else you would have to build up an entire industry from scratch. Iran is the best example currently where that is taking place and it has cost them a fortune to do so - hence few people think it is for purely civilian purposes there. In South Africa it was possible to take people from the weapons program to develop pebble bed. It is also far too big an investment for private enterprise - hence no new plants getting built while governments had cold feet on the issue and the "new generation" designs from companies like Westinghouse are just tweaked 1950s designs painted green.

Re:Standards!

[stonecypher]

Actually, power plants have to have a connection to the outside world.

Doesn't have to be on the Internet, though. 'Course, then again, the Internet itself was once an isolated network to prevent critical systems being exposed to this sort of thing. Maybe we need a new ArpaNET?

Re:Standards!

[grapes911]

I work for a private company that develops software for nuclear power plants. I VPN into various plants all the time. So all plants have outside access and thus it is not entirely impossible.

Re:Standards!

[Firethorn]

As for costs - you can't just conveniently ignore capital costs.

Who says that I'm ignoring them? Referenced site is for European power, and nuclear comes in cheaper at 23.7 E/MWh, vs 28.1 for coal. That's including capital costs, but not including CO2 tax, which raises coal to 44.3.

If you could hydro, wind, solar etc would win every time even in those places where it would be a stupid idea or where the capital costs are far too large for the return.

In the USA, hydro is considered 'overutilized' IE we can't install much more hydro power without negative ecological effects. Solar is orders of magnitude more expensive. A recent thread showed that they expected costs to be $300 million for a 40MW plant. Ten plants and you haven't even gotten half a GW, yet for the same capital you could have 3GW of nuclear plant going. Even wind has limited areas where it's truly usefull - You need a fairly strong and steady wind for it to work, and areas like that aren't everywhere. While you can ship power a long ways, you try to limit that.

Personally, I find the pebble bed to be missing a couple points - Personally, I want the fuel to be easy to recycle, or better yet burned more or less completely in the reactor, such as an IFR(yes, more research needed).

The mainstream is just chasing taxpayer supplied pork. If they were after more than a handout they would be putting in some effort - instead they spend orders of magnitaude in PR, advertising and outright bribes than R&D.

Hmmm.... Haven't seem much PR. Sure, there's a number of websites, but that's peanuts in comparison with nuclear research costs. On the other hand, at least in the USA they need the PR. NIMBY, nuke scares and outright stupidity* have stopped plant construction for years(well, high interest rates for a while helped). Today, they should be able to get lower interest rate loans, which makes the whole thing much more affordable.

As for the research, you do realize that we've managed to keep increasing the power produced by our reactors to the point it's been like building three new plants a decade?

and the "new generation" designs from companies like Westinghouse are just tweaked 1950s designs painted green.

In some ways that's like saying a Ford Focus is a Model-T painted green. The new generation designs, while still using the same basic technology, are far safer, more productive, and hopefully cheaper to build, since one of the 'safety' features was design simplification; fewer parts that can break.

*Why are we trying to build a million year shelter for stuff that's still 90+% fuel?

Re:Standards!

[Radworker]

Nice thinking but doesn't quite work like that in reality. A nuke maintains a steady output. The distribution system is separate from the generation system. The grid does communicate via a private network. The generation system would feel a loss of grid in the form of a turbine overspeed as the generator unloaded. This in turn would trip the reactor and actuate the MSIV's. Since I am not an operator I may be missing a few details but this is essentially correct. Hope this helps clear a few things up.

Re:Standards!

[dbIII]

Referenced site is for European power, and nuclear comes in cheaper at 23.7 E/MWh, vs 28.1 for coal. That's including capital costs,

Now we are getting somewhere - but to see what I mean you will have to see what the assumptions applied to get those numbers are, wonder why the British experience is vastly different in economic terms despite doing everything the same way, and wonder why private enterprise never builds these things even though they are portrayed as being the winning option. Once you've done that you'll understand what I mean and realise you have been conned.

As for the other bits - I should have stated more clearly that it is capital costs that make photovoltaics etc a bad idea for large installations - but if capital cost is not considered then that is not clear.

As for PR - many people are calling something toxic and radioactive that comes out of the ground with a processing process involving large quanitites of highly toxic chemicals "clean" - since I'd only use that to describe washing powder I think the PR is working. A lot of stuff we do involves toxic chemicals and they can be dealt with properly but pretending they don't exist is dishonest and counterproductive. There's a vast cloud of bullshit around the nuclear debate - look at how those numbers above were derived and you'll get a whiff of some of it.

Re:Standards!

[Firethorn]

Now we are getting somewhere - but to see what I mean you will have to see what the assumptions applied to get those numbers are, wonder why the British experience is vastly different in economic terms despite doing everything the same way, and wonder why private enterprise never builds these things even though they are portrayed as being the winning option. Once you've done that you'll understand what I mean and realise you have been conned.

We are seeing a number of new reactor construction projects - it's just that they're all taking place on existing plants.

It's a massive case of 'who's first?'. Current estimates place the first new plant of a type at $1400 per kilowatt, dropping to $1000 for subsequent installs.

As for PR - many people are calling something toxic and radioactive that comes out of the ground with a processing process involving large quanitites of highly toxic chemicals "clean" - since I'd only use that to describe washing powder I think the PR is working.

This is different from coal how? Oh yeah - It's not routinely released into the enviroment. Remember acid rain? Heck, mercury emissions? If you want to see something scary - take a look at the death rate atrributed to coal power. Hint: It's close to that of smoking.

Re:Standards!

[dbIII]

After dodging the question we get to the childish but common cry of "coal is bad so why can't we be bad too" argument, a bit odd since I originally mentioned hydro. Don't just recycle PR that uses these childish tactics - think for yourself, you can be better informed than that. Nuclear has to stand on it's own merits to be worthwhile and not just a lesser sacrifice at the expense of the economy to reduce CO2.

Would it be possible to name one of these new reactors? Are they new designs or the old ones we know are uneconomic? Are they actually new reactors or just modifications to the existing ones - that is the question that will really get to the heart of it.

Re:Standards!

[Firethorn]

After dodging the question we get to the childish but common cry of "coal is bad so why can't we be bad too" argument, a bit odd since I originally mentioned hydro.

Recycle them? I've looked up the research myself. As for the 'childish' cry, it's called ORM. Operational Risk Management. It means that rather than suck on our thumb in the closet we assess the risks and attempt to minimize them while still getting work done. Coal kills thousands of people a year. It's a big event when nuclear power kills someone; anywhere in the world. I'm not talking about CO2 emissions here, I'm talking about all the other stuff, such as uranium, thorium, sulfer dioxide, NOx, etc... I tend to mention CO2 because that's a big buzzword today, gotta stop global warming.

Sorry about missing the british question. My answer would be that there's evidence of massive incompetence. It happens in government all the time(just look at the Big Dig). Excessive paranoia, cumbersome regulations, and changing plans in the middle don't help.

Would it be possible to name one of these new reactors? Are they new designs or the old ones we know are uneconomic? Are they actually new reactors or just modifications to the existing ones - that is the question that will really get to the heart of it.

Last reactor online: Watt's Bar, in Tennessee, it became operational in 1996. Comanche Peak 2 was also a late build. Sometime this year Brown Ferry 1 is expected to be re-activated, shut down over a decade ago. It's a 1.1GW reactor.

They're expecting new construction to begin around 2010 on a whole bunch of new reactors; but are keeping details close to their chest(less worry about greenpeace that way I guess).

Re:Standards!

[dbIII]

Sorry about missing the british question. My answer would be that there's evidence of massive incompetence. It happens in government all the time

Jumping to conclusions like that based on no evidence can make people jump to similar conclusions about your honesty and character and discount everything you say - however I suspect it's just years of having "nuclear is good" PR shoved down your throat that exdpensive nuclear plants have to be due to incompetance and not harsh reality under detailed public scrutiny. Some history - after a change of government British Nuclear Fuels was forced to release it's financial details to the public and the subsidies it was given from the taxpayer were clear - but it is apparent that they are no less competant than anywhere else and have similar designs. It is worth enquiring where those numbers above come from and what assumptions are made - the US power industry over a range of energy types is notoriously corrupt and greedy for taxpayer money or tax cuts. Going off on a tangent about coal is pointless - nuclear power has to stand on it's own merits and it is worthwhile for any nuclear power advocate to know where those numbers come from and to be able to relate them to a physical plant. There are a lot of ways to cook the books - for instance adding in the output for plants that are offline, assuming that all fuel costs are the same as for the best unit and assuming all capital costs are the same as for the most recent unit of it's type. All this financial misrepresentaiton and silly hype is counterproductive and actually delays putting decent investment into designing new plants that can survive without government pork - in my view it is in the best interest of the nuclear industry to actually spend some R&D money and not stay stuck in the 1960s where they could sell weapons materials - thay cannot do that any more in the USA even if a very large number of new bombs are built so they have to get their money by building plants that can stand on their own merits.

Re:Standards!

[Firethorn]

Some history - after a change of government British Nuclear Fuels was forced to release it's financial details to the public and the subsidies it was given from the taxpayer were clear - but it is apparent that they are no less competant than anywhere else and have similar designs.

Look at the British rail system sometime and some of the complaints there. Meanwhile France, Japan, and the USA all manage to have safe and economical plants. I'm not saying that the plant managers are incompetant - I'm saying that the people performing the shutdowns are incompetant, or perhaps forced to be that way by politicians.

the US power industry over a range of energy types is notoriously corrupt and greedy for taxpayer money or tax cuts.

Welcome to economic reality. If you were given the option for a tax break or handout, would you take it? I'm willing to bet most would. All industries do this to one degree or another. The US steel companies and tariffs. The airlines and their writing off their pensions(picked up by the gov). Farm subsidies. Heck, tobacco subsidies. Universities, even 'private' ones, receive billions in federal and state monies.

Going off on a tangent about coal is pointless - nuclear power has to stand on it's own merits and it is worthwhile for any nuclear power advocate to know where those numbers come from and to be able to relate them to a physical plant.

To the contrary, I think that bringing in some points about coal is central to the subject.

Again, it's ORM. We need electricity(it's a net enabler of our lifespan and quality of life, after all). There are a number of sources. I've examined a number of them, gauging total cost, power availability, and pollution. Nuclear is only minorly more expensive than coal, especially when you consider the problems of wind/solar in most areas. It eliminates the vast majority of the pollution** issues. Possible install sites are extremely limited for effective wind power, and it's still more expensive than nuclear, especially when you figure in standby power requirements. Solar is orders of magnitude too expensive. Possible install sites for nuclear are actually more common than coal, as it's best to have the coal plant near the coal mine, while nuclear fuel is compact enough that transporting it is a pittance. One train car's worth is enough fuel for a year, while it can take 2-4 100 car trains a day to keep a coal plant fed. A water supply such as a river reduces costs, but is not strictly necessary.

Why do I pick on coal so much? Simple, Coal is 49.7% of the total electricity generation in the USA, it's the largest single source.

Nuclear is the second largest, at 19.3%, followed by NG at 18.7%. Hydro is a mere 6.5%. 'Other renewables" is only 2.3%. Petroleum sources is actually larger, at 3%.

In various posts I believe that I've touched on most of those sources, but I'll cover them again. Coal is cheap but incredibly polluting. I've seen signs that we're burning our sources of NG* faster than we're burning our oil. It's certainly on of the more expensive sources at this time. Hydro faces more problems than nuke plants for any significant expansions. We've actually blown up a number of dams due to the ecological damage they were causing. Petroleum(oil)? Just look at the pump to see how economical that is at the moment...

*While it's cheap and fast to build a NG power plant, it also has the highest operating costs, and increases in the price of NG isn't helping. Thus, except in California it's mostly used to satisfy peak demands only.
**It's not pollution if it's contained.

There are a lot of ways to cook the books - for instance adding in the output for plants that are offline, assuming that all fuel costs are the same as for the best unit and assuming all capital costs are the same as for the most recent unit of it's type.

Uhh.. Have you looked at t

Re:Standards!

[dbIII]

Uhh.. Have you looked at the statistics I've posted?

I did not see a single thing that can be traced back to the performance of a real plant - just rubbery derivative figures taken on trust and then the discounting of publicly available information as being warped due to completely supposed incompetance based apparently on railways! Find some figures from a single REAL plant that actually EXISTS and consider those in isolation - get those from the most efficient nuclear plant you can find. You will see then what I am talking about. Vague hand waving "corrected" numbers are often used for deception - consider inputs and outputs and not tax breaks, subsidies, real estate schemes, construction divsions, share specualtion and other things that will make up the finances of a nuclear power company.

Funny you mentioned France as having economical plants - google will help you there (you advocate nuclear power but have never heard of fast breeders like Superphoenix?), and as most are dual use facilities as part of a weapons program nobody really expects them to be able to produce economical power as distinct from purpose built plants. Japan has purpose built plants, they still are not cheap but the extra cost is OK due to the benefit of it reducing the severity of a possible blockade - but pretending nuclear power generation there is cheaper is the stuff of pure fantasy. In the USA as you are I'm sure aware a lot of it is about pork.

Where is this cheap plant, what was it's capital cost, what are it's operating costs? Why does this differ from the results about British plants published in the New Scientist a few years back - a real reason now please and not just insulting the British nuclear industry. Find that out and you'll be better able to talk about this instead of reguritating advertising and trying to distract people by talking about how bad coal is.

Re:Standards!

[Firethorn]

I did not see a single thing that can be traced back to the performance of a real plant - just rubbery derivative figures taken on trust and then the discounting of publicly available information as being warped due to completely supposed incompetance based apparently on railways!

Sure, it's a conglomeration of all plants in operation. It's statistically more proof than posting numbers for a single plant - which can be cherry picked. For every plant that's worse than the average, there has to be a plant or plants operating a corresponding amount above the average.

A for the point about rail, I was pointing out that England's government, on average, is more incompetant than many. Of course, the US government is down there too(especially for nuclear waste disposal), but at least we try to keep our hands out of many things. I could have said the same thing about healthcare and the problems that they're experiencing.

Funny you mentioned France as having economical plants - google will help you there (you advocate nuclear power but have never heard of fast breeders like Superphoenix?), and as most are dual use facilities as part of a weapons program nobody really expects them to be able to produce economical power as distinct from purpose built plants.

Sure I've heard of Superphoenix. And no, most of France's plants are NOT dual use facilities. Yes, an experimental breeder reactor, larger than any ever previously constructed, turned out to be uneconomical at the time. Meanwhile the smaller Phoenix, another breeder, remains an important part of their nuclear program, mostly for waste transformation, but it also generates power.

One thing that France did right was to stanardize their costs. Right now our nuclear system is a lot like space programs; every plant is unique. This drastically increases costs because you can't really share lessons learned or development costs.

If nothing, else, look at the latest fusion plant test - A test plant, costing as much as a gigawatt nuclear plant, taking up the same footprint, yet NO allocation or allowances to ever generate power from it. So it's a pure test plant. At least with the breeder there was the hope to produce power from it.

I'll also note that they started construction before I was born. We've learned a lot about nuclear processes and material science since then.

Why does this differ from the results about British plants published in the New Scientist a few years back - a real reason now please and not just insulting the British nuclear industry. Find that out and you'll be better able to talk about this instead of reguritating advertising and trying to distract people by talking about how bad coal is.
You mean an article like this? You know, where they acknowledge that they messed things up by playing politics with it?

And again, I'm not trying to distract people with coal, I'm comparing them. Nuclear power isn't 100% safe, but then, pretty much nothing is. To not acknowledge this is to hide in the closet sucking your thumb, accomplishing nothing. What Nuclear power is is safer than all other power methods of it's availability and economy. It even beats hydro.

Re:Standards!

[dbIII]

Sure, it's a conglomeration of all plants in operation. It's statistically more proof than posting numbers for a single plant

The problem is neither you nor I know where the numbers actually come from. Since there is no verifyable source of this information and PR companies are involved I will remain sceptical and actaully welcome cherry picked information since it has the advantage that it can be shown to be real and not just made up. Show me information about the best plant - that is what I am interested in and that is what the nuclear lobby should be pushing if they want to get an increased share of energy production by honest means.

Good effort with the linked article - however I was talking about plant operating costs and capital costs and generally ignoring the waste issue since that's what the nuclear lobby do (many tell you to ignore the waste costs because atomic power is "clean"). Some years ago British Nuclear Fuels opened up it's books - actually under a conservative government led by somebody with scientific training (Thatcher). I'll see if I can find more details but even the operating costs far exceeded those for other equivalent sized thermal plants - a bit of a shock considering the huge capital cost. These were second generation designs very similar to those in use in the USA. There was evidence of covered up leaks but no scandals of expensive incompetance - the plants seemed to be as economicly run as any. I suggest you look into the amount of taxpayer money that is going into those similar US plants - which are currently the best you have in service so "US knowhow" is no excuse when the knowlege was shared and the plants are of that vintage and of almost identical designs - Westinghouse was busy in both places and their newest designs that have never been constucted are very similar.

What Nuclear power is is safer than all other power methods of it's availability and economy. It even beats hydro.

I thought the whole point of what I've said on this long thread is that evidence is superior to blinding people with bullshit. Remember that anybody that pushes a single power source over everything else in every circumstance is selling something and also lying a great deal.

Re:Standards!

[Firethorn]

I suggest you look into the amount of taxpayer money that is going into those similar US plants - which are currently the best you have in service so "US knowhow" is no excuse when the knowlege was shared and the plants are of that vintage and of almost identical designs - Westinghouse was busy in both places and their newest designs that have never been constucted are very similar.

Virtually none. Companies pay for their own certifications, inspections and whatnot. Insurance starts as self-insured, then there's something like 200 million of private insurance per plant, then there's a group coverage where all the plants pitch in if there's an incident that goes past the $200 million, that's sitting somewhere around 9.5 Billion. Only after that does the government step in. Even counting TMI, the government's never paid out. Little fact: There are chemical plants in the USA that would cause similar levels of contamination and deaths, if not more, if they suffered a Chernobyl type containment failure. They don't have that much insurance.

Really, Britain seems to be a mostly isolated incident. Another article

I thought the whole point of what I've said on this long thread is that evidence is superior to blinding people with bullshit. Remember that anybody that pushes a single power source over everything else in every circumstance is selling something and also lying a great deal.

I believe that I've stated a number of power sources - I've just noted that many of them have already been maxed out, or are of limited use/restricted in install areas. On the one hand you complain about me brining up coal, and on the other complain about me not mentioning other power sources?

Wind can make sense in certain areas, solar makes sense in specific circumstances such as where a grid connection is impractical. Solar heating is very economical; I've tried to get my grandparents to install it for their hot water needs. Of course, they live in Florida, I live in North Dakota. I've looked at solar heating; but I just don't get enough sun and it'd need to wait until I replace the roof anyways. I've looked at wind generation, but I wouldn't be able to raise a tower high enough to get consistent winds for a price that'd come anywhere near the $.08/kwh I pay for electricity.

Good thing we have the DMCA

[tsstahl]

If this did come from The Outside, then this is the first domestic instance of nuclear terrorism (that we know of).

It is a darn good thing we have the DMCA to have their AOL accounts pulled, and prosecutorial power over their actions.

I feel quite relieved and secure now.
---
On a serious note, I really hope the problem was stupidity, or malfunction instead of malicious intent, no matter what the agenda of the attacker.

Re:Good thing we have the DMCA

[Anonymous Coward]

I say that if they had been so stupid to let outside data even come close to the nuclear plant, it would prove that Darwin is still right :-)

Re:Good thing we have the DMCA

[Anonymous Coward]

NO SHIT!!!

You missed one....

[iknownuttin]

FTFA: "What is happening in this marketplace is that vendors will build their own (network) stacks to make it cheaper," Peterson said. "And it works, but when (the device) gets anything that it didn't expect, it will gag."

Sounds to me that the vendors under-engineered their network and still charged mega-bucks for it. The auditors, I'm sure, are making the most out of this to justify their fee.

Nothing to see, move along - I'll say!

Re: The reason?

[neoform]

musta been the slashdot effect..

Redesign the entire infrastructure

[packetmon]

Firstly I would re-design that entire infrastructure and rid that power plant of incompetent IT people. Secondly I would hold those in power responsible for 1) not having failover measures in place 2) not having a stable and robust enough infrastructure in place 3) obviously not being SCADA compliant. If they can't pass IT security implement simplistic measures such as a properly designed network, it makes me wonder about the physical security aspects of it. What am I paying higher taxes for everytime the gov cries about strenghtening infrastructure when they couldn't even avoid something as stupid and as simple as a 1) safe 2) stable network. Why wasn't there any failover who knows. Insanity when three different agencies can all come down on one agency instead of WORKING with that agency to take corrective measures. US Tax dollars at work. We need to redesign infrastructure and some of these idiots in office.

Re:Redesign the entire infrastructure

[Detritus]

When you get back to the real world, let us know. You don't just wave a magic wand and completely redesign and reimplement a highly complex safety-critical system.

Re:Redesign the entire infrastructure

[Joe The Dragon]

It's not the IT people PCL are coded by EE not IT people.

Re:Redesign the entire infrastructure

[mrcdeckard]

i think the fact that an unforeseen erroneous condition caused the plant to *shutdown* and not *meltdown* is a pretty good indication that it was designed quite well.

There will always be unforeseen situations. The key is for the system to shutdown in an orderly fashion. In programming, this is accomplished through use of error traps.

Now, the hysteria surrounding terrorism is another thing the plant engineers have to worry about.

i just wonder if and when we get to put this hysteria behind us, and get along with our lives. unfortunately, terry gilliam's brazil is on a constant loop in my mind these days. . . .

mr c

Re:Redesign the entire infrastructure

[packetmon]

A highly complex safety-critical system? Oh you're right. It's ok if it fails. Its not like its safety-critical or anything. Besides it was only the network that caused it to collapse. You're right no need to wave a magic wand and have people collaborate on setting up something that works. We'll wait for true blue self defending/repairing networks to be available and let the network fix itself. Wow. Do you by any chance work for Cisco or someone else spouting these "Next generation self defending networks"?

Re:Redesign the entire infrastructure

[Anonymous Coward]

Hahah, man. I have never seen someone scream "Flip the Bozo Bit" so clearly before.

Re:Redesign the entire infrastructure

[Ajehals]

i think the fact that an unforeseen erroneous condition caused the plant to *shutdown* and not *meltdown* is a pretty good indication that it was designed quite well. really? you think that the loss of a power plant for a period of time due to network traffic is a sign of "quite good design"?

This might sound unreasonable but I would never expect a power plant (which has a lot of things depending on it) to shut down unless there was a major failure of a component or some other safety risk. Network traffic on its own, or its effects shouldn't ever be the cause. In a nuclear power plant you control ALL the nodes attached to the network, the nodes attached should not be in a position where they can saturate any individual node to the point of failure, especially if that failure causes a shut down of something as critical as a power station.

I can think of times where I have seen massive network spikes usually caused by issues with routing on fairly non-trivial networks, or loops where mistakes have been made and policies have not been followed, (lack of sleep or lack of patience), but then comparing an advertising companies internal network at 3am, or a paper factories network at midnight to a nuclear power station is taking it a little far.

There will always be unforeseen situations. The key is for the system to shutdown in an orderly fashion. In programming, this is accomplished through use of error traps.

That would be fair if we were talking about a software failure after some sort of unforeseen environmental issue, it would even be OK if an auto plant stopped production because of an unforeseen fault, and whilst power plants should certainly fail safe, they should be robust enough that a situation where failure is the only option is extremely difficult to achieve. whatever happened to redundancy?

Now, the hysteria surrounding terrorism is another thing the plant engineers have to worry about. As for the external angle terrorism or not, I doubt it. If there is a system that can be brought down by weight of traffic, and that system is important enough that failure requires a power-plant reboot (:)) then there needs to be an air-gap. Someone up thread suggested an employee's laptop with a virus as a possible method of infection.. Who in the hell allows an unchecked laptop of any description onto their LAN? never mind a network that also contains components that run a power plant!!

I would suggest that this is hype to 1) keep terrorism at the top of everyone's agenda, and make people feel unsafe, after all that sells papers and grabs viewers (which in turn sell advertising) 2) deflect some of the negativity that this incident would produce (I wish that I could blame terrorists for my mistakes sometimes... "no that project plan... I haven't got it, but I'm checking to see if my poor time management is caused by terrorism or simply my inability to organise my resources properly") and 3) Security risks presumably attract additional funding, sureley it would be nice to get an extra few million in the next budget.

Honestly, this probably shows a component failure and some poor design, understandable, but unacceptable in this area. If and I say If with some considerable doubt, this turns out to be, or is reported as an external event, then whoever enabled external network access to what appear to be critical systems within a nuclear power plant on the US mainland need to be identified and punished, together with the contractors who built or maintained it, the managers or consultants that assessed and managed it and the politicians who have responsibility for public safety. But as I said, it will probably turn out to be a simple component failure and some poor design.

Re:Redesign the entire infrastructure

[that this is not und]

Be careful. If you wave your hands around wildly like that for too long, you're likely to fly up into the air and bump your head against the ceiling.

Re:Redesign the entire infrastructure

[grumling]

Well, your plan will mean that you will replace "incompetent" IT people with "inexperienced" IT people. Better to use the FAA model of accident investigation, where liability is limited and the true cause is determined (and published). That way, engineers aren't worried about being sued for a simple mistake and the entire population benefits by learning from other's mistakes.

This story smacks of a slow news cycle and nuclear fear mongering.

Re:Redesign the entire infrastructure

[mrcdeckard]

sorry, i was going on the information another poster provided: it was not external network -- that is, it didn't happen over their DSL line . . .

a sensor evidently went haywire and started to dump a ton of data out on the internal sensor network. the way i imagine it, the metric shit ton of sensors in the plant are all networked in some fashion. one went bad, generating the analog of a DOS attack. the plant SHUT DOWN. this is good. this is better than chernobyl, eg. way better.

sure: redundancy, discrete wiring, etc., etc., etc. you CANNOT design for ALL possible scenarios. I'm sure (or at least expect) the system will be reworked in the aftermath of this event. but a shutdown is not necessarily a bad thing.

I agree shutdown should be something difficult to achieve, but we don't really know the details of what happened. I just know from my experience that it's difficult to achieve real results, and very easy to stand by and say, "gee, that could've been done better."

mr c

Re:Redesign the entire infrastructure

[stonecypher]

That depends. If the system shows repeated failure, sometimes it can be important to start on a replacement system. Most flaws can be repaired, but every so often you find an architecture built on unsound principles, or a core design failure on which other systems are irrevocably dependant.

Now, I'd like to think that every nuclear plant in the country has a better control system (not that I'd know.) That said, if I walked in and saw a defect rate that PSP and TSP said were a death spiral, you'd better believe I'd suck it up, put down the cash and buy a second engineering team to work in parallel on a replacement product. The only thing worse than a nuclear plant which is radically over budget because of critical system redesign is a nuclear plant which is reduced to hot gas by the lack of critical system redesign.

I applaud you for suggesting caution when discussing the gutting and re-stuffing of the fish. That said, "back in the real world," there are times at which starting over is significantly less expensive in time and money than fixing what you already have. They're rare, and I should hope they essentially don't exist at facilities with safety concerns at this level, but it is important to acknowledge that a software system can be so large a failure that replacement is the better option, so that when you do in fact hit such a situation, you handle it appropriately.

At times like those, I don't think I could act without a statistical defect control practice. Otherwise, getting out of analysis paralysis would seem to consist of the worst and scariest guess ever.

Re:Redesign the entire infrastructure

[bloobloo]

Yes, it is okay if it fails. You set the actuators up so that if they lose signal then they fail in their safe position. Then having done the design you carry out a HAZOP to make sure you've caught the problems.

Political FUD

[Bellum Aeternus]

As usual, the American government is looking to extend its control over things. "Oh noes, look what terrorists might have done. Homeland security needs more funding and less oversight to prevent this in the future." When will people learn to assume the government is lying first, then wait for them to prove themselves right later?

Re:Political FUD

[Anonymous Coward]

Every government does this type of thing, especially the socialist, communist, and religious-controlled ones. The American government may suck, but it's far better than the alternatives.

duh.

[twitter]

Do investigators also want to know how a "data storm" could have caused a nuclear plant to shut down?

The two questions, where and how, will be answered at the same time.

What network technology were they using?

[Angostura]

Isn't it a bit odd that they were using a non-deterministic network - something like Ethernet, by the sound of it. Back in the early 90s, I was always told that networks like Ethernet were great for office apps, but not where you wanted guaranteed times for message delivery. For that token ring, FDDI and the like were better. What is the network infrastructure of choice in a nuclear power station?

Re:What network technology were they using?

[mplex]

Using Ethernet is not odd, that's literally all there is these days. Sure, there are technologies like Infiniband, but Ethernet is far and away the cheapest and most widely supported networking standard. It sounds like they were experiencing a broadcast storm from a locked up device. I can't tell you the amount of times I've seen stand-alone devices lock up on a busy network because of a bad TCP/IP stack. Often times they will flood packets, especially broadcast frames. There are protections against bad devices such as broadcast limiters and a number of features that protect and limit unauthorized or undesirable traffic.

Ethernet isn't perfect but it's the only realistic option. Managed properly, it can be very reliable. The biggest problem I see from this article is that there is a lack of regulation and testing of the equipment that goes in to these plants. These poor TCP/IP stacks should have never gotten past the testing phase when it comes to a nuclear power plant.

Re:What network technology were they using?

[Eravnrekaree]

I find it particularly astounding that a nuclear power plant control network would have any connectivity to an external network. The article mentions the traffic flow may have come externally. That a nuclear power control system is anywhere near the internet really is quite disturbing. The article also mentions infected Windows computers contributing to the outages in 2003. I find it interesting that computers involved in electrical grid would be connected to the internet or have such lax security, and even run Windows of all operating systems at all. It really is inexcusable for security to be so poor. Simply keeping network programs running as non priveleged users in a jail one would think would be basic, to protect against exploits and systems becoming corrupted.

Re:What network technology were they using?

[Bo'Bob'O]

This are PLCs we're talking about, there are loads of network, protocol and connection systems, proprietary or otherwise, for all ranges of complexity.

Even stupider

[packetmon]

After yet re-reading, I find this government even more insanely stupider than I would have hoped for... Such failures are common among PLC and supervisory control and data acquisition (SCADA) systems, because the manufacturers do not test the devices' handling of bad data, said Dale Peterson, CEO of industrial system security firm DigitalBond.

"What is happening in this marketplace is that vendors will build their own (network) stacks to make it cheaper," Peterson said. "And it works, but when (the device) gets anything that it didn't expect, it will gag." So you mean to tell me pretty much there is no enforcement for manufacturers to maintain compliance on their products even if those products are going into a nuclear *ANYTHING... Which on the worst case scenario could cause catastrophe, yet we have regulatory commissions on the flow of ketchup, regulatory commissions/directions/etc., on weight loss products, lipsticks, etc. (FDA), but this place is not concerned with nuclear plants. Sinful.

Re:Even stupider

[fluffy99]

This is pretty common. Also consider that the PLCs are usually custom programmed by the end-user and bad data is usually not tested by the programmers either. Heck, there are tons of commercial network devices that behave very badly when face with too much or incorrect data. Try running a full-blown security scan on your network and see what pukes. I have to go power cycle a bunch of Intel piece-of-crap print servers every time I do a port scan. Don't even get me started on the crappy snmp implementation on some major brand UPSs and HP JetDirect cards.

Brown's Ferry *AGAIN!?!??!*

[ewhac]

People with longer memories may recall that Brown's Ferry had a massive fire a couple decades ago that burned in the wire racks underneath the reactor control room, very nearly destroying the staff's ability to control the reactor at all. It became a cause celebre among the anti-nuclear crowd alongside Three Mile Island.

At least their reactor failed to "off" this time...

Schwab

Re:Brown's Ferry *AGAIN!?!??!*

[cascadingstylesheet]

>At least their reactor failed to "off" this time...

It didn't just "fail to off", they manually shut it down. They followed procedures and placed it in a safe condition. No need to sensationalize it.

Re:Brown's Ferry *AGAIN!?!??!*

[AaronLawrence]

Only barely. Sounds like it was very close to a Chernobyl. If their makeshift coolant pump had also failed for some reason (like many others had already), then it would have been exposed core and meltdown time.

Even in a free market democracy, people are complacent, careless, greedy, dumb and just plain human.

The last thing we need

[Tablizer]

...nuclear viagra

Re:The last thing we need

[Cctoide]

ENL4:RG3 UR FU3L R0:DS! Z1R:C0NIUM R3:INF0RC3M3NT - CH3:4P35T PR1:CES!

Wow!

[cashman73]

We slashdotted a nuclear power plant! :-)

Behold, the power of Slashdot!

a cat

[eille-la]

A cat fell asleep on a keyboard

It's not stupid.

[twitter]

Seriously, how stupid do you have to be to think "OMG, Haxxors?" Answer: work at Homeland inSecurity, or be a Congresscritter. They already figured it out. It was a controller for a specific piece of equipment that flooded the network and triggered a bug in the variable-frequency-drive controllers for pumps.

As someone who used to work in system's engineering for a sister BWR, I think the inspection is a good idea. Oh, there's dumb and there's nuclear dumb but this is not a case of either. Nuclear dumb involves putting machine guns nests inside the plant. Finding the root cause of the accident is a good idea.

Handwaving about a PLC device won't do. What ultimately caused the PLC malfunction needs to be answered at a component level. There's going to be something wrong with it and that should be reported and every other device like it needs to be ripped out and trashed. If there is not component failure, there's a software problem which also must be understood.

Yes, it could have been hackers. The "internal control network" might at some point hits a desk that's connected to the wider world. It could be something mundane and unintentional, like an operator's virused up laptop.

An outage like that is something that's going to have both NRC and corporate ass-chewers looking at everything. Corporate might want to paint a nice picture for the NRC, but the poor devil that lies to them goes to jail. In either case, the problem will be identified and eliminated.

You might also have noted in the article that this is not the first plant to go thumbs down over some winblows born virus. In 2003, the slammer worm caused havoc at an offline Ohio plant. Yes, that was hackers. They did not mean to do it, but the plant's systems were open to it and failed. That's not acceptable from any standpoint.

Despite the better advice of the computer people at the plants, Entergy is a big M$ Partner. They take the big dogs out fishing and sell them the works. Ten years ago, M$ had something worth while and interesting. It was used in places it should not have been. Worse, the flaws from ten years ago have not been addressed or fixed. A good clean up is in order.

Re:It's not stupid.

[Z00L00K]

In general, software installations at industrial plants has to have a lifetime of decades. This means that even if the software selected isn't the cheapest initially it may prove to be the cheapest in the long run since it will have to be stable. Each unplanned stop of a large plant due to an undefined malfunction will take time and cause costs in the million dollar class.

One way to safeguard against data storms to some extent is to create network segmentation so that a malfunction in one part of the plant is less likely to affect the operation of another part of the plant. In some cases it may even be necessary to consider if a certain functionality actually shall be computer controlled or not. Cooling pumps of a nuclear reactor could be one such feature. This doesn't exclude computer monitoring of the devices, but a computer malfunction shall never be able to disrupt the core stability.

To err is human, but using computers you can really mess things up.

Life at a power plant.

[twitter]

Firstly I would re-design that entire infrastructure and rid that power plant of incompetent IT people.

You need to find the root cause. You don't know it yet, so you don't really know what to do.

Chances are, the cause has been written up by the four or five systems engineering people in charge of the plant. They ARE competent, but they are never given the resources they need.

Why wasn't there any failover who knows.

There was a failover - they overrode the broken thing. Had the operators been gassed, the plant would have turned itself off when the water level got too high or low. This is a big deal but ultimately the plant was safely shut down and no one got hurt. It's designed to do that even if you could shear the feed water pipe off and they did not let the new fangled control network mess with that.

A BWR dude, downmodded fast.

[Erris]

Wow, someone who's worked in a BWR down modded in less than ten minutes. Nice work, trolls.

Saturday is bought to you by the color Orange

[BillGatesLoveChild]

> data storm

Is that a nice way of saying they were downloading pr0n?

> US House of Representative's Committee on Homeland Security called further investigate

Boss: "So we don't have the backups for the first two weeks in April"
Employee: "Yes Boss. They were obviously misplaced by terrorists"

When Homeland Security is done, my refrigerator door was left ajar last night. I think it was terrorists too. Think I'll phone this one in.

The way I think the conversation went

[rush22]

"Ok, techie, give me the jist of it."
"It seems the problem was with the NC9828A chip"
"Oh? And what was the problem?"
"It melted, basically. It went bonkers."
"Ah, and then what happend?"
"Err... it caused the shutdown."
"But how?"
"Well, I presume the AH-982's got deluged with data, so they shut off."
"Ah, so it was some sort of data thing."
"Kind of, the failing chip would start sending data in the network t--"
"Hey, it's like a storm of data! Hah! I get it!"
"Umm, basically."
"Oh man. A data storm! I better tell the NRC"
"Ok, sure."

Later...

"Sir, I have the cause of the shutdown, it was caused what the tech guys here would call a data storm."
"A data storm? Wow. So your reactors got a bunch of bad datas, right?"
"Errr.. kind of, the microchips melted."
"Data can do that?"
"Yeah, it's like a storm on our, uh, logic networks. I guess that can melt the microchips"
"Uh oh. Maybe this storm came from outside the plant! One of those hacker attacks!"
"Hmmmm, the guy said it melted, but I suppo--"
"Oh crap I better inform Homeland Security!"
"Ok, sure."

Later still...

"Yeah, we had a data storm and it melted the reactor networks."
"How did this data storm happen?"
"I don't think they know yet, but it messed up big time."
"My God. Do you realize this could be Al Qaeda?!!"
"Could realize wha--"
"Al Qaeda! Terrorists. Internets terrorists."
"I don't know if the reactors are hooked up to the Interne--"
"Listen. Keep this quiet, but make sure you tell everyone you know. These reactors are not safe! No one is safe from the terror!"
"Well, it was a data storm. Can terrorists make data storms?"
"Yes. They caused your meltdown."
"No, no, the microchips melted down because of the storm. A meltdow--"
"In the terror business, there's more than one type of meltdown, you just let us handle this."
"Ok, sure."

Re:The way I think the conversation went

[KillerCow]

Awesome. I didn't even think of this as an example of telephone, but it pretty clearly is.

How else can a simple malfunction, causing an unintended cascade, leading to a failsafe being triggered turn into a terrorist threat? Details got cut as the report went up the chain, and substitutions and errors accumulated until the ultimate message was changed.

Re:The way I think the conversation went

[Anonymous Coward]

The game of telephone continues with a news report, a slashdot summary, and finally slashdotters expounding with conviction on something they just read a 50-word summary on. And making jokes about how the summary came about.

Mod parent up ...

[b0s0z0ku]

insightful. He's worked in a nuclear power station and seems to be clueful.

Re:Mod parent up ...

[Anonymous Coward]

Maybe if he wasn't such a child with "winblows" and "M$" everywhere, people would be more likely to mod him up. Instead, he looks like a fucking asshole and turns people away.

Re:Mod parent up ...

[that this is not und]

I've worked in a nuclear power plant, too.

So have you, for that matter.

On the internet.

Yeah, yeah.

Re:Mod parent up ...

[b0s0z0ku]

Maybe if he wasn't such a child with "winblows" and "M$" everywhere

Sadly, he's right, though. MS stuff isn't appropriate in life/safety-critical applications. Maybe for running e-mail systems, but not for anything that'll have bad physical effects if it fails. Embedded systems, specialized OS's, or even minimal UNIX systems do the job with far more reliability.

-b.

Storm in the tubes

[cyberianpan]

I've worked in IT a while now & have never heard of a "data storm". This reminds me of

And again, the Internet is not something you just dump something on. It's not a big truck. It's a series of tubes. Ted Stevens We have plant managers concocting an odd metaphor that will only further confuse senators. Why can't they just use actual language - is it because they are deliberately trying to confuse the issue to avoid blame ? The same way the red herring of terrorism is being floated re this ? In fact it is more serious that

1) They can't describe what happened

2) They can't tell if outside interference, whatever the nature occurred

3) That this might have an internal/design cause
... than if "terrorists" did it.

Re:Storm in the tubes

[ichigo 2.0]

Because "spike in network traffic" sounds lame. Data storm, OTOH, sounds cool and dangerous. Contact Jack Bauer quickly! We need to open a new port for the nucular plant, so the terrorists don't destroy us! And while you're at it, give us more money so we can prevent these awful storms in the future!

Re:Storm in the tubes

[Jugalator]

I've worked in IT a while now & have never heard of a "data storm".

Maybe it's the precursor to a logic bomb!

Wow, can't you request article deletion from Wikipedia on the basis of "ridiculous term"?
Or better yet, mind erasing for the very same reason... :-p

Re:Storm in the tubes

[binarysins]

I usually hear them called packet storms, but they happen and "storm" is usually somewhere in the description. In fact, we were just troubleshooting exactly that at my work last week and the network admin used the exact phrase "packet storm".

Re:Storm in the tubes

[ultranova]

4) Apparently the computers which control a nuclear plant are connected to the public Internet, allowing anyone in the world to send them commands, viruses, or random garbage, therefore allowing them to gain remote control over the reactors. Oh, and according to TFA, another nuclear plant runs Windows (since it was hit by the Slammer worm).

Someone please tell me that I'm wrong and the people who design these plants aren't this stupid. Please ?

Re:Storm in the tubes

[Anon99]

>I've worked in IT a while now & have never heard of a "data storm".

I used to work as embedded developer, and we used that term.

It was used in embedded communications when one or several devices went bonkers and flooded common bus.
Bit like packet storm, but without IP or other packet protocol, so it was called data storm.

It stands to reason that in nuclear plant there are a lot of old fogeys, so company jargon might be bit outdated and odd sounding to outsider.

Re:Storm in the tubes

[canuck57]

is it because they are deliberately trying to confuse the issue to avoid blame ?

That is the truth. 25 years ago when deploying ArcNet and later thin net we isolated our production systems, while we didn't have in-securable OS issues back then we realized PC/user behavior was not worth the risk to automated machinery. Today, it is more critical than ever.

But the gross incompetence here goes to management. They should all be fired with cause and with prejudice and shut the system down until it is designed correctly. In todays day and age there is no excuse for this other than gross negligence.

I can just hear some senator that owns the stock in a mutual fund, "We need a law..." -- BS. We need to shut these idiots down now and pull their regulatory licenses to operate the facility.

Re:Storm in the tubes

[autocracy]

Yes, actually... yes you can... neologism.

Re:Storm in the tubes

[Anonymous Coward]

When I think of datastorm it reminds me of various valid computer related terms that have the word "storm" in them like broadcast storm, packet storm...etc which all mean the network is saturated by nonsense making it hard/impossible for real work to move over the wires.

There are a few ususal suspects:

Poorly designed protocols AKA congestive collapse
Virus like the slammer worm.
Routing l3 / l2 loops.

I disagree with parent that "data storm" is an inappropriate, unususal or watered down term that is not used regularly in the industry.

The article however I doubt has anything to do with standard protocols that we all know and love... The central theme seems to be complex failure caused by well known and preventable conditions.

Heck even the control systems in our vechicles prioritize all messages to specifically protect against the result of malfunctioning modules or external interfaces spamming the control channels.

A control system for a reactor that has not been tested under these kinds of conditions makes me wonder if Homer Simpson designed it too.

There are several technical problems with this system and I will attempt to list all of them that come to mind from a position of extreme ignorance...like most slashdot posters :)

1. Excessive chatter/command most likely retorhical in nature should be a detectable condition flagged by neighboring modules in near realtime. (This should not require an investigation)

2. Each module should have logic that specifically limits the messages "flap detection"..etc (especially rethorical ones) sent in realtime. Causing a flood should not be an acceptable failure mode for any module.

3. The communication channel and relevent network controllers on all interacting module should be wide and powerful enough to absorb spam from several malfunctioning modules.

4. All traffic should be prioritized in this application I would go as far to say that control traffic needs to be conveyed out of band of the ususal sensor traffic.

It literally "boils down" to design and QC oversights.

Re:Storm in the tubes

[Splab]

I don't see the big deal. Putting the controller on the internet makes sense, just like you put ATMs and bank transactions on it. It's not like theres a big red button when you connect to it that says "Commence meltdown". Nuclear reactors might have software to set some of the controls, but they definitely have hardware fall back that can overrule the software decision.

One should never ever count on software to handle emergency shutdown of anything - you simply cannot risk having the emergency shutdown waiting to be scheduled on a processor.

Re:Storm in the tubes

[bloobloo]

The plant I'm working on the design of at the moment will have a VPN connection so that we can monitor it's performance from abroad. Running private cables over 7000 miles would not be feasible.

Re:Storm in the tubes

[vakuona]

Because data sounds technical, and storm sounds bad.

Two key ingredients if you want people to
a) Think you know what you are doing and,
b) Scare them enough into doing something.

Re:Storm in the tubes

[PPH]

It was used in embedded communications when one or several devices went bonkers and flooded common bus. Bit like packet storm, but without IP or other packet protocol, so it was called data storm.

Back in the old days, this happened (occasionally) with old coax-based ethernet LANs. Decent 10Base-T hubs have provisions for blocking individual devices that 'go nuts' and keep them from screwing up the entire network. Its still possible for a single device to lock up, but it would be a very poorly designed network that would allow it to take down others with it.

Its interesting that they are pursuing a theory that this may have had an external origin. I've seen this happen and it was traced to people that expect to be able to plug a laptop into any available network port and access YouTube. Or worse yet, people who expect to be able to use IE on any available factory terminal to access their web mail, visit MySpace, and other malware-infected external web sites. Meanwhile, some control function might be waiting for a command and, if its not written to deal with network latency or outright failure, Bad Things happen.

Re:Storm in the tubes

[RockDoctor]

4) Apparently the computers which control a nuclear plant are connected to the public Internet, allowing anyone in the world to send them commands, viruses, or random garbage,

Might I recommend you to RTFA?
The "data storm" appears to have been on a internal network (not seemingly connected to anything apart from other internal networks), where a data acquisition and control device barfed on some bad data and started to spew garbage onto the network. Inadequate data validation combined with inappropriate or ineffective error handling. Software fault.

Re:Storm in the tubes

[Anonymous Coward]

Yes, actually... yes you can... neologism.

Actually, no - it's a neologism with a lot of documented use. It's like in the real world: if a random schoolkid makes up a word, you can just laugh, but if a TV personality or a noted technology writer makes up a new word, we all have to suffer. =)

/.ed Again

[rssrss]

/. again.

SOCKPUPPET!

[Keith Russell]

Erris is Twitter's goddamned motherfucking sockpuppet.

Erris == twitter

[dedazo]

Wow, shilling your own posts does work!

This is scary

[kilodelta]

They device manufacturers create their own stacks and then don't test them? Are we just asking for a nuclear accident, or does this strike anyone else as stupidity?

At least it wasn't connected to the public internet. Can you imagine the havoc THAT would create. But I wonder why they're treating this as a criminal thing. Did someone modify a device?

Risks

[Ronin441]

There's an excellent summary on Risks.

Good news.

[Sj0]

Great news, guys. This is going to be a non-issue. People are freaking out because a digital device is involved, and freaking out because a nuclear power plant was involved, but I do industrial control system and DCS design for a living, and I'll tell you right now, that you simply can't access control networks from the outside. There are seperate, often redundant networks, and even then, depending on the way the plant was designed, we're talking modbus plus or something that PCs don't normally access.

Re:Good news.

[ScrewMaster]

We had a customer that wanted to know if our monitoring system was protected from viruses, because they were worried about a Windows worm getting into their plant DCS via the Modbus interface. I tried to explain how that wasn't really an issue, that it would be very difficult to transmit a worm via Modbus registers over an RS-485 connection, but I was cut off with, "we know it's a problem, how are you going to deal with it?" My own thought was, "you people are going to be a problem, how are we going to deal with you?" but I kept my mouth shut.

Unfortunately from a security perspective, Ethernet is becoming more and more common for industrial process control nowadays. Hell, even the likes of Honeywell are pushing Microsoft-centric protocols like OPC (the thought of OLE and DCOM anywhere near a nuclear plant doesn't exactly fill me with confidence.) But I agree: a nuclear plant is likely to be using older technology and individually-addressable devices wouldn't be accessible from the outside. On the other hand, there's usually a Windows box or two somewhere that does interact with the internal network. Even if such machines start out isolated from the Internet at large, they eventually tend to get connected, because lazy people will usually put their convenience before safety. For that reason alone, I think Windows should be banned from critical infrastructure.

One of our biggest customers uses our equipment and software to perform data acquisition for plant control purposes: we dump data continuously via Modbus to a Honeywell DCS. However, they run our servers and client machines on a dedicated subnet with no physical connection whatsoever to the company in-house network or the Internet. That's because their IT people are actually very intelligent and realize that all networks can be attacked, all operating systems can be compromised, and that the only way to keep a mission-critical system safe is to keep it isolated. If it doesn't need a remote connection it shouldn't have one.

That same principle should be applied to power plants, nuclear or otherwise, or any industrial facility where substantial damage could result from unauthorized or malicious access. I'm always amazed when I hear otherwise. You just want to grab these people by the necks and shake them, all the while asking, "Are you goofy? What's the matter with you!"

Re:Good news.

[PPH]

That same principle should be applied to power plants, nuclear or otherwise, or any industrial facility where substantial damage could result from unauthorized or malicious access. I'm always amazed when I hear otherwise. You just want to grab these people by the necks and shake them, all the while asking, "Are you goofy? What's the matter with you!"

That works fine until its the plant manager who insists on having a real time process monitoring app. running on his laptop and then be able to visit the goatse site at the same time. Grabbing and shaking such people by the neck is a carrer limiting move.

Re:Good news.

[ScrewMaster]

Grabbing and shaking such people by the neck is a carrer limiting move.

Yeah. But it's fun to think about. Although, in some ways it's too bad our society isn't run more along the lines of the Klingon Empire. I mean, a Klingon underling wouldn't bother grabbing a boneheaded supervisor by the neck and shaking him. He'd execute the bastard on the spot for incompetence and disloyalty to the Empire ... and take his job.

Klingon power plants are run very securely, I understand.

Why the hell...

[VorpalEdge]

Why, exactly, are the failsafe systems of a nuclear power plant hooked up to the Internet? Stuff like this needs to be completely sealed off from outside intrusion. And I mean completely. You don't need an internet connection to operate a power plant.

Granted, there's probably valid uses for it, but the computers with a 'net connection need to be isolated from the ones that actually keep the plant operating.

Re:Why the hell...

[VorpalEdge]

Hmm, I probably should have RTFA

"The deluge of data was apparently caused by a separate malfunctioning control device, known as a programmable logic controller (PLC)", as opposed to a DoS or slashdotting or some such.

Oh well; ignore me.

Wait a cotton pickin' minute here...

[Torodung]

Who in God's name connects a plant's coolant regulation systems to the Internet? How could it be an outside agent when the "data storm" happened on the plant's INTERNAL network.

The article says that explicitly. "Internal network." The DHD is worried about outside agents penetrating the plant personnel, not someone with a laptop uploading a virus like Jeff Goldblum in "Independence Day."

If there *was* such a "data storm" attack, it would _have_ to be caused by an inside saboteur. The plant needs to focus on HUMAN security, not computer security. Either that or they need to reconsider a faulty design.

But can we try, just try, not to write completely hysterical baloney? Hysterical baloney is a tradmark of "Homeland Security," and they might see fit to sue.

--
Toro

p2p?

[Menelkir]

Maybe someone inside the complex are using some p2p network...

Scary

[Anon99]

From article

---
Such failures are common among PLC and supervisory control and data acquisition (SCADA) systems, because the manufacturers do not test the devices' handling of bad data, said Dale Peterson, CEO of industrial system security firm DigitalBond.

"What is happening in this marketplace is that vendors will build their own (network) stacks to make it cheaper," Peterson said. "And it works, but when (the device) gets anything that it didn't expect, it will gag."

In many cases, a simple vulnerability scan will even cause the devices to crash, Peterson said. During tests in an electrical substation, Nessus running in safe scan mode crashed devices, he said. In some cases, sending out broadcast data on the network will crash several of connected devices, he added.
---

Scary, and really strange.

All pipes, pumps and other physical components have to pass multiple safety and quality checks before they are allowed as plant constructions materials.

While it seems that computing components don't have even rudimentary interoperability checks.

It seems that US nuclear safety board (or whatever is the name) is dangerously computer illiterate. Not a good thing.

One of my programming teachers was a retired system architect for Finnish nuclear power plant, and he had bit different stories to tell how they tested computer systems.
In one lecture that turned into a story of how they did things taught me more about software testing than I have gotten on any other course after that :)

So fortunately it is not that bad everywhere.

Re:five words

[Anonymous Coward]

"The power plant doesn't work"

Here is a citation

[aepervius]

First link in google.
list of incident at brown ferry (click on candle)

In Russia ...

[Anonymous Coward]

Russian nuclear engineers flood you of the lack of your "Research & Development" ...

Network stack has too high priority

[Esben]

I have actually seen such a problem myself: Controllers crashing because someone was testing the network. The problem was, ofcourse, that the CPU spent a lot of time to handle the amount of packages on the network and therefore didn't have time enough for it's real-time application. (It didn't help that the platform didn't support DMA.)

Solution: Make the network interrupt handler threaded and prioritize it below the real-time application. Sure, that doesn't help the SCADA performance, but you have to make sure that the real-time application meets it's deadlines no matter what is going on on the network. I simply don't buy that you can secure a network stretching over more than 1 meter against "data storms."

Here's what bothers me most about this...

[JetScootr]

In the letter from the Committe on Homeland security to Nookoolur Regulatory Commission about it,
In accord with current regulations, NRC staff decided against investigating the failure as a "cybersecurity incident" because 1) the failing system was a "non-safety" system rather than a "safety" system,

If failure of the component is dangerous enough to force a system shutdown due to lack of cooling, that IS a safety system. It's like saying there's no need for bracing the BOTTOM of the ladder, cuz it's such a short fall.
It wasn't the power plant that made this determination. It was the NRC.

Re:Here's what bothers me most about this...

[Anonymous Coward]

If failure of the component is dangerous enough to force a system shutdown due to lack of cooling, that IS a safety system.

It's a critical system but not a safety system. The safety system's job is to override the primary control system and shut the plant down, when necessary. Ideally, it should never have to step in and do it but shit happens.

Re:Here's what bothers me most about this...

[KudyardRipling]

Nuclear power facilities...it's one of the many factors that makes real estate...AFFORDABLE!

Oblig. Broken Arrow

[FreakyLefty]

"I don't know what's more disturbing, the fact that it happened or that fact that it happens so often we have a name for it."

Re:Oblig. Broken Arrow

[wwphx]

You'd get mod points from me for humor if they were mine to give. I still think Face/Off is a better movie, though. Slayter doesn't do it for me, but the Cage/Travolta combo was excellent!

important safety tip ..

[rs232]

"They were also using candles to determine whether or not the leaks had been successfully plugged .. The electrical engineer put the candle too close to the foam rubber, and it burst into flame"

Don't try and find sources of draft using inflammable foam and a candle ..

Don't route the backup system through the same conduits as the primary one ..

was Brown's Ferry *AGAIN!?!??!* (Score:4, AGhaaaaaa !!!, ohh God, make me a believer )

replace the Network stack ..

[rs232]

In this case it was a rogue device pumping out too much garbled date. The solution being to design nodes that isolate the network from such occurances. The devices communicate through some high level protocol that is validated by the nodes before getting on the network, not as seems to be in this case standard TCP/IP over Ethernet. The loss of two recirculation pumps because of a network event is not trivial.

Network stack has too high priority (Score:1)

Datastorm wrote procomm

[laing]

If you've worked in IT for a while, you would remember Procomm. It was probably the best PC based terminal emulation/RS-232 scripting programs. I know a few folks who still use it for automated embedded equipment telemetry & command applications. Procomm was written by none other than "Datastorm Technologies corp.".

Now back on topic: If you RTFA it says that there was an embedded networked controller that was "babbling" (flooding the internal network with unwanted traffic). Unless some hacker from outside penetrated their firewall and reprogrammed the embedded controller, this incident is probably due to a simple hardware failure. They should be looking at setting up redundant network interfaces for their "mission critical" systems to prevent this sort of thing from happening in the future.

JSL

Perfectly cromulent

[DrSkwid]

http://kb.iu.edu/data/aibq.html

the data storm was caused by...

[Anonymous Coward]

Q.What caused the data storm?

A.When cold bits and hot bits collide a storm like formation is produced.

sometimes its just rain bits, other times it can be hails bit(or even bytes!) and also light bits(normally occur in fiber optic networks)

IP or serial network?

[heh2k]

People here are talking about ethernet and ip, but all the PLCs i've dealt with use 4 wire rs-485. Some do allow access to the serial network via ip, using a windows app. I don't know if the plant uses ethernet (using ip, or another protocol), but it's possible, if they have many devices over a large area. However, you shouldn't make that assumption.

Data Storms Have Lots Of Causes

[maz2331]

A "data storm" can be caused by lots of things, even an unstable driver causing a NIC to spew garbage packets. Or an application that hits a bug and begins spewing to the network. Or a failure of Spanning Tree causing network loops to arise (which can really mess up an Ethernet).

The wierdest I ever saw was a situation at a school where the entire network (built around high-end Cisco switches) crashed hard. It took 3 hours of troubleshooting and disconnecting various segments to finally pin down the cause. It was a little mini-switch that some teacher attached to the LAN that somehow had a meltdown and began spewing "valid" Ethernet packets with all kinds of random garbage source and destination MAC addresses, random payload, and valid checksums. No hosts were attached to the mini switch, so it had to be something in its microcontroller going haywire. This cause every switch to go nuts trying to maintain its forwarding tables ("show cpu" was 100% utilization) and resulted in no traffic going anywhere. It even crossed VLAN boundaries since all the switches had "trunk" ports using tagged VLANS, so the garbage packets still made it through the entire LAN.

These things happen sometimes. Network gear is generally pretty robust, but can still fail in wierd ways.

Bullshit FUD

[The Bungi]

winblows born virus. In 2003, the slammer worm caused havoc at an offline Ohio plant.

Not so, FUDster. The follow up report to the blackout by CERT was published in 2004 (and Slashdot also linked to it).

On page 133-134 of the report itself we find:

The Cyber Analysis sub-team was led by the CERT(R) Coordination Center (CERT/CC) at Carnegie Mellon University and the Royal Canadian Mounted Police (RCMP). This team was focused on analyzing and reviewing electronic media of computer networks in which online communications take place. The sub-team examined these networks to determine if they were maliciously used to cause, or contribute to the August 14, 2003, outage. Specifically, the SWG reviewed materials created on behalf of DHS's National Communication System (NCS). These materials covered the analysis and conclusions of their Internet Protocol (IP) modeling correlation study of Blaster (a malicious Internet worm first noticed on August 11, 2003) and the power outage. This NCS analysis supports the SWG's finding that viruses and worms prevalent across the Internet at the time of the outage did not have any significant impact on power generation and delivery systems. The team also conducted interviews with vendors to identify known system flaws and vulnerabilities.

Bold emphasis mine.

Just more of the same sad, tired bullshit FUD you're famous for. And the mods fell for it because you shilled your own bullshit with your sockpuppet account.

Re:Bullshit FUD

[cmacb]

I think the point still stands that Windows is a poor choice for secure applications. The report (as you cited it) didn't say that viruses and worms had NO impact, but simply that it was not significant. You can drive a truck through that. These institutions do NOT like to admit to error.

When the Chinese got into the State Department networks a year or so ago they trotted out an "official" to tell the press that their computers were unaffected. Only catch was that official was in the mainframe group (and the mainframes were truly not affected). Of course what they covered over for as long as they could was that the PC networks were largely incapacitated. You couldn't get to the data on the mainframes to actually do any useful work because the end user stuff is now all PC based.

The only way to have reasonable secure Windows systems is to take them totally off he Internet, better yet, power them off completely and just pay MS for the license so you can get your kick-back.

Re:Bullshit FUD

[The Bungi]

I think the point still stands that Windows is a poor choice for secure applications.

That might be true or not, but the "proof" that twitter so helpfully provided is obviously a lie, which is not a rare occurence in his body of work, if you will.

You can share your opinions and you can make value judgments all you want, and you can present them as such. Trotting up links presented as fact to hold up your lies and FUD is quite another.

better yet, power them off completely and just pay MS for the license so you can get your kick-back.

LOLZORZ and all that.

sniffer?

[jeoeoeoeorb]

Jebus, you'd think someome would have the forethought to attach a passive sniffer if the whole plant depends on on a working network.

Wrong incident, try again Fanboy.

[twitter]

The CERT report you quote is about Yet Another incident where Winblows was fingered, the 2003 Blackout, that has nothing to do with Davis-Besse besides M$.

The fine article is rather clear:

The Slammer worm penetrated a private computer network at Ohio's Davis-Besse nuclear power plant in January and disabled a safety monitoring system for nearly five hours, despite a belief by plant personnel that the network was protected by a firewall, SecurityFocus has learned.

The Slammer worm entered the Davis-Besse plant through a circuitous route. It began by penetrating the unsecured network of an unnamed Davis-Besse contractor, then squirmed through a T1 line bridging that network and Davis-Besse's corporate network. The T1 line, investigators later found, was one of multiple ingresses into Davis-Besse's business network that completely bypassed the plant's firewall, which was programmed to block the port Slammer used to spread.

By 4:00 p.m., power plant workers noticed a slowdown on the plant network. At 4:50 p.m., the congestion created by the worm's scanning crashed the plant's computerized display panel, called the Safety Parameter Display System.

An SPDS monitors the most crucial safety indicators at a plant, like coolant systems, core temperature sensors, and external radiation sensors. Many of those continue to require careful monitoring even while a plant is offline, says one expert. An SPDS outage lasting eight hours or more requires that the NRC be notified.

At 5:13 p.m., another, less critical, monitoring system called the "Plant Process Computer" crashed. Both systems had redundant analog backups that were unaffected by the worm, but, "The unavailability of the SPDS and the PPC was burdensome on the operators," notes the March advisory.

It took four hours and fifty minutes to restore the SPDS, six hours and nine minutes to get the PPC working again.

They go on to mention three other incidents, including the later whitewashed Blackout account.

Just more of the same sad, tired bullshit FUD you're famous for.

Your love of M$ has blinded you again. Why do you feel so much for a big dumb company and their software? How many screw ups does it take to convince you that M$ does not belong everywhere and they have serious issues to resolve before they can be trusted anywhere.

Re:Wrong incident, try again Fanboy.

[dedazo]

I have to admit I also thought you were bitching about the blackout so I didn't click on your silly link. But my god, this is even more stupid. I was installing the Slammer patch on some of my machines almost 7 1/2 months before anyone had even heard of the exploit. These people allowed it to get into their network, which had not been patched to begin with.

I wonder twat, who would you blame if this was some other product? The vendor, of course, as opposed to the user? 7 1/2 months is never nearly enough, I guess.

This is even less about security or whether or not "M$ Winbloze" should be run in certain situations, it's just simple and plain incompetence and SecurityFocus just milking that incompetence for those important ad impressions.

including the later whitewashed Blackout account.

Please elaborate on this, with specifics about where, how and why you believe that report was "whitewashed". Thanks.

Oh, that's worse

[The Bungi]

Oh, I see what you'e saying. So you're bitching about an offline plant whose network was reached by a worm for which a patch had been available for six months prior to an exploit being observed in the wild.

Here I'm thinking this is a massive DOS situation, but these people could have been easily nailed by that phpBB exploit on Linux for all practical purposes. That's even more fucking pathetic.

Traffic Storm

[jaredmauch]

Likely they're talking about some sort of 'Traffic Storm' (which is some type of data). I have seen and heard of a lot of devices that are very poorly designed and don't expect a lot of extraneous data on their lan. Most commonly these are things like PBX'es and small 'appliance' devices that have some simple SNMP or web mgmt capabilities. You stick them on an internal lan with lots of broadcast traffic, where there may be other interesting things going on and i've seen them either die under the interrupt load (insufficent cpu for the 10Mb or 100Mb they negotiate) or just lock-up because of what it thinks is a corrupted frame.

My evidence?

[jd]

• Make a theory as simple as possible, but no simpler.
• Do not ascribe to malice that which can equally be described by incompetence.
• Never multiply elements unnecessarily.
• The vast majority of errors blamed on computers are human errors.
• A high probability event is more likely than a low probability event.
• Having worked in nuclear labs as a programmer and totally thrashed their internal networks when stress-testing, I can say from personal experience that these people don't buy third-rate components.

Do not meddle in the affairs of wizards, for it makes them soggy and hard to light.

Re:My evidence?

[DerekLyons]

In other words, you don't actually have any evidence. You just make shit up (your original post), then handwave and smokescreen (the post I am replying to) in an attempt to cover that fact up.

Power Station Control

[Anonymous Coward]

There is a lot of ignorance and selective reading of the article going on here! I've worked in power stations, both nuclear and conventional, for nearly 20 years designing, building and commissioning control systems.
1. The control system failure did not trip the reactors - the operators did manually as per their operating instructions which dictate the action in the event of both of the affected pumps failing.
2. The PLCs and drives will not have been using Ethernet for controlling - they will simply have been connected to the site-wide LAN for monitoring.
3. Using a PLC on a non-nuclear safety piece of equipment is common practice.
4. Network/Ethernet/Data storms are a known phenomenon, and are acknowledged by the PLC manufacturers.

I first came across this phenomenon about 8 or 9 years ago. Several PLCs connected to the Ethernet network (the Ethernet connection only for remote monitoring/programming - not for control), and all had faulted, shutting down a hydro-electric power station. When investigated, I discovered all the PLCs had erased their application software, looking like they'd just come out of the box.

An IT engineer had been on site at the time replacing a blade in a hub.

On discussion with the manufacturer, this was a known problem, due to what they called an 'Ethernet Storm,' but not a problem they thought was a serious issue and needed publicising. They even had a fix, but wanted £2000 per processor to upgrade the firmware.

I pointed out that there were serious implications especially for Chemical/Nuclear plants etc. and that they should be proactively addressing the problem. They eventually agreed to upgrade the problem to something called a 'code 10' and that way all our processors would be upgraded for free. In the new revision, there was a new register called an 'Ethernet Storm counter'.

Since then, the problem has re-occured and we are now uprevving all our processors to the latest revision of firmware - which they say is now definitely Ethernet storm resistant (we'll see!).

In critical applications where control system components need to communicate with each other, we do use Controlnet, Modbus, etc., monitor for device failures and all the other good practice that Ranjan advocates, but Ethernet is widely used for non critical connections for MIS, Remote Monitoring etc. Who would foresee that a non-critical connection to an Ethernet network could erase the memory of a processor? Possibly someone who designed the equipment, but not your experience Control Engineer who would expect a piece of kit to be 'fit for purpose'.