Slashdot Mirror
Data Storm Caused Nuclear Plant To Shut Down
rs232 writes to let us know that the US House of Representatives Committee on Homeland Security called this week for the Nuclear Regulatory Commission to further investigate the cause of excessive network traffic that shut down an Alabama nuclear plant. Investigators want to know whether the data storm could have been initiated from outside the plant.
It's not the IT people PCL are coded by EE not IT people.
RTFA, bozo.
Mea navis aericumbens anguillis abundat
At least their reactor failed to "off" this time...
Schwab
Editor, A1-AAA AmeriCaptions
I work at a nuke plant as a system engineer. One of my systems are the reactor recirculation pumps, these type of pumps. I know for a fact there is no way hackers could "data storm" my pumps and there is extreme doubt in my mind that the same thing could happen at Browns Ferry. The pumps digital control system isn't even near any outside network.
However, I will fully put the blame on the PLCs. Those little suckers come in handy but if you don't completely understand every line of code and every instruction they can f_ck you over.
I also love how they say "well if you can't prove it wasn't, then it must have been".
Firstly I would re-design that entire infrastructure and rid that power plant of incompetent IT people.
You need to find the root cause. You don't know it yet, so you don't really know what to do.
Chances are, the cause has been written up by the four or five systems engineering people in charge of the plant. They ARE competent, but they are never given the resources they need.
Why wasn't there any failover who knows.
There was a failover - they overrode the broken thing. Had the operators been gassed, the plant would have turned itself off when the water level got too high or low. This is a big deal but ultimately the plant was safely shut down and no one got hurt. It's designed to do that even if you could shear the feed water pipe off and they did not let the new fangled control network mess with that.
Friends don't help friends install M$ junk.
You just have to love Browns Ferry don't you? This is the same plant that had wired its control cabling for two nuclear reactors through the same area. Then they had workers check the air tightness by using candles near their flammable insulation. It wasn't air tight and the flame of a candle was sucked into the insulation. Thus a fire broke out, $100 million of damage occurred, and control was lost of their two nuclear reactors for something around 8 or more hours. Why 8 hours? Because their fire team tried to fight the fire with portable CO2 extinguishers. Yes, for 8 hours. Until the local fire department (which they previously obstructed) put it out with water in 5 minutes. Idiot designers and idiot employees. I'm surprised that plant didn't have a meltdown before TMI. But boiling water reactors are a little harder to destroy.
This is pretty common. Also consider that the PLCs are usually custom programmed by the end-user and bad data is usually not tested by the programmers either. Heck, there are tons of commercial network devices that behave very badly when face with too much or incorrect data. Try running a full-blown security scan on your network and see what pukes. I have to go power cycle a bunch of Intel piece-of-crap print servers every time I do a port scan. Don't even get me started on the crappy snmp implementation on some major brand UPSs and HP JetDirect cards.
For a fact the network recirc pump controllers at Browns Ferry are on a private network...because 3-4 years ago the recirc system engineer tripped both pumps off sitting at his desk in the administration building while playing with the software. Oops.
As far as the "OMG they're gonna melt down because of a packet storm", the real nuclear-safety-keep-the-core-from-melting systems (reactor protection systems, emergency core cooling systems) don't even have any computerized control -- they all rely on simple electrical relays first designed and manufactured 40 years ago to trigger automatic action.
I usually hear them called packet storms, but they happen and "storm" is usually somewhere in the description. In fact, we were just troubleshooting exactly that at my work last week and the network admin used the exact phrase "packet storm".
Great news, guys. This is going to be a non-issue. People are freaking out because a digital device is involved, and freaking out because a nuclear power plant was involved, but I do industrial control system and DCS design for a living, and I'll tell you right now, that you simply can't access control networks from the outside. There are seperate, often redundant networks, and even then, depending on the way the plant was designed, we're talking modbus plus or something that PCs don't normally access.
It's been a long time.
It looks like it was a modbus plus network. We're talking a proprietary physical layer on up, specifically designed for PLCs to communicate with one another.
If there was a communications problem and a PLC blinks out of existence on a mission critical system, it's only the safe thing to fail the entire system to prevent damage to people, the environment, and equipment.
It's been a long time.
Having to work with a seperated network myself, I'd have to agree about doing as little as possible with it.
In my case it's for two reasons. One, the disconnected network is considered the critical one, and is far more locked down than the one connected to the internet. Second, the one connected to the internet is the one used 99% of the time.
Anytime we touch a system there's a chance we'll screw it up/break it. Our treatment of the isolated network is pretty much 'don't fix what isn't broken'. It wasn't too long ago that we had a P200 still acting as a PDC on it. It worked, we didn't touch it.
I don't read AC A human right
>I've worked in IT a while now & have never heard of a "data storm".
I used to work as embedded developer, and we used that term.
It was used in embedded communications when one or several devices went bonkers and flooded common bus.
Bit like packet storm, but without IP or other packet protocol, so it was called data storm.
It stands to reason that in nuclear plant there are a lot of old fogeys, so company jargon might be bit outdated and odd sounding to outsider.
After some R&D and building some prototypes of promising new designs I'd be right with you - but our current best bets are things out of South Africa (pebble bed) and India (accelerated thorium) done on very small buidgets with very small teams and they need more work. The mainstream is just chasing taxpayer supplied pork. If they were after more than a handout they would be putting in some effort - instead they spend orders of magnitaude in PR, advertising and outright bribes than R&D.
As for costs - you can't just conveniently ignore capital costs. If you could hydro, wind, solar etc would win every time even in those places where it would be a stupid idea or where the capital costs are far too large for the return. Nuclear power is a possiblity in those places that have the infrastucture of a weapons program but everywhere else you would have to build up an entire industry from scratch. Iran is the best example currently where that is taking place and it has cost them a fortune to do so - hence few people think it is for purely civilian purposes there. In South Africa it was possible to take people from the weapons program to develop pebble bed. It is also far too big an investment for private enterprise - hence no new plants getting built while governments had cold feet on the issue and the "new generation" designs from companies like Westinghouse are just tweaked 1950s designs painted green.
1. It wasn't my assertion -- I did not make the original post about Browns Ferry. Try reading next time!
2. I just happened to hear an article on PBS about Browns Ferry the day of this post.
3. As I mentioned before, you can confirm it using Google. Here, I'll even show you how to find it using google
4. What is it about "/. is not an encyclopedia" that you don't understand?
There may be many case where one might claim that a post on /. is pure BS, but in the case of the great-grandparent post, the facts are easily confirmed.
The real "Libtards" are the Libertarians!
The plant I'm working on the design of at the moment will have a VPN connection so that we can monitor it's performance from abroad. Running private cables over 7000 miles would not be feasible.
A "data storm" can be caused by lots of things, even an unstable driver causing a NIC to spew garbage packets. Or an application that hits a bug and begins spewing to the network. Or a failure of Spanning Tree causing network loops to arise (which can really mess up an Ethernet).
The wierdest I ever saw was a situation at a school where the entire network (built around high-end Cisco switches) crashed hard. It took 3 hours of troubleshooting and disconnecting various segments to finally pin down the cause. It was a little mini-switch that some teacher attached to the LAN that somehow had a meltdown and began spewing "valid" Ethernet packets with all kinds of random garbage source and destination MAC addresses, random payload, and valid checksums. No hosts were attached to the mini switch, so it had to be something in its microcontroller going haywire. This cause every switch to go nuts trying to maintain its forwarding tables ("show cpu" was 100% utilization) and resulted in no traffic going anywhere. It even crossed VLAN boundaries since all the switches had "trunk" ports using tagged VLANS, so the garbage packets still made it through the entire LAN.
These things happen sometimes. Network gear is generally pretty robust, but can still fail in wierd ways.
Might I recommend you to RTFA?
The "data storm" appears to have been on a internal network (not seemingly connected to anything apart from other internal networks), where a data acquisition and control device barfed on some bad data and started to spew garbage onto the network. Inadequate data validation combined with inappropriate or ineffective error handling. Software fault.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"