Feds Investigating Water Utility Pump Failure As Possible Cyberattack

SpuriousLogic writes with this quote from CNN: "Federal officials confirmed they are investigating whether a cyber attack may have been responsible for the failure of a water pump at a public water district in Illinois last week. But they cautioned that no conclusions had been reached, and they disputed one cyber security expert's statements that other utilities are vulnerable to a similar attack. Joe Weiss, a noted cyber security expert, disclosed the possible cyber attack on his blog Thursday. Weiss said he had obtained a state government report, dated Nov. 10 and titled 'Public Water District Cyber Intrusion,' which gave details of the alleged cyber attack culminating in the 'burn out of a water pump.' According to Weiss, the report says water district workers noted 'glitches' in the systems for about two months. On Nov. 8, a water district employee noticed problems with the industrial control systems, and a computer repair company checked logs and determined that the computer had been hacked. Weiss said the report says the cyber attacker hacked into the water utility using passwords stolen from a control system vendor and that he had stolen other user names and passwords."

Comments

Darned commies

[Anonymous Coward]

Tryin to interfere with America's precious bodily fluids

SCADA vulns

[sl4shd0rk]

SCADA systems were sold en masse under the presumption that they were "secure" because they were not connected to public networks. It will be interesting to see which entities did, or did not, follow their policies. Stuxnet was a USB infection but it was still able to route over the internet to phone home. I'm going to bet that a lot of SCADA networks are implemented to allow egress packets. It will be interesting to see how many SCADA systems are actually "isolated".

Re:SCADA vulns

[Anonymous Coward]

I worked for a Utility in the early 2000's..I was on the post-9/11 security team that had to investigate and close loopholes for that utility. Many sites had interconnected the SCADA systems with the corporate network for GIS information. We were hard pressed to find adequate solutions that would meet the requirements that the federal government set at that time; as the engineering staff didn't want to give up the real-time GIS information they got from the SCADA systems.

Re:SCADA vulns

[mlts]

The ironic thing, there is a secure way to get GIS info out, although it isn't the fastest method. I did this on a lab network that needed to be air-gapped from everything else:

1: Build two machines, each has a NIC, and both have a serial card ($60 from NewEgg for a PCIe to Serial.)

2: Build a custom cable with the RX wires cut, so data only goes one way. I did this so an intruder has no chance of being able to send anything to the box on the secure network, much less create a SLIP or PPP connection.

3: Configure one box on the secure network. It scrapes input from the embedded boxes, formats it (so stuff from one device is marked as such so it can be told apart from a different one and to help keep both machines in sync), then pushes it over the serial device.

4: The other box is configured to passively take what comes over the serial port, un-format it (so stuff from one device goes to one web server, stuff from another device gets E-mailed to an admin, alerts get set if something is wrong, etc.)

The result of this is being able to get reports from the embedded boxes on a real-time basis, but without any way of a remote intruder ever getting on the network. Since the physical serial cable cannot send any data to the machine on the embedded network, it would take a physical attack in order to compromise the boxes.

I'm sure there are faster ways to get data across a cable one-way, but this was ideal, as the data obtained was not much, and the latency of the multiple steps to shoot it to a box, stuff it across a serial pipe, then on the other side, send it where it needs to go was just fine.

AWESOME

[WindBourne]

That is possibly just a kid playing, however, it could be somebody learning. The nice thing is that it has now been detected. Perhaps it is time to push not just security, but to insist that the parts be western or better yet, American made. Seriously, this is infrastructure that should be local to friendly nations. China is hard at work to make sure that they have the ability to import zero food as well as all of their equipment is from local sources. In doing that, they claim national security. Makes sense. But we should be doing the same.

Perhaps Not All Remote Management Worth The Risk

[stating_the_obvious]

Perhaps it's time to start we stop believing that everything in the world needs to be connected to external networks.

In the battle of the sword and the shield, the sword eventually wins, but it takes a hell of a lot longer when the sword and shield are separated by the moat and a thick stone wall...

Could be something incredibly simple

[slewfo0t]

As a controls engineer, I program these type of systems all the time. A simple incorrect setting for when the pumps turn on and off (Lead,Lag) could cause this type of problem. It could literally be a new operator that fat fingered a parameter in the SCADA system. To hack these systems requires specific knowledge of exactly what kind of control architecture is in place at the facility and then having the appropriate software to gain access to the control system. Not that this type of hack cannot be done, but it does require specific knowledge. This really sounds like operator error to me.

Re:Could be something incredibly simple

[ColdWetDog]

This really sounds like operator error to me.

From TFA:

But in its statement, the DHS said the water system was located in Springfield, Illinois.

Springfield....

Operator error....

Something in the back of my mind....

Re:Could be something incredibly simple

[Anonymous Coward]

Sort of. To program or configure the specific SCADA system requires specific knowledge of the device, installation architecture, firmware, and version supplied by the system operating manual. Until you get to the S part of SCADA and it all goes into some sort of aggregation platform with a big old GUI on a windows 2000 or windows XP box hooked into a cable modem.

Well, to program them correctly requires that knowledge.

These manuals are often trade secrets for the manufacturer, but are 'openly' passed around by maintenance technicians and field installers, and probably controls engineers such as yourself--although I never had the pleasure to work with one.

Depending upon the organization, such manuals are often shipped to other third party contractors with a "legitimate need" as determined by an engineer or manager.

When you tell them you have a corporate filter on PDFs, they will send to a personal email address if they would send it to start with. If they won't send it directly to you, their client will find a way to get their hands on it and forward it to you.

These manuals contain relatively complex documentation--including ports, encoding types, bit masks, register sizes and addresses that may be remotely configured by a couple of pretty common protocols which tend to be "extended" by the vendor in odd ways.

Sure, every bigwig in the industry has their own special program for everything that talks some proprietary clusterfuck. But mostly, they all have legacy support and some sort of shitty standard that will do basics.

Admittedly, any piece of hardware may implement complicated control processes specific to the device at hand, but all of which (that I've seen) generally fall into about three different "protocol families" for control purposes once you're down to a sensor or switch. Maybe you can't calibrate the device over your basic serial port, but you can throw a relay with it.

All of which I once wrote software for to control via plaintext text message at the demands of a former employer. Who insisted on static vendor passwords, and no encryption or even authorized whitelists to make our controllers easier and faster to install for subcontractors. Plug and Play. Or Pray. Or Plug and hacker prey. Whatever.

Now, you can say it's operator error to use that device. But the bottom line is even in your wealthy industries that do readonly monitoring over encrypted VPN--sooner or later somebody insists on remote control in order to cut maintenance costs. The moment that happens, they're hooked up to hardware that might be 25 years old. And then they're gonna hire somebody with a cheap solution to plug into it.

No Reason

[sycodon]

I can think of no reason facilities such as this should be accessible via a public network. You should have to be physically present to access these control systems.

Re:No Reason

[Sarten-X]

Unless something goes catastrophically wrong, such as a fire in the control building, in which case the pumps (which must still operate) will need to be controlled remotely. Even during routine operation, the control system is likely connected to a monitoring network of some kind, to make sure things run smoothly.

That means either wiring up a physically-isolated network (and constantly checking it for unauthorized alterations), which is ridiculously expensive, or connecting to the public network physically, and relying on software to keep it secure. Given that this system is probably a few decades old, and probably installed by the lowest bidder, you can make some reasonably-depressing assumptions about how secure that software is.

Re:Perhaps Not All Remote Management Worth The Ris

[Mr. Freeman]

Perhaps it's time that people realize that a lot of things do need to be connected to external networks and that "air gap them" is simply a cop out response equivalent to saying "use a typewriter".

Yes, some things should be air-gaped, nuclear gas centrifuges come to mind. However, many industrial control systems need to report information over the internet. Remote pumping stations, unmanned power distribution centers, etc. Having a lot of data is not simply a convenience. This data allows engineers to troubleshoot failures, predict future failures, and adjust systems for optimum efficiency.

What's really necessary is for some kind of device that will communicate the data to remote places, but refuse to pass any messages from the outside onto the control system. I don't know how difficult this is, but it's certainly harder than "air gap it". On the other hand, this solution actually addresses the problem.

Real Cause of Failure

[fsckmnky]

Connecting your water pumps to the public internet.

Der der der.

Re:Perhaps Not All Remote Management Worth The Ris

[idontgno]

What's really necessary is for some kind of device that will communicate the data to remote places, but refuse to pass any messages from the outside onto the control system. I don't know how difficult this is, but it's certainly harder than "air gap it". On the other hand, this solution actually addresses the problem.

So, what you're saying is, if a utility is too cheap to lay in dedicated network assets and buy their own blacknet (which is not hard to do if you want to), it's ok to just connect the the Internet?

That said, the thing you're looking for is called a unidirectional network. Back in my military network operations days, the colloquial name was "data diode". Data goes one way but nothing (no data, no handshakes, no signaling at all) goes the other way. In that environment, they were used to promote data from a lower-level security environment (say, Secret-only) to a higher-level one with no risk of leak-back.

Yeah. They exist. They're considerably lower-bandwidth than your average gigabit Ethernet switch, but if you're just talking SCADA telemetry, they should suffice.

I call bullshit.

[Lumpy]

I have worked with SCADA and water filtration plant pumps, big ass pumps, like 650hp pumps that run on 7200volts.

You cant set it to "burn out". you can adjust the speed of the pump from 10% to 100% the only way to kill a pump is to drop power to it without dropping power to it's valve so it will not close. wait for the pump to start spinning backwards from the water running back downhill through the pump and then slamming the power back on at 100% after the pump was free wheeling in reverse at full speed.

Then they don't burn out, they freaking explode.

This happened when we lost power plant wide and a hydraulic failure kept the valve from auto closing. (not electronic, it's a mechanical/hydraulic thing, a blockage in the pressure line)

Unless the plant was designed by a utter moron and made it so a programming error could blow up parts of the plant.

'Been in the water/SCADA industry for 10 years...

[kackle]

I've been in the water SCADA industry for 10 years. What I'm seeing lately are water operators, IT people, and system integrators who are overzealous when it comes to connectivity and all the "neat" things that can be done remotely via technology. It's the standard human foible when it comes to technology, writ dangerous: they consider what can be done versus whether it should be. The water industry isn't that exciting, so when flashy tech. comes along, and the taxpayer is footing the bill, I can see where they say "Yes!" And who is the salesperson to refuse this order?

I'm all for automation, and crying out when a system is in trouble. But I haven't yet seen where humanized remote control is critical. Hackers aside, it's probably better if it's not.

New information

[mcgrew]

The local TV news is on, and they just said that it was Curran, a tiny town five or ten miles from Springfield. They're concerned that the system might have been hacked because the company that designed the system discovered evidence of a breach of sensitive data... passwords, maybe? They did say it was gigabytes of data.

Re:'Been in the water/SCADA industry for 10 years.

[Animats]

What I'm seeing lately are water operators, IT people, and system integrators who are overzealous when it comes to connectivity and all the "neat" things that can be done remotely via technology.

Yes. Read "Access Your Embedded Controller with Ease through a Web Server", from Texas Instruments, which ought to know better. "The designer should also make it as easy as possible to change the settings on a piece of equipment, reconfigure its operation, or fine-tune the system. The more intuitive and explicit that activity is, the more likely the result will be what the operator desires. Losing the instruction manual can seriously impair the user's operation of many systems."

What that paper describes is a family of embedded controllers with a web server in each controller and no security. What's wrong with this picture?