Slashdot Mirror


Feds Investigating Water Utility Pump Failure As Possible Cyberattack

SpuriousLogic writes with this quote from CNN: "Federal officials confirmed they are investigating whether a cyber attack may have been responsible for the failure of a water pump at a public water district in Illinois last week. But they cautioned that no conclusions had been reached, and they disputed one cyber security expert's statements that other utilities are vulnerable to a similar attack. Joe Weiss, a noted cyber security expert, disclosed the possible cyber attack on his blog Thursday. Weiss said he had obtained a state government report, dated Nov. 10 and titled 'Public Water District Cyber Intrusion,' which gave details of the alleged cyber attack culminating in the 'burn out of a water pump.' According to Weiss, the report says water district workers noted 'glitches' in the systems for about two months. On Nov. 8, a water district employee noticed problems with the industrial control systems, and a computer repair company checked logs and determined that the computer had been hacked. Weiss said the report says the cyber attacker hacked into the water utility using passwords stolen from a control system vendor and that he had stolen other user names and passwords."

30 of 136 comments (clear)

  1. Darned commies by Anonymous Coward · · Score: 5, Funny

    Tryin to interfere with America's precious bodily fluids

    1. Re:Darned commies by bobcat7677 · · Score: 2

      That's why I only drink distilled water and pure grain alcohol!

  2. SCADA vulns by sl4shd0rk · · Score: 5, Interesting

    SCADA systems were sold en masse under the presumption that they were "secure" because they were not connected to public networks. It will be interesting to see which entities did, or did not, follow their policies. Stuxnet was a USB infection but it was still able to route over the internet to phone home. I'm going to bet that a lot of SCADA networks are implemented to allow egress packets. It will be interesting to see how many SCADA systems are actually "isolated".

    --
    Join the Slashcott! Feb 10 thru Feb 17!
    1. Re:SCADA vulns by Anonymous Coward · · Score: 3, Interesting

      I worked for a Utility in the early 2000's..I was on the post-9/11 security team that had to investigate and close loopholes for that utility. Many sites had interconnected the SCADA systems with the corporate network for GIS information. We were hard pressed to find adequate solutions that would meet the requirements that the federal government set at that time; as the engineering staff didn't want to give up the real-time GIS information they got from the SCADA systems.

    2. Re:SCADA vulns by mlts · · Score: 5, Interesting

      The ironic thing, there is a secure way to get GIS info out, although it isn't the fastest method. I did this on a lab network that needed to be air-gapped from everything else:

      1: Build two machines, each has a NIC, and both have a serial card ($60 from NewEgg for a PCIe to Serial.)

      2: Build a custom cable with the RX wires cut, so data only goes one way. I did this so an intruder has no chance of being able to send anything to the box on the secure network, much less create a SLIP or PPP connection.

      3: Configure one box on the secure network. It scrapes input from the embedded boxes, formats it (so stuff from one device is marked as such so it can be told apart from a different one and to help keep both machines in sync), then pushes it over the serial device.

      4: The other box is configured to passively take what comes over the serial port, un-format it (so stuff from one device goes to one web server, stuff from another device gets E-mailed to an admin, alerts get set if something is wrong, etc.)

      The result of this is being able to get reports from the embedded boxes on a real-time basis, but without any way of a remote intruder ever getting on the network. Since the physical serial cable cannot send any data to the machine on the embedded network, it would take a physical attack in order to compromise the boxes.

      I'm sure there are faster ways to get data across a cable one-way, but this was ideal, as the data obtained was not much, and the latency of the multiple steps to shoot it to a box, stuff it across a serial pipe, then on the other side, send it where it needs to go was just fine.

  3. AWESOME by WindBourne · · Score: 4, Interesting

    That is possibly just a kid playing, however, it could be somebody learning. The nice thing is that it has now been detected. Perhaps it is time to push not just security, but to insist that the parts be western or better yet, American made. Seriously, this is infrastructure that should be local to friendly nations. China is hard at work to make sure that they have the ability to import zero food as well as all of their equipment is from local sources. In doing that, they claim national security. Makes sense. But we should be doing the same.

    --
    I prefer the "u" in honour as it seems to be missing these days.
  4. having solved all other by nimbius · · Score: 2

    major federal crimes such as the collapse of the united states economy at the hands of wall-street, human trafficking between south america and north america, net neutrality compliance that is largely being ignored by major carriers, civil rights abuses in united states prisons, and protestor police brutality in major metropolitan cities, federal officials target their laser-like scrutiny upon the teeming cesspool of violent crime and evildoings that is Springfield Illinois. their objective? prove a small and unsubstantial water pump in a city of 116,000 people has been nefariously compromised and destroyed by cyber (attackers/hackers/criminals) from (china/iran/north korea/syria) in order to deprive american citizens of their shitty and unaccountably safe drinking water for an evening while the district manager oversees a few dozen pipefitters and welders as they replace a pump on a blustery november weekend.

    --
    Good people go to bed earlier.
  5. Perhaps Not All Remote Management Worth The Risk by stating_the_obvious · · Score: 5, Insightful

    Perhaps it's time to start we stop believing that everything in the world needs to be connected to external networks.

    In the battle of the sword and the shield, the sword eventually wins, but it takes a hell of a lot longer when the sword and shield are separated by the moat and a thick stone wall...

  6. Makes sense.... by TheCarp · · Score: 2

    Lets face it, when they are putting out advisories actually advertising that one of the FBI's "Most Wanted" is some dude who blew up a package at a building, in the middle of the night, injuring noone, just so he could make some statement about "Animal Liberation".... you really have to wonder what the hell these people actually do for a living anyway.

    I mean.... if that dude is one of the top 10 threats out there.... then I think we can all relax.

    Quick, somebody find a tenuous link to terrorism so we can look relevant!

    --
    "I opened my eyes, and everything went dark again"
  7. Could be something incredibly simple by slewfo0t · · Score: 5, Interesting

    As a controls engineer, I program these type of systems all the time. A simple incorrect setting for when the pumps turn on and off (Lead,Lag) could cause this type of problem. It could literally be a new operator that fat fingered a parameter in the SCADA system. To hack these systems requires specific knowledge of exactly what kind of control architecture is in place at the facility and then having the appropriate software to gain access to the control system. Not that this type of hack cannot be done, but it does require specific knowledge. This really sounds like operator error to me.

    1. Re:Could be something incredibly simple by ColdWetDog · · Score: 3, Funny

      This really sounds like operator error to me.

      From TFA:

      But in its statement, the DHS said the water system was located in Springfield, Illinois.

      Springfield....

      Operator error....

      Something in the back of my mind....

      --
      Faster! Faster! Faster would be better!
    2. Re:Could be something incredibly simple by Anonymous Coward · · Score: 4, Informative

      Sort of. To program or configure the specific SCADA system requires specific knowledge of the device, installation architecture, firmware, and version supplied by the system operating manual. Until you get to the S part of SCADA and it all goes into some sort of aggregation platform with a big old GUI on a windows 2000 or windows XP box hooked into a cable modem.

      Well, to program them correctly requires that knowledge.

      These manuals are often trade secrets for the manufacturer, but are 'openly' passed around by maintenance technicians and field installers, and probably controls engineers such as yourself--although I never had the pleasure to work with one.

      Depending upon the organization, such manuals are often shipped to other third party contractors with a "legitimate need" as determined by an engineer or manager.

      When you tell them you have a corporate filter on PDFs, they will send to a personal email address if they would send it to start with. If they won't send it directly to you, their client will find a way to get their hands on it and forward it to you.

      These manuals contain relatively complex documentation--including ports, encoding types, bit masks, register sizes and addresses that may be remotely configured by a couple of pretty common protocols which tend to be "extended" by the vendor in odd ways.

      Sure, every bigwig in the industry has their own special program for everything that talks some proprietary clusterfuck. But mostly, they all have legacy support and some sort of shitty standard that will do basics.

      Admittedly, any piece of hardware may implement complicated control processes specific to the device at hand, but all of which (that I've seen) generally fall into about three different "protocol families" for control purposes once you're down to a sensor or switch. Maybe you can't calibrate the device over your basic serial port, but you can throw a relay with it.

      All of which I once wrote software for to control via plaintext text message at the demands of a former employer. Who insisted on static vendor passwords, and no encryption or even authorized whitelists to make our controllers easier and faster to install for subcontractors. Plug and Play. Or Pray. Or Plug and hacker prey. Whatever.

      Now, you can say it's operator error to use that device. But the bottom line is even in your wealthy industries that do readonly monitoring over encrypted VPN--sooner or later somebody insists on remote control in order to cut maintenance costs. The moment that happens, they're hooked up to hardware that might be 25 years old. And then they're gonna hire somebody with a cheap solution to plug into it.

    3. Re:Could be something incredibly simple by onepoint · · Score: 2

      that's the same shit they said back in 1985ish when those hackers were moving satellites around. nothing is unhackable, sometimes it just takes more time to figure out.

      --
      if you see me, smile and say hello.
  8. Since when is this hacking? by rudy_wayne · · Score: 2

    the cyber attacker hacked into the water utility using passwords stolen from a control system vendor

    WTF?

    It's not hacking if you know the password.

    1. Re:Since when is this hacking? by Kikuchi · · Score: 2

      It can't be anything else than hacking, not when the word cyber appears seven times in the summary.

      --
      There's no scientific consensus that life is important.
  9. The moral of the story is... by Gyorg_Lavode · · Score: 2

    ...a hacked pump at a water station DOES NOT DESTROY THE COUNTRY.

    --
    I do security
  10. No Reason by sycodon · · Score: 5, Insightful

    I can think of no reason facilities such as this should be accessible via a public network. You should have to be physically present to access these control systems.

    --
    When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
    1. Re:No Reason by Sarten-X · · Score: 3, Informative

      Unless something goes catastrophically wrong, such as a fire in the control building, in which case the pumps (which must still operate) will need to be controlled remotely. Even during routine operation, the control system is likely connected to a monitoring network of some kind, to make sure things run smoothly.

      That means either wiring up a physically-isolated network (and constantly checking it for unauthorized alterations), which is ridiculously expensive, or connecting to the public network physically, and relying on software to keep it secure. Given that this system is probably a few decades old, and probably installed by the lowest bidder, you can make some reasonably-depressing assumptions about how secure that software is.

      --
      You do not have a moral or legal right to do absolutely anything you want.
    2. Re:No Reason by mcgrew · · Score: 2

      An update: I just discovered that it's my own city, Illinois' capital, Cartoon City. From the State Journal-Register:

      CWLP denies reports it was victim of cyber attack

      By DEANA STROISCH (deana.stroisch@sj-r.com)
      The State Journal-Register
      Posted Nov 18, 2011 @ 11:05 AM
      Last update Nov 18, 2011 @ 11:31 AM

      City Water, Light and Power officials are denying reports that the utility was a victim of a cyber attack that may have been responsible for the failure of a water pump.

      âoeCWLP has not had any breach of its Water or Electric Department Supervisory Control and Data Acquisition (SCADA) systems,â the utility said in a statement issued this morning.

      SCADA is the computer control network that operates various systems at the utility.

      In a story that has since been picked up by CNN, Joe Weiss, a cyber security expert, says he obtained a state government report dated Nov. 10, which allegedly gave details of a computer hacking that led to the âoeburn out of a water pump.â

      The Department of Homeland Security identified the water system as being located in Springfield, Ill.

      "DHS and the FBI are gathering facts surrounding the report of a water pump failure in Springfield Illinois, said Peter Boogaard, DHS spokesman. âoeAt this time there is no credible corroborated data that indicates a risk to critical infrastructure entities or a threat to public safety.

      âoeIf DHS ICS-CERT identifies any information about possible impacts to additional entities, it will disseminate timely mitigation information as it becomes available."

      Amber Sabin, CWLPâ(TM)s public information officer, said there have not been any water pump failures of any kind in the last month.

      Links to the reports:

      http://www.cnn.com/2011/11/18/us/cyber-attack-investigation/index.html

      http://www.wired.com/threatlevel/2011/11/hackers-destroy-water-pump/?utm_source=co2hog

      Copyright 2011 The State Journal-Register. Some rights reserved

       

    3. Re:No Reason by BanjoBob · · Score: 2

      This is the way it used to be. The only true security is to be isolated. When I worked on secret stuff, anything that went into the facility stayed in the facility and, there was absolutely no connection to the outside world -- None.

      Also, these wireless network electric/water/gas meters are easily hackable. Why not just but a big wide open door into their data center?

      It's hard to take these utility companies seriously when they talk up security. If they want it secure -- secure it.

      --
      Banjo - The more I know about Windoze, the more I love *nix
  11. Re:Perhaps Not All Remote Management Worth The Ris by Mr.+Freeman · · Score: 4, Interesting

    Perhaps it's time that people realize that a lot of things do need to be connected to external networks and that "air gap them" is simply a cop out response equivalent to saying "use a typewriter".

    Yes, some things should be air-gaped, nuclear gas centrifuges come to mind. However, many industrial control systems need to report information over the internet. Remote pumping stations, unmanned power distribution centers, etc. Having a lot of data is not simply a convenience. This data allows engineers to troubleshoot failures, predict future failures, and adjust systems for optimum efficiency.

    What's really necessary is for some kind of device that will communicate the data to remote places, but refuse to pass any messages from the outside onto the control system. I don't know how difficult this is, but it's certainly harder than "air gap it". On the other hand, this solution actually addresses the problem.

    --
    -1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
  12. Real Cause of Failure by fsckmnky · · Score: 3, Insightful

    Connecting your water pumps to the public internet.

    Der der der.

  13. Re:Perhaps Not All Remote Management Worth The Ris by idontgno · · Score: 3, Informative

    What's really necessary is for some kind of device that will communicate the data to remote places, but refuse to pass any messages from the outside onto the control system. I don't know how difficult this is, but it's certainly harder than "air gap it". On the other hand, this solution actually addresses the problem.

    So, what you're saying is, if a utility is too cheap to lay in dedicated network assets and buy their own blacknet (which is not hard to do if you want to), it's ok to just connect the the Internet?

    That said, the thing you're looking for is called a unidirectional network. Back in my military network operations days, the colloquial name was "data diode". Data goes one way but nothing (no data, no handshakes, no signaling at all) goes the other way. In that environment, they were used to promote data from a lower-level security environment (say, Secret-only) to a higher-level one with no risk of leak-back.

    Yeah. They exist. They're considerably lower-bandwidth than your average gigabit Ethernet switch, but if you're just talking SCADA telemetry, they should suffice.

    --
    Welcome to the Panopticon. Used to be a prison, now it's your home.
  14. Re:Perhaps Not All Remote Management Worth The Ris by Obfuscant · · Score: 2

    However, many industrial control systems need to report information over the internet.

    Maybe over AN internet, but not over THE Internet. "Report information" is not the same as "allow incoming control or information."

    This can be as simple as a Lantronix XPort (or equivalent) tied to a serial port TX line on a secure machine, allowing telnet connections to read the serial data coming out but not send anything back. Or any terminal server with the RX lines cut.

    What you need to be careful of in the planning of this system is that the information coming out of the secure system isn't being fed back into the system as the result of an external control. I.e., "Water level low in reactor 5" as outbound information cannot cause an "increase water flow to reactor 5" command from outside.

  15. I call bullshit. by Lumpy · · Score: 4, Informative

    I have worked with SCADA and water filtration plant pumps, big ass pumps, like 650hp pumps that run on 7200volts.

    You cant set it to "burn out". you can adjust the speed of the pump from 10% to 100% the only way to kill a pump is to drop power to it without dropping power to it's valve so it will not close. wait for the pump to start spinning backwards from the water running back downhill through the pump and then slamming the power back on at 100% after the pump was free wheeling in reverse at full speed.

    Then they don't burn out, they freaking explode.

    This happened when we lost power plant wide and a hydraulic failure kept the valve from auto closing. (not electronic, it's a mechanical/hydraulic thing, a blockage in the pressure line)

    Unless the plant was designed by a utter moron and made it so a programming error could blow up parts of the plant.

    --
    Do not look at laser with remaining good eye.
  16. 'Been in the water/SCADA industry for 10 years... by kackle · · Score: 5, Insightful

    I've been in the water SCADA industry for 10 years. What I'm seeing lately are water operators, IT people, and system integrators who are overzealous when it comes to connectivity and all the "neat" things that can be done remotely via technology. It's the standard human foible when it comes to technology, writ dangerous: they consider what can be done versus whether it should be. The water industry isn't that exciting, so when flashy tech. comes along, and the taxpayer is footing the bill, I can see where they say "Yes!" And who is the salesperson to refuse this order?

    I'm all for automation, and crying out when a system is in trouble. But I haven't yet seen where humanized remote control is critical. Hackers aside, it's probably better if it's not.

  17. default passwords? by blivit42 · · Score: 2
    From TFA (and the summary):

    "Weiss said the report says the cyber attacker hacked into the water utility using passwords stolen from a control system vendor and that he had stolen other user names and passwords."

    How likely is it that a control system vendor would have the usernames and passwords of their client, used in the actual production system? Maybe they actually do, as part of some sort of remote support agreement, but if this is the case, that's already a bad security practice.

    It seems more likely to me that the vendor has a list of default usernames and passwords, and THIS is what was obtained. Perhaps what Weiss *really* meant to say would be be something like: "Someone got ahold of the default usernames and passwords that our vendor uses. Since we never changed them from the default values, it's our own damn fault."

    After seeing SO many stories like this, it's usually a case of not changing default passwords. Given that Weiss's statement *could* be read as I have read it, this seems the most likely scenario to me. I'm going to write this one up as stupidly bad security policies until I have sufficient evidence contradicting this assumption.

  18. New information by mcgrew · · Score: 3, Informative

    The local TV news is on, and they just said that it was Curran, a tiny town five or ten miles from Springfield. They're concerned that the system might have been hacked because the company that designed the system discovered evidence of a breach of sensitive data... passwords, maybe? They did say it was gigabytes of data.

  19. Re:'Been in the water/SCADA industry for 10 years. by Animats · · Score: 3, Interesting

    What I'm seeing lately are water operators, IT people, and system integrators who are overzealous when it comes to connectivity and all the "neat" things that can be done remotely via technology.

    Yes. Read "Access Your Embedded Controller with Ease through a Web Server", from Texas Instruments, which ought to know better. "The designer should also make it as easy as possible to change the settings on a piece of equipment, reconfigure its operation, or fine-tune the system. The more intuitive and explicit that activity is, the more likely the result will be what the operator desires. Losing the instruction manual can seriously impair the user's operation of many systems."

    What that paper describes is a family of embedded controllers with a web server in each controller and no security. What's wrong with this picture?

  20. Screenshots by generic · · Score: 2
    --
    Microsoft aggravates my tourettes syndrome.