Slashdot Mirror
Feds Investigating Water Utility Pump Failure As Possible Cyberattack
SpuriousLogic writes with this quote from CNN:
"Federal officials confirmed they are investigating whether a cyber attack may have been responsible for the failure of a water pump at a public water district in Illinois last week. But they cautioned that no conclusions had been reached, and they disputed one cyber security expert's statements that other utilities are vulnerable to a similar attack. Joe Weiss, a noted cyber security expert, disclosed the possible cyber attack on his blog Thursday. Weiss said he had obtained a state government report, dated Nov. 10 and titled 'Public Water District Cyber Intrusion,' which gave details of the alleged cyber attack culminating in the 'burn out of a water pump.' According to Weiss, the report says water district workers noted 'glitches' in the systems for about two months. On Nov. 8, a water district employee noticed problems with the industrial control systems, and a computer repair company checked logs and determined that the computer had been hacked. Weiss said the report says the cyber attacker hacked into the water utility using passwords stolen from a control system vendor and that he had stolen other user names and passwords."
Tryin to interfere with America's precious bodily fluids
...thinks innocuous event is a cyber security attack. News at 11.
SCADA systems were sold en masse under the presumption that they were "secure" because they were not connected to public networks. It will be interesting to see which entities did, or did not, follow their policies. Stuxnet was a USB infection but it was still able to route over the internet to phone home. I'm going to bet that a lot of SCADA networks are implemented to allow egress packets. It will be interesting to see how many SCADA systems are actually "isolated".
Join the Slashcott! Feb 10 thru Feb 17!
That is possibly just a kid playing, however, it could be somebody learning. The nice thing is that it has now been detected. Perhaps it is time to push not just security, but to insist that the parts be western or better yet, American made. Seriously, this is infrastructure that should be local to friendly nations. China is hard at work to make sure that they have the ability to import zero food as well as all of their equipment is from local sources. In doing that, they claim national security. Makes sense. But we should be doing the same.
I prefer the "u" in honour as it seems to be missing these days.
major federal crimes such as the collapse of the united states economy at the hands of wall-street, human trafficking between south america and north america, net neutrality compliance that is largely being ignored by major carriers, civil rights abuses in united states prisons, and protestor police brutality in major metropolitan cities, federal officials target their laser-like scrutiny upon the teeming cesspool of violent crime and evildoings that is Springfield Illinois. their objective? prove a small and unsubstantial water pump in a city of 116,000 people has been nefariously compromised and destroyed by cyber (attackers/hackers/criminals) from (china/iran/north korea/syria) in order to deprive american citizens of their shitty and unaccountably safe drinking water for an evening while the district manager oversees a few dozen pipefitters and welders as they replace a pump on a blustery november weekend.
Good people go to bed earlier.
Maybe one of the local citizens was just upset about low water pressure and decided to take matters into their own cyber-hands?
It would be interesting if the system hacked was similar to the ones used for the hacked Iranian centerfuges, as both attacks involve spinning a motor too fast.
Live Free or Die Hard Reference. I'm the good guy here. I told them this could happen if they didn't prepare. Did I get a "Thank you"? They have been warned now we are all going to pay for their ignorance!
Perhaps it's time to start we stop believing that everything in the world needs to be connected to external networks.
In the battle of the sword and the shield, the sword eventually wins, but it takes a hell of a lot longer when the sword and shield are separated by the moat and a thick stone wall...
gallons, towing it to China in huge bladders and hacking our cities' pumps?
According to Weiss, the report says water district workers noted 'glitches' in the systems for about two months. [...]
and a computer repair company checked logs and determined that the computer had been hacked.
It's not really a stretch to say this event was the result of lax maintenance.
"Oh...the water pump runs on 480 VAC? I thought it ran on 600! That explains the smoke..."
guess we need to setup the great firewall of USA
Lets face it, when they are putting out advisories actually advertising that one of the FBI's "Most Wanted" is some dude who blew up a package at a building, in the middle of the night, injuring noone, just so he could make some statement about "Animal Liberation".... you really have to wonder what the hell these people actually do for a living anyway.
I mean.... if that dude is one of the top 10 threats out there.... then I think we can all relax.
Quick, somebody find a tenuous link to terrorism so we can look relevant!
"I opened my eyes, and everything went dark again"
As a controls engineer, I program these type of systems all the time. A simple incorrect setting for when the pumps turn on and off (Lead,Lag) could cause this type of problem. It could literally be a new operator that fat fingered a parameter in the SCADA system. To hack these systems requires specific knowledge of exactly what kind of control architecture is in place at the facility and then having the appropriate software to gain access to the control system. Not that this type of hack cannot be done, but it does require specific knowledge. This really sounds like operator error to me.
WTF?
It's not hacking if you know the password.
Perhaps it's time to start we stop believing that everything in the world needs to be connected to external networks.
Perhaps it's time to stop believing that everything in the world that goes wrong is due to a 'cyberattack'.
Faster! Faster! Faster would be better!
...if a 'security researcher' (aka whitehat, ethical hacker, etc.) had informed them of the potential for this occurring before, they could be sued into oblivion.
And the real threat - if a malicious actor did indeed do this - would walk away, laughing at how idiotic the whole scenario is.
I for one welcome our new stone wall making overlords!
It doesn't mean much now, it's built for the future.
...a hacked pump at a water station DOES NOT DESTROY THE COUNTRY.
I do security
I can think of no reason facilities such as this should be accessible via a public network. You should have to be physically present to access these control systems.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
Perhaps it's time that people realize that a lot of things do need to be connected to external networks and that "air gap them" is simply a cop out response equivalent to saying "use a typewriter".
Yes, some things should be air-gaped, nuclear gas centrifuges come to mind. However, many industrial control systems need to report information over the internet. Remote pumping stations, unmanned power distribution centers, etc. Having a lot of data is not simply a convenience. This data allows engineers to troubleshoot failures, predict future failures, and adjust systems for optimum efficiency.
What's really necessary is for some kind of device that will communicate the data to remote places, but refuse to pass any messages from the outside onto the control system. I don't know how difficult this is, but it's certainly harder than "air gap it". On the other hand, this solution actually addresses the problem.
-1 disagree is not a modifier for a reason. -1 troll, flaimbait, redundant, overrated are NOT acceptable substitutes.
Connecting your water pumps to the public internet.
Der der der.
Perhaps it's time to realize both statements are true and completely orthogonal to each other.
Leaping to the conclusion that pump failure in a SCADA-controlled utility is cyberwar is foolish.
Believing that anything remotely important should be connected to a publicly-accessible network is also foolish.
Both skepticism, and air-gapped networks, are very good ideas.
Welcome to the Panopticon. Used to be a prison, now it's your home.
I am soooo damn tired of word 'cyber' now. Used to be kind of a neat word, way back when it actually meant something.
What's really necessary is for some kind of device that will communicate the data to remote places, but refuse to pass any messages from the outside onto the control system. I don't know how difficult this is, but it's certainly harder than "air gap it". On the other hand, this solution actually addresses the problem.
So, what you're saying is, if a utility is too cheap to lay in dedicated network assets and buy their own blacknet (which is not hard to do if you want to), it's ok to just connect the the Internet?
That said, the thing you're looking for is called a unidirectional network. Back in my military network operations days, the colloquial name was "data diode". Data goes one way but nothing (no data, no handshakes, no signaling at all) goes the other way. In that environment, they were used to promote data from a lower-level security environment (say, Secret-only) to a higher-level one with no risk of leak-back.
Yeah. They exist. They're considerably lower-bandwidth than your average gigabit Ethernet switch, but if you're just talking SCADA telemetry, they should suffice.
Welcome to the Panopticon. Used to be a prison, now it's your home.
Is all this crap attached to the intertubes?
OK, now we don't even have to come into the office to change the position of the control rods and avert a meltdown, we can do it from home, or heck... Kazan, Russia if you really wanted to.
This sig is not paradoxical or ironic.
Weiss said the report says the cyber attacker hacked into the water utility using passwords stolen from a control system vendor and that he had stolen other user names and passwords."
In other words, people are not capable of understanding the situation they are in. Computers are mysterious, magical creatures, with pink tails and fluffy hair from which you can hold on when riding on the waves of the cyberspace holding a pink bunny, a packet of noodles and wearing the everlasting Viking helmet.
Do we have Water Over IP already?
Retards, stop trying to connect everything to the Internet.
manuals for all this equipment on the the internets in conspicuous locations in the control network with special attention given to tolerances and acceptable operating parameters. So that hacker wont accidentally damage the critical infrastructure component they are the playing with.
Heck, we may want to make more foolproof by publishing user dashboards, with very strict input checking, at playwithcriticalinfrastructure.com.
It the responsible thing ...and knowing is half the battle.
This sig is not paradoxical or ironic.
Yes, yes, yes!!! If your system integrator needs remote access, then when he's done, UNPLUG THE %$^^&^ cable!!
o_0
"Joe Weiss, a noted cyber security expert ..."
"... cyber attacker ... "
Seriously! Who fucking talks like that? Cyber cyber cyber .....
Will 'digital outlaw' ever catch on?
Folks, let's be honest about this. You only attach crap to the intertubes so you can send people home and not have people on hand to cover their jobs. "The water system is out of whack again? Call Johnson at home, he can get in through his PC." Cheaper than hiring a new person and the Republicans get to claim a big, fat score on the reducing the gub'ment goal.
Easy, two-part solution here.
1) Disconnect all of our stuff from the net
2) Hire some people who know about the systems to work the other shifts
Problem solved and not a single right was harmed in this solution.
Who did they use a IT consulting firm? well some use remote admin / monitoring and for something like this I hope that some one would at least say something about how unsafe it was.
Did they use a outsourced IT firm that may do stuff like have call centers out side of the usa. They may have on site desktop guys but at some they have to fix / what the clueless call center go wrong.
Now I hope that a system hooked to the water pump was not manged the same way that all other desktops are. Now some firms may do that and the water pump system ends getting software pushed that it does not need and opens it up from some to use that software with a hole to hack in.
But a computer repair company said it was hacked? Did use the geek suard or some one like them if so then the techs can be very hit or miss. And some may say they where hacked when it was some other fault.
This what out souring and useing vender systems get's you people who are not there and or see you as just a other client you want use to come out a look at a system out of it's window that's a added change.
Why do you hate our freedom?!?!?
However, many industrial control systems need to report information over the internet.
Maybe over AN internet, but not over THE Internet. "Report information" is not the same as "allow incoming control or information."
This can be as simple as a Lantronix XPort (or equivalent) tied to a serial port TX line on a secure machine, allowing telnet connections to read the serial data coming out but not send anything back. Or any terminal server with the RX lines cut.
What you need to be careful of in the planning of this system is that the information coming out of the secure system isn't being fed back into the system as the result of an external control. I.e., "Water level low in reactor 5" as outbound information cannot cause an "increase water flow to reactor 5" command from outside.
Why such systems are online and accessible via the Internet? Is this a cost cutting measure? Why aren't critical passwords changed every week? Why isn't database information stored in encrypted containers or hard drives? Why does this happen again and again and again?
Several stories online of laptops containing massive DB's get stolen, in fact a previous employed of mine (major chip manufacture) got one of their HR laptops stolen out of a car at Starbucks, I was sent a letter by said company giving the excuse "The laptop hard drive could did not support full drive encryption" which is complete bullshit, full drive encryption has been around for a long time, as have encrypted containers.
Why was that DB allowed on a laptop? Why was it left in the car, but the best question is why wasn't the entire drive encrypted, or at the very least the DB put into an encrypted container?
"If any question why we died, Tell them because our fathers lied."
I have worked with SCADA and water filtration plant pumps, big ass pumps, like 650hp pumps that run on 7200volts.
You cant set it to "burn out". you can adjust the speed of the pump from 10% to 100% the only way to kill a pump is to drop power to it without dropping power to it's valve so it will not close. wait for the pump to start spinning backwards from the water running back downhill through the pump and then slamming the power back on at 100% after the pump was free wheeling in reverse at full speed.
Then they don't burn out, they freaking explode.
This happened when we lost power plant wide and a hydraulic failure kept the valve from auto closing. (not electronic, it's a mechanical/hydraulic thing, a blockage in the pressure line)
Unless the plant was designed by a utter moron and made it so a programming error could blow up parts of the plant.
Do not look at laser with remaining good eye.
So is that idiot/moron/incompetent-high-school-dropout saying that the so called attacker hacked through the internet into a physical facility (that requires a person to physically push buttons from the control room) ?? Did they just gloss over the fact that a person has to control things from the control room and hope people just ignore that part?
I've been in the water SCADA industry for 10 years. What I'm seeing lately are water operators, IT people, and system integrators who are overzealous when it comes to connectivity and all the "neat" things that can be done remotely via technology. It's the standard human foible when it comes to technology, writ dangerous: they consider what can be done versus whether it should be. The water industry isn't that exciting, so when flashy tech. comes along, and the taxpayer is footing the bill, I can see where they say "Yes!" And who is the salesperson to refuse this order?
I'm all for automation, and crying out when a system is in trouble. But I haven't yet seen where humanized remote control is critical. Hackers aside, it's probably better if it's not.
Don't worry with http://motherjones.com/mojo/2011/11/america-getting-domestic-indefinite-military-detention-thanksgiving on the way its all good
Domestic spying is now "Benign Information Gathering"
Water pumps don't normally affect water pressure. They pump water to the top of a water tower for storage until gravity pushes it out.
That is not how they "normally" work. It is only one way that a water system can be designed to work. It can also be designed with pumps that pump directly into the system.
Most water systems of any decent size have a combination of both elevated storage and pumps. Some parts of the water system's distribution are may be pressurized by elevated storage tanks, while other parts of the distribution area are pressurized by pumps.
"Weiss said the report says the cyber attacker hacked into the water utility using passwords stolen from a control system vendor and that he had stolen other user names and passwords."
How likely is it that a control system vendor would have the usernames and passwords of their client, used in the actual production system? Maybe they actually do, as part of some sort of remote support agreement, but if this is the case, that's already a bad security practice.
It seems more likely to me that the vendor has a list of default usernames and passwords, and THIS is what was obtained. Perhaps what Weiss *really* meant to say would be be something like: "Someone got ahold of the default usernames and passwords that our vendor uses. Since we never changed them from the default values, it's our own damn fault."
After seeing SO many stories like this, it's usually a case of not changing default passwords. Given that Weiss's statement *could* be read as I have read it, this seems the most likely scenario to me. I'm going to write this one up as stupidly bad security policies until I have sufficient evidence contradicting this assumption.
It'a called a data diode
The local TV news is on, and they just said that it was Curran, a tiny town five or ten miles from Springfield. They're concerned that the system might have been hacked because the company that designed the system discovered evidence of a breach of sensitive data... passwords, maybe? They did say it was gigabytes of data.
Free Martian Whores!
Joe Weiss is fairly notorious in the control system security world as the first to say, "Hey! That was a cyber incident!" For example, he said this about the BP spill, when they were still investigating it...and while it turned out to be true that some alarms were turned off because of computer issues, the real root causes had to do with faulty mechanical equipment and bad concrete, and that the cyber aspect was pretty much entirely irrelevant. Hear him speak, and it's a safe bet that you'll hear about his book, his conference, and other ways in which he can make money telling you how awful things are in the world and how much you need to listen to him.
For your security, this post has been encrypted with ROT-13, twice.
So, what you're saying is, if a utility is too cheap to lay in dedicated network assets and buy their own blacknet (which is not hard to do if you want to), it's ok to just connect the the Internet?
Because all utilities are in developed areas and have tons of cash to burn, right? Natural gas utilities have equipment that must be monitored and/or controlled remotely that may not even have electric service in the vicinity, much less telephone or fiber optic cable, leaving satellite and cellular modems as their only options. Do you really think building a private WIMAX network or launching a satellite is within the budget of a local utility serving 25,000 customers? There is no doubt that many utilities are being lax in their risk assessments and security precautions these days when it comes to using the Internet, but your statement is just silly.
Most industrial control systems I've worked with (typically power plants) have their root passwords set to the same one from the OEM. They are rarely changed. Many of these HMIs are now networked on the company LANs to data historian servers. Sometimes there's a firewall, sometimes not. Most HMIs and historians are running a variant of Windows Server, with a few Unix flavors out there still.
The other, much cheaper solution to this is to place a router that has ACL's allowing unidirectional udp traffic...
It's called a pair of wire cutters for the receive wire of the Ethernet cable and UDP feed on the transmit side.
That'll be $500k consulting free please!
There are devices called "data diodes" that serve this purpose.
Error is much more likely than malice, even if the computer is infected.
In a place where the infrastructure is as wide-open and completely unprotected as it is in the US, there are much better targets that require much less investment of effort and expertise.
What I'm seeing lately are water operators, IT people, and system integrators who are overzealous when it comes to connectivity and all the "neat" things that can be done remotely via technology.
Yes. Read "Access Your Embedded Controller with Ease through a Web Server", from Texas Instruments, which ought to know better. "The designer should also make it as easy as possible to change the settings on a piece of equipment, reconfigure its operation, or fine-tune the system. The more intuitive and explicit that activity is, the more likely the result will be what the operator desires. Losing the instruction manual can seriously impair the user's operation of many systems."
What that paper describes is a family of embedded controllers with a web server in each controller and no security. What's wrong with this picture?
http://pastebin.com/Wx90LLum
Not by me.
Microsoft aggravates my tourettes syndrome.
intergenerational cultural references
"precious bodily fluids" is something someone in their 60s would get
"zerg rush! kekekekekekeke" is something someone in their 20s would get
for those of you who have never seen dr. strangelove (and it should be required viewing for any geek culture nerd, or just plain kubrick film buff):
http://www.youtube.com/watch?v=N1KvgtEnABY
and for you older folk, you will find Starcraft to be quite an enjoyable strategy game, perhaps while listening to General Ripper discuss anti-Communist strategy
intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
Wow. Not a relevant word in that document about security, isolation, or protection. And it was written in March of 2010! It's not like it's from 1998 when malware meant the GOOD TIMES virus in your inbox.
That would really disturb me if I thought it would impact me in any meaningful way. Now if you'll excuse me, I have to go figure out why the water isn't working.
John
I know that it's not often acknowledged, but in the long run there's one ironclad rule, enforced with all the ruthlessness of natural selection: If you can't afford to do it right, you can't afford to do it at all.
You don't tackle vast projects with half-vast security. You're just spending lots of money to embarrass yourself and let down people who depend on you, if you try.
Security is not optional. All the impediments you described are merely challenges to engineer around. The only real insurmountable obstacle is not giving sufficient damn to actually try to overcome the other issues rather than handwaving them away as "too hard" or "too expensive."
And that, my cynical friend, is what's silly.
Welcome to the Panopticon. Used to be a prison, now it's your home.
Spoken like someone who has never worked in the real world. Life isn't black and white - real life decisions are made on a relative basis. Few organizations have the luxury of a DOD budget yet they still have to continue operating. You seem to equate "right" with "expensive". Whether a small co-op, statewide regulated utility, or publicly-traded mammoth, cost always has to figure into the equation. Business is about risk management, not spending unlimited dollars in search of the "perfect" solution.
Security is not optional. All the impediments you described are merely challenges to engineer around. The only real insurmountable obstacle is not giving sufficient damn to actually try to overcome the other issues rather than handwaving them away as "too hard" or "too expensive."
And the only solution you've offered is to build a dedicated network. There are other solutions that you can implement to engineer around the challenges, make use of the public Internet, and are considered good enough. But I guess when all you have is a hammer, every security problem looks like a big nail.