Slashdot Mirror
Industrial Control Software Easily Hackable
jfruh writes "CoDeSys, a piece of software running on industrial control systems from hundreds of vendors, has been revealed to be easily hackable by security researchers, giving rise to a scenario where computer hacking could cross the line into the physical world. Worse, many of these systems are unneccessarily connected to the Internet, which is a terrible, terrible idea."
Now you're just being paranoid. Instead, you should develop an artificially intelligent system to defeat would-be attackers and malicious software. That sounds like the best idea.
- Skynet
It's not necessary.
Mostly random stuff.
Make the first episode of BSG Season 1 required viewing for "intro to computers" class.
This is a mouse, this is a keyboard, this is why you don't jack your global defense grid into a wifi hotspot.
may need unions as well so the coders can stand up to the PHB's and say that...
That time table is to tight
We need more staff and the 80 hour weeks are just makeing us make more errors.
We can't cut QA
You can't hire people who can't pass the certified test but have BA/BS while passing over people who have passed the test but don't have a BA/BS.
No I will not add this new stuff to the code this late in the roll out hell we still have some big bugs in the code base to work out.
No will not use that POS best buy special as the system that will run the PLS hell it's PSU is a very poor one per this review of it.
"Did professional engineers prevent the Challenger Disaster?"
.. ref
No, nor did they cause it, what did cause the disaster was political interference, such as the decision to manufacture the solid booster rockets in another state, necessitating them being made from segments bonded together with O-rings
AccountKiller
"Did professional engineers prevent the Challenger Disaster?"
No, they did not. They tried like hell to prevent it, they were quite certain there was going to be an issue, because they knew the seals failed with lower temperatures, and seals had failed at temperatures not as extreme as on that day, so they were pretty certain there would be a problem and tried to stop the lunch. Sadly, it was not the engineers who were ultimately responsible for that launch, but folks more worried about bad PR.
So, what was your point?
I was doing some electrical work at an oil refinery up north in Canada about 5 years ago. I wasn't specifically attached to their control systems or PLCs, though since the electrical was intertwined with a bunch of the automation I naturally knew all the guys who were taking care of that portion of the project since we were required to collaborate together.
On one particular day, I entered the facility as usual and was heading to an unfinished section to check out some conduit. On my way there I noticed a CAT5 cable stretched across a walkway, disappearing into a stairwell. This was so incredibly absurd and suspicious that I just had to see what the hell was going on, even though something in my head told me I didn't want to know. I traced the cable back to the management office where it was plugged into one of the network switches. Okay, weird- follow it back in the opposite direction, all away across the plant- after about 80 meters there was a hub/repeater dangling over a walkway rail plugged into the wall and another CAT5 cable stretching off into the oblivion. Following the second extension cable led me to a set of PLCs and a group of the control guys throwing vulgar insults at an Allen Bradley PLC unit.
Turns out the PLC was a "new" model. Instead of handling the licensing through a floppy disk (!) like all the old units did, this one used some sort of a proprietary activation scheme that had to run over the friggin' internet before the PLC would actually do anything. The CAT5 cable I'd traced about 180 meters across the plant going back into the office internet connection was setup to allow this process to complete, since they had apparently failed to do it earlier when the system was OOTB but not yet hooked up.
They eventually got it all working, but it took them about 5 hours of fiddling to get the damned thing working properly.
Shit like that is the reason why things are hooked up to the internet, sometimes improperly. I know there's certain requirements for remote monitoring and such, and that should all be done over an isolated, encrypted VPN- but then you've got licensing bullshit like this that expects to phone home to a random server on the internet with little or no fire walling in-between. There's no reason for it otherwise- apart from the PLC guys wanting to make sure you're licensed and all paid up, god forbid anyone should buy a second hand PLC and reprogram it to do something useful again.
-AC
I've said it before, and I'll say it again: What possible reason could industry have to put controls networks on the internet? I can understand putting some type of reporting on the internet, so the bigwigs can keep track of up to the minute production. *disclaimer: I am an industrial electrician. I work on industrial controls in a sawmill. The day a production foreman asks us to give him control of machinery over the internet is the day I find a new industry.
Talk is cheap. My guess: They cannot do it, but enough people will believe them. Once the OS is in place, they cannot migrate away anymore.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
??? they cant pick a live Linux CD?
Sadly no one will listen until something bad happens.
If you told someone pre-2009 about the need for financial regulations and the upcoming collapse people would call you a communist and a liberal! Peter Schiff did jsut that and was laughed at before he earned fame when the Great Financial Collapse hit.
Same is true with nuclear powerplants after fukashima, airport security after 9-11, and same after the space shuttle Challenger exploded, IE 6 security after code red. Money talks and shit walks. Only when deemed necessary does something change.
Right now sadly we might be without power or worse another nuclear powerplant meltdown here in the US caused by Iran before anything gets done. Not unions or professional software orgnaizations or even licensing.
People hate change and especially MBA PHBs who never have heard of a single internet security attack on a PLC piece of equipment. If you can't do it MR. Slashdotter reading this then someone else will since it is never a problem.... therefore it is perfectly secure etc.
I mean they hated upgrading browsers too until IE 6 was shown a risk and they still love XP despite it. Why? Money. Until it becomes a liability and laws come into effect and PHBs shit their pants the problem will nto be solved
http://saveie6.com/
Actually works better if you read it as 'clown' services.
Faster! Faster! Faster would be better!
The CAT5 cable I'd traced about 180 meters across the plant going back into the office internet connection was setup to allow this process to complete, since they had apparently failed to do it earlier when the system was OOTB but not yet hooked up.
Assuming it was all Rockwell/Allen+Bradley gear then it was undoubtedly the FactoryTalk Activation system they were struggling with, and they were undoubtedly unqualified to be doing the work they were assigned to do (disclosure: I am a former Rockwell Automation employee so I have familiarity with the subject, but apart from that I do not speak on behalf of any employer past or present here).
First and foremost, Allen+Bradley(AB) PLCs don't need activations, so the licensing really isn't relevant to this story. AB makes a crap-pile of profit on that hardware the moment they've sold you the box--activation makes no sense. What DOES need to be activated (and is what creates profit for the Rockwell Software division) is the RSLogix programming software, without which the PLC is as useful as a doorstop. So unless they were completely clueless they'd have just taken their laptop into the office and activated their software then come back, rather than break all sorts of IT, security and safety rules stringing out 180m of CAT5 and a spare switch to get internet. The same goes for their drives--the drive units don't need activating but DriveTools software on the programming laptop may have.
That said, there may have been an industrial PC like a VersaView or third-party unit running the Rockwell HMI software and was bolted into the cabinet with un-activated software for some reason, but Rockwell/AB have thought of that...
The legacy licensing system used utility software called "EVMove" and relied on "master disks" (towards the end you could set up a USB flash drive) and in the field this was a royal pain in the ass--floppies and their drives are far too sensitive for such an environment, and USB memory sticks are terrible to manage and secure. Thus the development of the FactoryTalk Activation internet service-based scheme. Though it requires the internet the end system does not need to be connected to activate. The easy "wizard" way sends a "host ID" (the ethernet MAC address or some such number) from the end device to Rockwell via the internet. However, you can actually write down the mac address, or generate the hostID file on the target machine, then go to an internet-connected computer and type the hostID into a secure web form or upload the hostID file. The website then generates a license file that you can save to removable media or a laptop/portable machine to take over to the target machine physically, thus preserving the air gap (and making the method more similar to the old EVMove floppy method).
I do agree that licensing/DRM/activation is a big problem that costs end users millions of dollars globally (above and beyond the actual purchase cost of the products). It adds complication and downtime and confusion and contributes exactly zero value to its users. One might argue about its value to the vendor as well--FactoryTalk activation and many other similar schemes are just as trivial to circumvent as CoDeSys' ladder logic runtime for hackers, and adds the burden of extra support costs from the honest users it keeps honest. But the problem in industrial automation is bigger than that. The problem is that the world in general moves faster than industrial control systems can keep up, and the people who have "experience" honed their skills in the mid 1990s or earlier and haven't kept up. In the meantime, PHBs of the world in management and government demand of them far more than they are capable of delivering.
It used to be that refineries/factories/etc were content with paper chart recorders where operators and plant managers could peruse them if something came up to troubleshoot. Then came data recorders where you could plug in a serial cable or transfer via floppy to a computer for more deta
Preaching that automation systems be kept off the internet is like preaching abstinance until marriage to teens. It sounds like the lgical solution to all the problems but it is unreasonalbe to ever expect it to happen, so the best course of action is to educate on how to do it safely and responsibly.
Ther are many valid reasons that automation systems are connected to the internet in some fashion (though they never need direct internet access). Some of those reasons relate to not braking the law.
In industries like oil and gas, regulators require data to be collected 24/7/365 on all critical aspects of an operation. If an environmental or safety incedent were to happen and such data was not available for scruitiny it could lead to the permanent closure of that operation in extreme cases. Lack of due diligence in such matters can mean huge monetary fines and even jail time for wilful violations.
As such, in those operations a "process historian" server is standard equipment. These are central data logging servers that have essentally full read-only access to the industrila control system, and even some limited write access too (say, to assert a bit in a PLC to confirm it has received data, or to reset a totaliser or set a new batch number). Becasue of how vital the data is, there has to be some way to get the data off-site for archival and reporting purposes, and because of the volume of data and the immediacy that is demanded removable media is not an option. Thus these systems end up with some means of corporate network access. This does NOT mean the need "direct internet" access, but very commonly it means tunnelling through public/internet infrastructure via VPN (the "condom" if you will). Though technological measures can be taken to make this route into the plant impeneratable, it is complex enough to set up that people make mistakes and thus you end up with "holes in the condom".
The other use for outside conenctivity relates to support from off-site engineers, vendors and operators. A control system can be set up to report critical alarm conditions to smartphones, email inboxes and the like automatically with much more rapidness than a human operator at the board can do. The more rapid response to a critical incident the less likelihood for loss of revenue, damage to equipment, and injury or death of workers (again, in the case of "sour sites"--thouse that deal with natural gas containing deadly H2S, rapid response is vital to evacuate the facility and surrounding area and some of these are required by law).
So "preaching abstinence" in the complete absence of "sex education" is a bad idea. It is ineffective to say "disconenct from the internet" and not say how you can manage network security safely and responsibly, because at some point these people will be pressured into doing it and need to be able to "say no" if they aren't ready, and to know when and why it is "the right time", becasue if you DO use that internet connection responsibly it can actually be a great experience ;-)
Speaking as the system administrator for a large DCS system: the OS will be no good without a complete redesign of the application level software. The problem is not really the OS, but the fact that in order to make everything work together 'automagically', there are hardcoded service accounts, and much of the app executables (which are often executed with system permissions) are writable because the entire installation folder is writable. And of course, the controllers that do perform all control actions use a protocol whose only real claim to security is obscurity.
And from what I can tell, the system I manage is fundamentally no different in that regard from DCS or SCADA systems from other vendors. While it is true that a secure-by-design would be a good place to start, the main problem atm is that the application architecture is hopelessly insecure.
Necessity is the mother of invention. That, or an article in the business newspaper your boss reads.
My solution to that problem was simply to subscribe to the same magazines my boss reads, peruse them for articles supporting my case and getting him to read it. Not only will he listen to them more than to you, he'll also think that you read "relevant" magazines and start listening to you, at least from time to time.
I know it's silly. hey, it's management!
We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
I'm a P.Eng. I work in the control system industry. Most of the people who work in this industry are P.Eng.'s or certainly have an Engineering degree. Most of the ones I've met know *nothing* about computer security. These Engineers are the ones plugging PLCs directly into office networks because they're EE's. They have little to no training in computer networks (short of setting up their home routers). They have no idea what a VLAN is. They have heard the term firewall but don't really know what one does. Usually it's also EE's who are the ones writing the software (like CoDeSys) that runs on these devices (because it's embedded software and it helps to have lots of good hardware knowledge).
The only reason I know something about computer security is because I'm very interested in it, and I really think we, as an industry, need to know this stuff. Unfortunately nobody else seems to agree with me.
At any rate, we *are* professionals. I pay hundreds of dollars per year to maintain my P.Eng. license. I had to write law and ethics exams to get my license. I can lose my license if I don't follow industry best practices. Unfortunately the best practices *in our industry* for this kind of thing are completely inadequate. In fact, none of the stuff we're talking about here actually requires a P.Eng. to stamp the design. Sure, the *safety* system has to be stamped by a P.Eng., but anybody can plug network switches together. So no, requiring control system guys to be licensed won't help... many already are. You need to educate the professional organizations and standards organizations, and have them change the regulations.
"I have never let my schooling interfere with my education." - Mark Twain
Ethernet is actually good enough for a number of things if and only if the network is unloaded. Reading values from a Modbus based protection relay for example. The values are not critical and even if the network fails the protection relay will still trip, but they are useful values to have and mean someone doesn't have to keep walking into the substation to look at them. I can think of a number of other such use cases where ethernet/ip is more than good enough. For remote IO, I would use something better, like DeviceNET(RS485, basically). But Even then, some remote ethernet IO is also good enough. It depends entirely on your use case. One size does not fit all. I am quite aware of the limitations of ethernet IP, and a lot of systems use the same physical layer as ethernet, but make special hubs (remember those?) mandatory instead of switches. I believe ethercat is one such system if you must know. In general ethernet IP is more than sufficient for any SCADA system (with a few exceptions). Time critical stuff should always be done in the (far more reliable) PLC not the (inherently suspect) SCADA system. But that is standard practice. Mostly. I have seen some horrible systems....
My skillset includes setting up linux security as well as programming PLCs and setting up windows security. I run various OS systems in industrial environments. But it is obvious you have never really worked with these systems? Your ideals are nice, but the real world called and wants to know if you'd like to meet for coffee sometime.
When people buy a machine they buy a machine. They don't think about the password because they bought a machine and they need no password to operate it. The salesman comes in and gives his spiel, and then they buy it. The SI or manufacturer password protects the PLC to protect his "IP" and that is that. It is annoying, yes, but I am more inclined to blame the SI/manufacturer than the customer. The customer's skillset does not include programming the PLC and if the system is made right, it should not have to. That is the point of the entire system - so that the customer does not have to worry about it. That is what sells. That is how it is, and how it works, and it is unlikely to change.
I have determined that my sig is indeterminate.
Erm, no. True engineers use metric. Anyone not using metric is by definition not a true engineer.
I do like the sophistry in that CNN article - blaming the "English" units, rather than "Imperial" units. Nobody calls them "English" units, that's just an attempt to try to distract blame from shoddy American sloppiness.
Also FatPhil on SoylentNews, id 863
No security expert will code their own software as long as OpenSSL, IPSEC, GPG and so on will do the job. Too expensive and too many mistakes to be make by bespoke software.
You sure about that? I mean, it's one thing to try to write a correct implementation of a very generic protocol (the ones you named), but who is to say that a tailored protocol sporting just the things you need for the app and nothing else can't be correctly implemented with a much lesser effort?
Ezekiel 23:20
Even if they could do it, very few ICS admins would switch to it. Most people there are responsible for stability as their most important attribute - and that means running a solution that has proven itself over and over and over again. Related to this concern is downtime: often times these plants are running 24x365 schedules, controlling furnaces that keep ovens full of molten iron from freezing solid, which could destroy the oven. Shutting down a production line takes time and planning to prevent damage, and every minute that line is down, they are not making money.
When there is a credible threat, they look at addressing the threat on an individual basis. Firewalls between the controller and the LAN. Epoxy in the USB ports. A locking cabinet around the CD-ROM drive. But replacing the core of the factory, on an unproven software package, just "in case" a hacker might target them? Not terribly likely.
John
Even if they could do it, very few ICS admins would switch to it. Most people there are responsible for stability as their most important attribute - and that means running a solution that has proven itself over and over and over again. Related to this concern is downtime: often times these plants are running 24x365 schedules, controlling furnaces that keep ovens full of molten iron from freezing solid, which could destroy the oven. Shutting down a production line takes time and planning to prevent damage, and every minute that line is down, they are not making money.
Indeed. What they actually need to do is to really isolate these control systems in the hard sense. I.e. no ports network, data import only manually, data export via CD-R or the like, clear message to employees that connecting any USB media, Laptops, etc. will result in immediate termination, ...
It can be done, even if it may require some people to suffer first, as Iran found out. They did execute the people that imported Stuxnet via USB drive. My guess is they will not have that problem again anytime soon.
When there is a credible threat, they look at addressing the threat on an individual basis. Firewalls between the controller and the LAN. Epoxy in the USB ports. A locking cabinet around the CD-ROM drive. But replacing the core of the factory, on an unproven software package, just "in case" a hacker might target them? Not terribly likely.
This is not enough. Firewalls are insufficient. They need to implement real isolation, i.e. only an isolated net may be used and that has to be very heavily protected. It will take quite some time for them to find out how to do that, although competent IT security people could tell them today. The problem is that they are asking the wrong questions and are looking for IT experts that understand their business, instead of looking for competent IT security folks.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
And the further simplistic (but still dumb and vulnerable) solution for you might be "two laptops". Only the red laptop connects to the equipment. The other laptop connects to the Internet and lets you read the manuals, docs, slashdot, etc. If you need to download a file, you format a flash drive in the red laptop, insert it into the black laptop and copy the file, then read it back in the red laptop. It's cheap enough, and adds another layer of difficulty. It might not have stopped Stuxnet, but it would have stopped everything else.
But I get that the network connections make these systems far more valuable than isolated systems. If you are working on a municipal water system, having you sit at a desk and remotely connecting in to monitor valves and pumps around the city means you can be effective in dozens of places at once. If you are working in a manufacturing plant, your manufacturing systems can tell your warehouse systems that productivity means you're filling 100 pallets of product per hour today, and your shipping system can schedule the right number of drivers and trucks. Disconnecting is no longer an option in 2012.
The best solution would be to create a test platform that everyone trusts is effective. Prove that you can test security upgrades and guarantee that they won't be bringing the factory down. Get the CEO to sign off on the plan that says "we will test every security patch in this new test system, and install every patch in production within one month." And have billion dollar lawsuits hanging over the vendor's contracts.
It's all very expensive. But how expensive are the vulnerabilities if an attacker does get in?
John