Hackers Gain "Full Control" of Critical SCADA Systems

mask.of.sanity writes "Researchers have found holes in industrial control systems that they say grant full control of systems running energy, chemical and transportation systems. They also identified more than 150 zero day vulnerabilities of varying degrees of severity affecting the control systems and some 60,000 industrial control system devices exposed to the public internet."

Comments

Note the mention of insufficient entropy

[davecb]

I suspect the Siemens and Sietec people are now on a wide-ranging entropy hunt, probably along with the German Federal Security Service (:-))

Comment removed

[account_deleted]

Comment removed based on user account deletion

i hope people with SCADA systems learned.

[Gravis Zero]

do NOT connect SCADA systems to the internet.

Re: i hope people with SCADA systems learned.

[paugq]

The air gap is not the solution. Proper isolation, firewalling and virus/malware is.

Re: i hope people with SCADA systems learned.

[Gravis Zero]

The air gap is not the solution. Proper isolation, firewalling and virus/malware is.

wrong. if you dont want someone from altering your system, you make it completely inaccessible. in the age of compromised security systems, firewalls and "proper isolation" are just pesky things to slow down/dissuade most hackers. if it's a state sponsored hack attack (NSA or China), then you are going to get slammed until they find a way in.

Re: i hope people with SCADA systems learned.

[clovis]

Proper isolation? If by proper isolation you mean an air gap, then OK, I agree.

"Proper firewalling" is a pipe dream. If you have a firewall, then you have external access and a vulnerability right there.
Whatever port you have open is an access point, and thus a vulnerability.
Keep in mind that many of these systems have hidden backdoors or default admin accounts for maintenance.
And the reply "it's OK if it's properly configured" would be true if every system had network admin that was 100% competent. Do you wish to make that claim?

"virus/malware"? I suppose you mean anti-virus/malware. There is no such thing a 100% effective anti-virus/malware software. They are not even close.
Keep in mind that the anti-virus software in itself is a vulnerability.

Re: i hope people with SCADA systems learned.

[Billly Gates]

To prevent piracy and sales of used Scada these require internet access to stay activated. We wouldn't want to deprive income now would we

Re: i hope people with SCADA systems learned.

[Cley Faye]

The air gap is not the solution. Proper isolation, firewalling and virus/malware is.

No. Firewalling, virus protection, malware detection... all these techniques can be flawed, either by design, because of oversight...
It is acceptable for most system (because these issues get fixed after a while), but for a SCADA system you don't want a zero-day to be exploitable *at all*. Your system can have a ton of backdoor/vulnerabilities/exploits, if it can't be reached by any other mean than physical access they are not an issue.

Re: i hope people with SCADA systems learned.

[aaarrrgggh]

The problem with making some of these systems inaccessible means they have almost no real functionality at that point. Using the tritium JACEs as an example, the whole point of them is the network, and to exchange information in higher level protocols.

In the old days we separated systems and interfaces between systems with relays and analog i/o. While it worked then, now we have 100x points (many diagnostic rather than control) and it just isn't practical. Today's practical solution would be the SCADA as primary, with a lot of hard-wired safety interlocks. The problem is there really is a shortage of people that can troubleshoot those things, so it is likely to be disabled within 5-10 years, or once needs change.

Proper security is hard, and when 80% of it is in a black box provided by a (adversarial) third party, this is what you get.

Re: i hope people with SCADA systems learned.

[aaarrrgggh]

An air gap just limits the remote attack capability, and is fairly easy to defeat with local access. At every level you need to limit the attack surface.

Re: i hope people with SCADA systems learned.

[fisted]

What use is an air-gapped machine? How do you communicate, how do you control it? Build your own physical network infrastructure (preferrably with blackjack and hookers)?

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

Why not just use a cryptographic dongle? And when the company goes under talk to these guys to have the need to have a dongle removed.

As an aside, this *appears* to be the same guy that wrote Raw Copy for the Amiga back in the day.

Re: i hope people with SCADA systems learned.

[ebno-10db]

"Proper firewalling" is a pipe dream. ...Keep in mind that many of these systems have hidden backdoors or default admin accounts for maintenance. And the reply "it's OK if it's properly configured" would be true if every system had network admin that was 100% competent. Do you wish to make that claim?

I think some people used to "conventional" IT don't appreciate how unrealistic it is "properly configure" (in terms of security) every box on a SCADA network. A typical network consists of a plethora of different types of boxes, with different OS's (often just RTOS's, which are usually not that security conscious), and all sorts of configuration, testing and latency requirements that go beyond what's needed in normal IT. Think in terms of making sure that robot arm doesn't smash into anything after your latest security update. Also, these boxes aren't, and realistically can't be, monitored all the time by checking log files and so forth.

A similar situation occurs in aircraft, including military aircraft. I assure people there aren't firewalls or other security provisions between various avionics boxes. The big concern is reliable, error free and low latency communications between boxes. It's bad news if an actuator/sensor for a flight control surface has trouble, or takes too long, to talk to the main fly-by-wire system. Security is about "don't let it through unless you're sure", which obviously conflicts with the more important goals.

Want security? Don't connect to the Internet.

Re:i hope people with SCADA systems learned.

[StripedCow]

And do not allow USB-sticks or other media to be inserted into these systems.

Re: i hope people with SCADA systems learned.

[ebno-10db]

What use is an air-gapped machine? How do you communicate, how do you control it?

As hard as it may be to remember these days, it is possible to communicate without the Internet (especially when that communication need only be local).

Re: i hope people with SCADA systems learned.

[Z00L00K]

Don't forget that you now and then see ads that are infected.

Makes me wonder how many ad servers that serves ads with a hidden bomb that we haven't seen yet because it waits for the right conditions.

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

Why does a SCADA system need access to the interwebs?

Re:i hope people with SCADA systems learned.

[Ol Olsoc]

do NOT connect SCADA systems to the internet.

Not bloody likely. We're expanding, with lot's of home surveillance systems, ans coming soon, the "internetted" automobile.

The great thing is that nothing can go wrong with this sort of stuff.....

Re: i hope people with SCADA systems learned.

[Ol Olsoc]

What use is an air-gapped machine? How do you communicate, how do you control it?

So we ran these machines with no control or communication before the interwebz?

If you want to run these things on the internet, they will be hacked.

Re: i hope people with SCADA systems learned.

[Ol Olsoc]

Why does a SCADA system need access to the interwebs?

So they can update their Facebook pages?

Re: i hope people with SCADA systems learned.

[pacman on prozac]

You can make it accessible without putting it on the public Internet.

A lot of the companies who run SCADA devices will already have some form of MPLS WAN, most providers can give you DSL links onto that network rather than Internet. Lets you reach the device but doesn't let the rest of the world.

Or if that's not an option then stick a cheap VPN endpoint infront of it and run the comms over IPSec.

Re: i hope people with SCADA systems learned.

[fisted]

As hard as it may be to remember these days, it is possible to communicate without the Internet

We're talking about systems here, not intersocial communication.
 
If you air-gap a machine, then you need to hire people to maintain the machine locally. This just does not scale.

especially when that communication need only be local

For instance?

Re:i hope people with SCADA systems learned.

[satuon]

Can't they put a computer before them, that requires SSL/TSL connections, and authenticates any socket before forwarding it to the SCADA computer? A proxy, so to speak.

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

If you air-gap a machine, then you need to hire people to maintain the machine locally. This just does not scale.

Sure it does. Just because it doesn't scale the way you think it should, doesn't mean it doesn't scale. It may cost them more to hire more people, but if that cost is less than what it might cost if their system is compromised, then things are good.

Re:i hope people with SCADA systems learned.

[Anonymous Coward]

If someone's dumb enough to have auto-run on USB then I doubt they would do anything else related to the security properly.

Re: i hope people with SCADA systems learned.

[sumdumass]

So they can be monitored and administered from a central office 2000 miles away by a few employees at a location which houses all the accountants, sales reps, and so forth that the companies rely on in order to maintain production levels. This allows them to drastically reduce costs of administering them as a t1 connection is about 1/10 or less of the cost of one of several IT staffers that would be required to maintain them at local only access. And much more cheaper then travel and housing expenses of transporting central IT employees to the sites.

Another reason is that some SCADA systems aren't actually purchased. They are sort of rented and need to contact a server in order to validate their installs and operate periodically. This happens when there is a yearly or some sort of fee associated with the devices. It seems the more you spend on devices, the more common this seems to be. Even in the software world, I watched a company spend over $20k on a hospitality management suit in order to manage about 100 rentals and they had to purchase a license yearly for around the same amount in order to keep using it. One year, I blocked internet access to it through a change in the firewall rules months before the renewal process and it couldn't update it's license and stopped working for half a day before I figured out what happened. The only reason it ever needed internet access was specifically to update it's license once a year when the contract was renewed. Credit card processing happened on the phone lines using POTS through the PBX until the phone system got replaced and the changed to an entirely different system offering free in country phone calls to all guests.

Re: i hope people with SCADA systems learned.

[schwit1]

Then use a VPN. This allows remote access without internet access.

Re: i hope people with SCADA systems learned.

[sumdumass]

As long as the other end you are needing to contact will use one too, this is viable. However, that isn't always the case or possible. VPNs can also be exploited and defeated. If one machine that is allowed in the VPN becomes compromised, the entire security model of a VPN is defeated. It really is a lot more complicated then doing one thing.

Re:i hope people with SCADA systems learned.

[istartedi]

do NOT connect SCADA systems to the internet.

Do have employees running around in trucks to check things, or actively monitoring larger systems that need constant attention. Do charge customers more money to support those extra employees. Do make decisions based on daily dumps from mag tapes somebody drove over to the central office. Note, I'm not saying that's a bad idea. I'm just pointing out the trade. I bet a lot of things were done like that up into the 1980s. I have personally driven mag tapes from one office to another. It helped me earn spending money for when I went back to school. Maybe we fix the employment problem and the security problem by dialing back technology just a bit?

Re: i hope people with SCADA systems learned.

[frisket]

This allows them to drastically reduce costs of administering them as a t1 connection is about 1/10 or less of the cost of one of several IT staffers that would be required to maintain them at local only access.

Until someone cracks their way in. Then the falsity of this economic model is exposed.

Another reason is that some SCADA systems aren't actually purchased. They are sort of rented and need to contact a server in order to validate their installs and operate periodically.

This can be done over something other than the Internet, as several people have explained.

Re: i hope people with SCADA systems learned.

[sumdumass]

Until someone cracks their way in. Then the falsity of this economic model is exposed.

Sure, but when it was developed, this entire threat was pretty much non existent in reality. That has changed but the model hasn't exactly caught up yet. That is why exposure and working on it needs to happen.

This can be done over something other than the Internet, as several people have explained.

Sometimes it can be and some times it cannot be done. The problem is actually having both sides participate in doing so which isn't always the case or even possible to some degree. Anything can be done if the technology permits it, but if the manufacturer or some piece in the necessary puzzle doesn't participate, then you are screwed into doing something else. And even when it can be done, all it takes is a compromised machine inside the network in order to undo anything related to securing the systems.

Re: i hope people with SCADA systems learned.

[reboot246]

You speak the truth. I know a natural gas company right now that is installing a SCADA system. Why do they think they need it? I don't know, but I can tell you that they are a small system and don't need such a system. They operated for decades just fine without it. I think sometimes shiny new technology blinds managers to reality.

Re: i hope people with SCADA systems learned.

[whoever57]

if it can't be reached by any other mean than physical access they are not an issue.

Tell that to the people running centrifuges in Iran. Their machines were air-gapped, but they still fell victim to Stuxnet.

Re: i hope people with SCADA systems learned.

[paugq]

It seems you have little knowledge of the SCADA world. The air gap is an illusory security. Iran's nuclear plants had SCADA computers air gapped from the IT network. It did nothing: a USB, a CD, a virus infecting an update to your very SCADA software, etc will bring you back to reality.

Re: i hope people with SCADA systems learned.

[DarkOx]

Security is about "don't let it through unless you're sure", which obviously conflicts with the more important goals.

No security is about, availability, integrity, and authorization. If the system needs low latency communications that is an availability concern; its absolutely part of the security practitioners job to make sure those availability and integrity goals are met. They are not competing goals they are complementary goals.

Security experts who don't understand that are not in fact experts. People who think security just gets in the way also need to shut up and listen.

Re: i hope people with SCADA systems learned.

[greenbird]

Proper isolation, firewalling and virus/malware is.

No it isn't. That is a recipe for failure. Simplify and secure the system. Reduce the points of failure to the minimum and make sure the few that are required are secured. Adding more complexity and more points of failure just increase the probability of failure.

Re: i hope people with SCADA systems learned.

[PlusFiveTroll]

If you're the only company in the industry and exploits are very common your thinking would work.

Reality disagrees.

Your competitor will have vastly lower cost and higher efficiency up until the point that he is exploited. If it's 5 years between exploits, then it's likely you've already been put out of business.

Re: i hope people with SCADA systems learned.

[PlusFiveTroll]

Right, just like we ran computers before the internet existed. Why don't you just unplug yours and I'll mail you DVDs?

Re:i hope people with SCADA systems learned.

[Jeremi]

And do not allow USB-sticks or other media to be inserted into these systems.

That's going to make installing bug-fixes interesting... perhaps they send a new computer from the factory and swap out the existing one?

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

Not strictly true, for the most part in the SCADA systems I have seen, activation is handled via dongles/encrypted files.

That's not to say that people aren't connecting them to the internet, they are and its a problem. A recent audit of a control system's company's SCADA package I used to work for resulted in the best practice being to not connect it to the internet as there we a significant number of vulnerabilities in the software.

Re: i hope people with SCADA systems learned.

[radiumsoup]

hell, even cheapo Avaya telephone systems use this as a licensing scheme, and it works well enough (until it doesn't, anyway)

Re: i hope people with SCADA systems learned.

[CowTipperGore]

A natural gas company should at least monitor pressure at critical locations via a SCADA system.

Re:i hope people with SCADA systems learned.

[CowTipperGore]

I get your point, but none of that requires the SCADA system to be connected to the Internet. It does require a dedicated network for SCADA completely separate from your LAN/WAN but you can do all of that with technology and not touch the Internet.

Re: i hope people with SCADA systems learned.

[thegarbz]

Proper isolation? If by proper isolation you mean an air gap, then OK, I agree.

"Proper firewalling" is a pipe dream. If you have a firewall, then you have external access and a vulnerability right there.
Whatever port you have open is an access point, and thus a vulnerability.
Keep in mind that many of these systems have hidden backdoors or default admin accounts for maintenance.
And the reply "it's OK if it's properly configured" would be true if every system had network admin that was 100% competent. Do you wish to make that claim?

"virus/malware"? I suppose you mean anti-virus/malware. There is no such thing a 100% effective anti-virus/malware software. They are not even close.
Keep in mind that the anti-virus software in itself is a vulnerability.

I'm sorry but your solution is akin to trying to prevent murders by removing all the guns. The way to good security is to practice and learn good security, not to put in an airgap and be done with it. Someone who doesn't understand how to make an internet connected system secure is the same person who doesn't understand how to make it physically secure or how to do proper change management.

Fire-walling is not a pipe dream especially when you layer networks together two or three layers deep with proper DMZs and computers and firewalls which are configured to whitelist or only transmit data one way.

You say air-gapping is the only way? I will say that there's a great many plants and systems which you have just made unviable, unmanageable, and in some cases even inoperable as they rely on an external connection to someone located out of reach of a closed circuit fiber run. Think pumping station on a oil pipeline.

Re:i hope people with SCADA systems learned.

[Anonymous Coward]

Actually, they send hard drives. Next day air.
by the way, when I spend upwards of 750,000 on on of these setups, a "bugfix" is going to cost the manufacturer a trip out to my facility on his dime, with an ass chewin to boot.

And do not allow USB PORTS or other media to be BUILT into these systems. FTFY

Have any of you guys actually touched these systems?

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

When is an air gap not an air gap?

When it isn't.

Iran's nuclear plants had SCADA computers air gapped; but if someone could get in there with a USB or optical media to install malware, then it was not really air gapped. There are also some theoretical means of getting software onto air gapped systems (like compromising the hardware at design-time, etc). There are measures that can be taken, and these measures end up being extreme, and costly - and then we need to sit down and question how expensive TRULY secured systems are, and whether this saves any money over manual ops.

Re:i hope people with SCADA systems learned.

[Anonymous Coward]

One of the POINTS of SCADA is to be able to access the Supervision and Control from where you are.

The thing is that SCADA systems are bad at security, I get that (seeing how I work with them), and that the *best* solution would be to air-gap them. However if you have an unmanned power distribution site you want your operator to be able to access that site do you choose low cost internet and hope that proper VPN will save you or do you build your own private network?

When you need that financial rapport for the boardmeeting, do you VPN in from a secure PC to get it? Nice, but how do you get it out from the secure PC without risking the secure PC?

There are of course many ways to secure SCADA servers depending on what level of connectivity you need but stop repeating the "do NOT connect SCADA to the internet" as if it's a catch all solution.

Re:i hope people with SCADA systems learned.

[Anonymous Coward]

This can be done: either by disabling the USB ports, or installing and configuring the OS so that USB drivers can not be installed. (disable usbstor.inf, on windows).

This is a pretty typical posture for DIACAP - and while users will complain about the inconvenience - the fact is, that USB devices are a huge, and real threat, and they really DO need to be kept off of secured networks.

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

Yes we ran these systems before the internet, with many more employees and much slower response time.

Re: i hope people with SCADA systems learned.

[EETech1]

We use industrial VPN router / gateways that allow us to have our own certs and encryption keys. They have a digital input connected to the PLC that the customer turns on with a password protected maintenance screen on the HMI and / or a lockable key switch. The customer will call us, and we then calculate and give them the password to enable the VPN, they unlock it, and then we are able to remotely connect to the machine and service it.

Unless we have walked the customer through enabling remote access, the router is not connected to any network, and it will not respond to a ping or port scan. It has all the security of the network cable being unplugged, but we can activate it without having to lockout the machine and open the control cabinet.

These routers are very popular, and can also log access information, performance data, and error messages. There are even versions that can verify the PLC program, configuration, and databases to be sure it has not been (Stuxnetted) modified. If the firmware detects anything has been changed it will automatically reload the system from a self-signed backup contained in the router, send us an EMail with what was changed, and attach various logs.

Links:
www.mbconnectline.com/index.php/en/produkte/mbconnect24net/item/mbconnect24

http://www.ewon.biz/en/home.html

I have had factory training on these routers, and after seeing the features and interfaces, I've asked what operating system they run. Of course it's Linux, but the manufacturers do not release their source code! They would make an awesome little hackable home router / VPN / Fileserver, but they are locked down, and most have cryptographically signed firmware and bootloaders (ala Tivo) that refuse to run any modified code. Some of them will even erase themselves if they detect anything out of the ordinary going on.

I've tried to plead my case that as a buyer of that hardware I am entitled to the source code, and even mentioned that they should sell an unlocked and unsupported version to allow customers to integrate them into different applications. I always get the same answer. "These devices use OUR OWN PROPRIETARY LINUX, and we do not provide it to anyone for security reasons"

So now I am on the search (anyone???) for a BSD based (anyone???), or properly supported Linux version (anyone???) so I do not have to support the GPL violators. I'm tempted to notify RMS, I want to hack on one of these in the worst way:)

The routers have multiple VLANs that can be configured on a port by port basis because most of these systems are connected to (at least) the intranet nowadays, and it is nearly impossible for them not to be. The setpoint recipes, quality control, inventory, ordering, historical data, productivity, and monitoring of these machines is always done remotely, even if it is in the same plant.

You just configure the router to give the the device its own VPN / VLAN to the exact device it needs to connect to. Hardly anyone walks down on the factory floor to copy data with a memory stick anymore, it's too risky for the poor unpatched naked Windows installations being used. Some of them are configured to automatically connect once per hour / shift and dump their data to a central database and then disconnect, but most of them are always connected to various supervisory control systems. Most of the HMIs and data access servers are MS Windows NT / 2K / XP based, and rely heavily on MSSQL, OLE, and crappy vendor supplied closed source drivers to interface Windows to the various Industrial Networks.

Usually these old unpatched Windows machines just connect to the various databases with a specific VLAN for each one, and then the properly patched and AV'd supervisory systems (now mostly Windows 7) make their own connections to the databases and push or pull the necessary data from them without ever connecting directly to the vulnerable unpatched PCs that are actually controlling the process.

No updates or

Re: i hope people with SCADA systems learned.

[Anonymous Coward]

The air gap is not the solution. Proper isolation, firewalling and virus/malware is.

+ radiowave sheilding to stop inadvertent Wifi Access to "air-gapped" systems.

"One NSA slide revealed how a combination of the SOMBERKNAVE, VALIDATOR and OLYMPUS exploits can be used to extract data from Windows XP PCs that are “air-gapped”, i.e. not connected to any public networks. After taking control of a nearby wireless access point, SOMBERKNAVE is able to connect to a machine even if its embedded 802.11 device (WiFi cards are standard fare in business and consumer PCs) has been disabled."

Re: i hope people with SCADA systems learned.

[omglolbah]

They dont need internet access, but it usually goes something like this...

Secure network ---firewall--- plant network ---firewall--- corporate network ---firewall --- internet!

Becaaause....

Plant network people need access to the secure network.. so they link em
Corporate network epople need access to the plant network... so they link em..
Corporate ALSO needs internet for obvious reasons and link to that...

Technical people and security people scream bloody murder at the security implications, but are overruled for financial reasons and we end up with a hodgepodge of connections possible... sigh

Re:i hope people with SCADA systems learned.

[omglolbah]

We bring new software on usb sticks, get em scanned and then inserted into clients on the secure network.
Hell, we even bring windows updates in .exe form this way...

And this is for oil rig control systems.

Practicality and immediate cost seems to win everywhere sadly...

These issues have been flagged for 10 years

[msobkow]

These issues have been flagged for roughly a decade. I have ZERO SYMPATHY for anyone who gets taken over.

Re:These issues have been flagged for 10 years

[Anonymous Coward]

It's not about sympathy, it's about the effective destruction of our entire infrastructure without dropping a single bomb. The first sign that China or Russia is at war with us will be all our utilities and factories going dark. This is everyone's concern.

Re:These issues have been flagged for 10 years

[Billly Gates]

These issues have been flagged for roughly a decade. I have ZERO SYMPATHY for anyone who gets taken over.

MSOBKOW this is your boss.

What do you mean it is a security risk to put this on the internet? Everyone else has no problem doing this and I never heard of anyone being hacked. Like a billion dollar company would ever design such a thing when an internet connection is required to stay activated. Are you telling me that firewall you said we needed doesn't make is impenetrable?! Why can't you secure it? Do I need to hire someone who will?

Re:These issues have been flagged for 10 years

[Endloser]

I too have zero sympathy for those who get taken over. But the citizens it puts at risk are a different story.

Re:These issues have been flagged for 10 years

[tlambert]

Why can't you secure it? Do I need to hire someone who will?

Yes. Yes you do. And when they fail, you should know that my contract rate for you, with the negative discount, is $500/hour, in hour increments.

Re:These issues have been flagged for 10 years

[M0HCN]

Most of these things can be taken to at least a semi manual mode of operation (It might require more people out on the floor manually tweaking things) but I suspect that most of these systems are actually simple enough on a local level that a good tech team with screwdrivers and set of schematics can fairly quickly get the PLCs out of circuit and some switches and pots and meters wired in (Most systems have switches on things like pumps and switchgear labelled along the lines of auto-off-manual already), worst case a laptop, a can card and use canoe or canalyser to talk to the valves and inverters directly.

Doing this does of course then depend upon having enough process engineers who really understand the plant to be able to run it with a board full of switches (and few if any interlocks) rather then letting the computer handle the details, this is probably the real issue as keeping such people on staff is expensive and is the reason you went heavily computerised in the first place. Getting management signoff could also be a problem, boards with billion pound assets like to hire consultants before letting the local on site guy fiddle with the flow rates and heat levels on the refinery heavy oil cracker without any interlocks.

There are of course systems that need the computer support, but even things like power stations (yes, even the nuclear ones) actually do not strictly need it, for all that bringing a set on line without it may require getting some people out of retirement to demonstrate the trick to it, and running without the computers would probably require emergency permission to violate all sorts of regs.

Damaging? Of course.
Disaster? Only if you cannot find the people who can deal with the loss of PLC support or if the attack causes the PLCs to damage the plant before the humans can step in.

The other major issue here is that while the scada controls may be more or less homogenous (Lots off Simens stuff out there) the systems they are controlling are anything but so a broad attack would probably be able to take the automation off line or change set points at random, but you could not easily write an attack to say cause the grid frequency to try to rise to 400hz, because there are far too many variations in the physical connections between the PLCs and the rest of the plants out there.

The scary thought is that it is not an attack on the SCADA running the pumps and power that would be really damaging so much as one of the machines running say the stock exchanges, repairs to some damaged pipes, boilers and transformers might take a few years and cost a few billion, repairs to the confidence in the financial system after some banker has diddled the risk models to ignore the sub prime lending risks.......

Regards, Dan.

Re:These issues have been flagged for 10 years

[ebno-10db]

repairs to the confidence in the financial system after some banker has diddled the risk models to ignore the sub prime lending risks.......

That confidence was destroyed by the financial system itself several years ago. Considering what the financial scam artists got away with, I don't see how hackers could make it any worse. By contrast, water and power actually work.

Re:These issues have been flagged for 10 years

[gmuslera]

If you use jelly as the basement of your house is your fault that the house is unstable. Putting and approving to put critical infrastructure directly accesible on the open internet, that can have present or future vulnerabilities is bordering criminal behaviour. That people should be the first on the line to be jailed, and now, not when something bad happens.

And remember, the ones that started with big scale "war" has been the US. Don't start a war of breaking glasses if your entire house is made of (specially fragile) glasses.

Re:These issues have been flagged for 10 years

[sumdumass]

Putting and approving to put critical infrastructure directly accesible on the open internet, that can have present or future vulnerabilities is bordering criminal behaviour.

Lets stop being overly dramatic and think about reality. When a lot of these systems were placed in the open, the entire thought of exploiting them was pretty much non existent. It's like the early Microsoft security models that completely missed the communications implications of the internet and the reason why after windows 98, they started- rather unsuccessfully I might add, working on improving the security. Windows XP started getting some of it right with a built in firewall but still had blaring flaws in IE, allowing root log in as a primary desktop and in some cases requiring it for popular software to function correctly, and other portions of it.

The bottom line is that nothing involved with how we got where we are is borderline criminal unless you consider not knowing the future to be criminal. Now that we do know, we have to make a competent cost effective plan to address and limit the implications and bringing the information about the security risks and potentials for exploitation to the front is the start of that plan. If everything was fixed today, in 20 years, something else will crop up and we will be having the same discussions about things that weren't even envisioned at the time we implemented the changes to secure the older systems.

Re:These issues have been flagged for 10 years

[ThreeKelvin]

I ran a part of the process plant by hand during the commisioning phase for the last automation project I was on. Working together with an operator I could barely keep up with one fifth of full capacity for four hours and we were both completely drained afterwards.

The complexity of modern process plants is mind-bogling to people who haven't seen them - and even when they've seen them they don't understand that all the valves, pumps, heat exchangers, etc., around them are doing a finely choregraphied balet behind the scenes. The manpower needed for running a process plant by hand is in the neighborhood of 10-20 times that of running an automated plant, and even then the throughput will be less and the quality of the resulting product lower.

Re:These issues have been flagged for 10 years

[gmuslera]

Since the 90's ive seen constantly scanned every internet connection for open ports, vulnerabilities, and common software with flaws. And when something had a known (may not by you, but by the exploiter) vulnerability, and was interesting enough (profit, fun, proof of concept, following political agenda or whatever) it was exploited. It is not the 90's anymore, the whole internet can be scanned in 45 minutes (and exists scans ready to use if you don't want to spend any time), if something can be used, it will. If you put the key to operate a critical system on a busy street or a shopping mall and a kid turns it causing chaos, it was nenligence from your part or from the one that ordered you to do so.

Too bad NSA is too busy checking what can be exploited by them (and planting backdoors every time they can, specially in foreing critical systems) instead of warning and fixing what can be exploited by others. Can't blame others if do the same as them.

Re:These issues have been flagged for 10 years

[sumdumass]

Interesting you mention a kid causing chaos. Ever hear of a molly guard and how it got it's name?

Negligence is not criminal though. That was the point of my comment. Negligence that happened in the past without advanced knowledge of the future cannot be criminal. It can be short sighted, stupid, clumsy and a number of other things, but not criminal. Many of these exposed systems were developed before the 90's and switched to using the internet during the 90's to save costs. Many of these systems were put into use using industry standards which did not catch up to the level of knowledge about the put falls you have today. I still see companies with sensitive customer information using WIFI with WPA type encryption because it was industry standard when implemented.

After the Snowden debacle, it isn't entirely positive that many of the other secure industry standard are exactly secure any more either. In 10 years, we will be having the same discussion about companies who continue using products that just work because they just work before realizing that everything about them is easily exploitable.

As for the NSA, I can blame others. To take the position otherwise would seem to validate the NSA doing what they did. I don't want to be in that position. But in keeping things realistic, we cannot blame companies who have entire switching hardware and gateways with NSA or other back doors in them because the multi million dollar investment was industry standard at the time of install and we just now find there might be issues with it. Eventually, through discussion about the dangers and perhaps a few incidents, it will be replaced with more industry standard equipment and procedures and we will end up having the same discussions in the future.

Re:These issues have been flagged for 10 years

[lennier]

When a lot of these systems were placed in the open, the entire thought of exploiting them was pretty much non existent.

Only "non-existent" to people who weren't thinking and weren't paying attention to the literature. There had been a LOT of academic warnings back to the 1970s about the potential security problems of interconnected networks. Heck, the entire genre of cyberpunk science fiction in the 1980s - Neuromancer was 1984 - didn't come out of thin are but was based around the then-current academic discussions of the security problems of the early Internet. The first IBM PC virus was 1986, the Morris Worm was 1988, pretty late in the game.

Yes, it wasn't headline gossip-reality-show news like it is today - but industrial control designers? In the 1990s? Nope, there's no excuse. They were definitely in a position to know, should they have bothered to care.

Re:These issues have been flagged for 10 years

[ebno-10db]

Judging by your "ThreeKelvin" name, it must have been a liquid helium plant.

Re:These issues have been flagged for 10 years

[Anonymous Coward]

It would probably be called SuxNet or BlowbacksABitch

Re:These issues have been flagged for 10 years

[thegarbz]

Yes if only technology hasn't changed at all in 10 years.

The reality is in the changing world whatever you were told 10 years ago probably doesn't apply to security anymore.

Re:These issues have been flagged for 10 years

[msobkow]

SCADA systems have been notoriously vulnerable for 10 years. If they STILL haven't been fixed, then it has nothing to do with what I was told, but a complete and utter FAILURE on the part of the vendor and it's customers to address the KNOWN problems. Ten years is a long, long time to not get it right...

Re:These issues have been flagged for 10 years

[thegarbz]

No my point was that the issues that were a problem 10 years ago are not the issues that are a problem today. The attack vectors were different. The security issues were different.

The outcome is the same (insecure system) but 10 years ago I doubt something like wireless control of a process plant was even on the radar, yet nowadays that's something that needs to be designed by someone competent. 10 years ago a PC in a control system (which was likely 20 years old) exposed PS/2 connectors which supported nothing but the mouse, yet today they expose USB sockets which need to be physically and virtually locked down or your uranium enrichment plant may start playing up.

Unfortunately these systems are usually designed by those who just got around to fixing the problem of 10 years ago.

Re:These issues have been flagged for 10 years

[Anonymous Coward]

Gross negligence is one form of criminal intent. If it's not just the case thay you should have known but that you were wreckless in not knowing, it can create criminal liability. However the corporate veil is almost never broken for this.

Re:These issues have been flagged for 10 years

[omglolbah]

Especially if you have a separate emergency shutdown (ESD) system that is fully up and running...

Goes something like this:

"Shit, close that valve we're getting an over-pressure on line 3"
"Valve closed"
"Bah, too slow, we're in a blowdown..."

Aaaand you have 300 feet of flame coming from the flare. Fun times for all :p

Some of them expose to the internet via VNC...

[M0HCN]

At 30C3 someone ran a portscan on the VNC port of the entire IPv4 internet, with 'interesting' results, highlights of which included a swimming pool chemical dosing control system, various power generation and control systems, building environmental control systems, air handlers, all sorts of wild and whacky things, some of them lacking in even the rudiments of passwords never mind proper crypto....

The best one looked to me like a medium voltage distribution cabinet where the setpoints on the overload trips looked like they could be reconfigured from the internet!

Ahh the things you can do in reasonable time with a 100Gb/s of bandwidth, the rsulting slides at the closing event (which is where I ran across it) were very, very scary.

SCADA on the internet is a really, really bad thing.

73 M0HCN. :wq

Re:Some of them expose to the internet via VNC...

[gmuslera]

You can scan the entire internet in less than an hour by now. And there are databases of open ports on all of it already if you want to save that hour. If is critical, should not be even visible on internet.

Re:Some of them expose to the internet via VNC...

[satuon]

What's interesting is, why are news of anyone actually exploiting those vulnerabilities so rare? It seems even though the vulnerabilities are there, nobody is exploiting them.

Re:Some of them expose to the internet via VNC...

[doesnothingwell]

Some ot them are not real. I sometimes start a virtual machine with Vnc wide open on 5800 and use a DOD emblem for wallpaper.

I've found hackers trying ports 5802 and when I tracert them I get a weird 2900ms delay leaving the last US hop at San Diego headed to the Orient.

Re:Some of them expose to the internet via VNC...

[M0HCN]

Yea honeypots can be amusing to run sometimes.
The scary thing is that I suspect that some of them are real, and for a state actor the honeypots are not a big deal!

Exploits are rare for three reasons, firstly there is little profit to be had as a non state actor, no obvious oppertunity to profit in a way that doesn't attract a drone strike, secondly to actually do anything really interesting with these systems requires a level of familiarity with the tools and languages which is rare enough that these systems are seldom the low hanging fruit for script kiddies, thirdly nobody is going to fess up in public to having had their chemical plant hacked, the regulators response would be a nightmare.

Regards, Dan.

Re:Some of them expose to the internet via VNC...

[Anonymous Coward]

No ones figures out how to make money off it yet, without getting caught?

Re:Some of them expose to the internet via VNC...

[thegarbz]

SCADA on the internet is a really, really bad thing.

73 M0HCN. :wq

No. Unsecured SCADA on the internet is a really bad thing. Air-gapping is not the answer for many of these systems which are not run locally but are part of a wider network. Firewall, VPN, DMZ, and hiring a bloody network engineer instead of some in house electrical PLC guy is the answer.

In some cases things need to be configured over the internet, that does not mean that it should be possible to do it without any security (and no a VNC password isn't security I'll definitely give you that much).

Re:Some of them expose to the internet via VNC...

[omglolbah]

Getting caught with your pants down or a major shutdown due to a breach could put you in legal trouble.

So they're usually kept hushed the hell down by most companies.

Just wait for what comes next

[Gim Tom]

SCADA systems are bad enough, but the push to "THE INTERNET OF EVERYTHING" should make it far more interesting for everyone.

I remember, far back in the late 1960s, when a popular DJ on a local radio station joked for everyone on a particular Interstate leading into the city to "CHANGE LANES". I was on that road and an amazing number of people did. With TIOE the cars can just do the lane change without having to tell the drivers to do it! Of course most of the drivers did make sure that the lane they were moving to had room for them. I doubt that will be the case next time.

Re:Just wait for what comes next

[maxwell demon]

Indeed, thinking of the smart grid, you could probably get the grid down by issuing a command to sufficiently many household appliances to switch on at the very same time. Those will be even less protected than the power stations, because "who would want to attack my dishwasher?"

Re:Just wait for what comes next

[Gim Tom]

Good point. The soft undefended target is the ripe target.

Another Tao of math: For Electrical Engineers imaginary numbers are real.

Re:Just wait for what comes next

[Ol Olsoc]

Indeed, thinking of the smart grid, you could probably get the grid down by issuing a command to sufficiently many household appliances to switch on at the very same time. Those will be even less protected than the power stations, because "who would want to attack my dishwasher?"

New Jersey's Governor will be able to more tightly focus his retribution efforts. Instead of old school shutting down lanes of traffic, he'll be able to turn off the electricity to every registered Democrat.

Re:These systems are a product liability nightmare

[Anonymous Coward]

True - However, most (I would hope, ours is at least) are behind a hardware firewall / VPN with pretty restrictive rules (no connecting backwards from the remote system into the central office, for example). That means that, barring some unknown remote exploit in the VPN box, the big bad 'internet' can't contact the unpatched systems..

SCADA on one side

[Anonymous Coward]

Embedded XP running all those banking ATMs on the other.

2014 will likely prove very interesting for the "Internet of things".

Why the hell

[no-body]

are those systems connected to the Internet?

Plain stupidity or folks managing those don't know what this Internet stuff is?

Re:Why the hell

[M0HCN]

Because actually it is really very operationally useful, and USEFUL in normal use trumps security EVERY SINGLE TIME.

Consider someting simple like a public building heating control system, this is probably a modest PLC from the usual suspects, now if I am the poor sap in charge of the building systems (Nightmare, been there, done that), and the thing alarms at say 2100 on my day off, I have a choice:
I can go in and clear the (often but not always) unimportant problem, takes me an hour to get there and I was on my way in to see a show when it went off, or I can log in over the internet from my phone, see that the problem is that the number two AHU intake filter is showing high backpressure, clear the alarm and make a mental note to replace the filter next time I am in.
Same thing if the office phone up wanting me to change the setpoint on the air in the art gallery because some conceptual art is made of butter and is tending to melt (I kid you not, really happened).

Remote access to these systems is USEFUL, and nobody considers security until it bites them.

Further plant engineers still think in terms of 'ladder logic' which is essentially logic consisting conceptually of relays and coils and the connections between them, they are not by and large networking folk, and plugging the plc into a port on the external side of the firewall makes everything work where plugging it in inside the firewall makes the remote control not work properly....

Regards, Dan.

Re: Why the hell

[Anonymous Coward]

Systems Control AND Data Aquisition. How to acquire data from an air gapped computer with all the ports plugged with epoxy? I guess you gotta stare at the monitor.

Re:Why the hell

[ebno-10db]

Point taken, but I think the appropriate security/convenience tradeoff needs to be assessed for different situations. Messing up a building's HVAC is going to wreak a lot less havoc that messing up water, power or sewage systems.

I also have a question. How is connection between PLC's to the Internet handled for such things? Is the PLC directly connected (probably a very bad idea) or is it through a computer that can be used as a firewall?

Re:Why the hell

[M0HCN]

Security/convinience tradeoff? You try explaining that to a building contractor sometime!

As to the interfacing, it depends, sometimes it is a direct link to the plc, sometimes the plc talks CAN or RS485 or such to a windows xp box which runs a web gateway... I personally think the first option is likely more secure, especially when the machine in the corner of the plant room is found by the local security guard to be a good place to browse porn sites and download videos on the night shift (It happened, and I bet we were not the first, I found out when we got a phone call from the ISP about something on our network abusing port 25 outbound).

Generally security is not mentioned in the contracts for the installation of this stuff, and is at best an afterthought by non specialist developers, the effectiveness of this is left as an excersize for the reader.

Note also that the support contract with the installer often specifies that no software is to be installed on the user control computer except by their engineers (Who might come out once a year and then forget to do it) and this includes updates for security fixes.

73 Dan.

Re:Why the hell

[Ol Olsoc]

Point taken, but I think the appropriate security/convenience tradeoff needs to be assessed for different situations. Messing up a building's HVAC is going to wreak a lot less havoc that messing up water, power or sewage systems

True. ALthough there might be some business reasons to do so. Imagine making your competitor's HVAC systems go down during important meetings, or in the dead of winter before a big deadline. ANd considering that we live in a country where American on American attacks are political gold: http://www.latimes.com/nation/la-na-christie-bully-20140111,0,3128420.story#axzz2qD3vqu1x

No, I think this is an untapped market of Screwing With Your Competition.

Re: Why the hell

[hierofalcon]

Supervisory control and data acquisition.

The air-gapped computer isn't "plugged with epoxy" The data acquisition part can come from many sources, but usually an internal network on the side being monitored - either the normal private network you think of today via copper or fiber with private IP space, or another telemetry data link (cell phone, radio and repeaters or the like used as the transport mechanism). In some cases it directly feeds into cards on the back plane of the computer doing the SCADA operation but this isn't seen as often anymore.

Regardless, the number of computers that are truly air-gapped from the real world is growing smaller by the day because corporate doesn't want that. The whole "just in time" philosophy also applies to SCADA systems. The want their production figures, product on hand, supply levels, maintenance reports and all fed back to the corporate mainframes ASAP so they can do a better job of pricing and hedging and increase the profit margin by .001%.

There are also fewer small operators. Too many companies have been bought out by the big boys who have experts, but not enough experts to be everywhere at once. So SCADA is used to bring the data to the experts so they can maximize the production of every place.

While they wouldn't hire every expert available as that would decrease their profit, they would hire more if they could. But the reality is that there are a lot of experts who are retiring and there aren't enough new graduates in the required fields to make up for the drop. So the analysis gets centralized or outsourced or both. The same SCADA plant that used to present the data to local experts years ago, may now be sending the data around the world to where the experts (and cheaper experts at that) now live.

It's going to be nasty when it all comes crashing down.

unlocked doors

[markhahn]

These systems are the moral equivalent of leaving your door not just unlocked but ajar. It doesn't change the morality of anyone trespassing to steal or destroy, but it does make the owner much more culpable. We do not face a threat to our cyber-infrastructure, but rather have irresponsibly left the infrastructure unprotected, and should not be surprised that people of varying motives might take advantage.

We do not need a cyber-infrastructure police force, unless they're actually tiger teams who publicly shame the idiots who leave their systems unprotected...

The Internet of Things

[RotateLeftByte]

could someone a lot wiser than me please explain why we need to connect everything and anything to the internet?
I expect the hackers are rubbing their hands with glee at the prospect of being able to hack all sorts of things. Imagine all the havoc they could cause by making all the freezers in a country suddenly defrost?

Frankly, I think this drive to connect everything is totally misguided.

 

Re:The Internet of Things

[LoRdTAW]

Cost.

Why pay a person to stay on site or make periodic visits to maintain equipment or change settings when a few people can do it remotely? It does sound convenient but it opens a whole can of worms as any one anywhere on earth can potentially wreak havoc on your low cost maintenance systems.

Re:The Internet of Things

[fisted]

Thats typically because fully air-gapped machines are terribly useful, unless they inherently do not need to communicate for the task they are doing. for example .... uh.

Re:The Internet of Things

[fisted]

eh and that should ofc read 'terribly useless'

Re:The Internet of Things

[Anonymous Coward]

Getting design files, software updates, etc. onto production machinery can probably be done fairly easy with USB sticks (though they might also be infected). But MRP wants live updates of widgets-made, and the MRP system is accessed by people with a poor grasp of network security, such as accountants.

Re:The Internet of Things

[ebno-10db]

That's not the whole answer. First, there were remotely monitored and controlled systems before the Internet (though I'm not sure how the various links were implemented). Second, I suspect that the convenience, or perceived convenience, may be as important as cost. Lastly, anything you can't connect to the Internet seems outdated (whether or not the connection is a good idea).

Re:The Internet of Things

[ebno-10db]

unless they inherently do not need to communicate for the task

Unless they inherently do not need to communicate beyond the local network.

for example .... uh

Most SCADA systems. There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.

Re:The Internet of Things

[Lumpy]

It is trivial to make a "one way, unhackable" ethernet connection to export data to a unsafe network device.

you have a machine on the SCADA network with TWO network cards. One connects to another PC on the insecure network via an ethernet cable with ONLY the TX wires connected. no RX lines. set both to a static IP and then UDP broadcast your information from the secure PC to the insecure one.

There is no hacker or security expert on this planet that can hack that connection and gain access to the SCADA system. Unless they found a way around physics or can teleport things with their mind.

http://www.stearns.org/doc/one-way-ethernet-cable.html

The problem is most places refuse to hire educated IT staff with experience in security. They want low cost MCSE holders that can barely do their job at the lowest cost possible.

If updates to SCADA software are needed, "most are not in reality" you use write once media such as a DVD or BluRay created on a machine that has nothing to do with the SCADA system and based on an OS that is drastically different to further reduce the chances of homogenous OS infection vectors. If it's important, then the files are inspected byte by byte on a security computer designed to look for infections and injection. then after full and careful inspection you apply the updates.

THIS is how you run a critical system SCADA network. and 99% of them out there are not ran this way as the people in charge of it have zero education in security let alone networking and IT.

Re:The Internet of Things

[Lumpy]

Not Cost.

Profit.

Please do not confuse the two as Profit has a higher driving force than Cost does.

Re:The Internet of Things

[fisted]

Administration of said machine is a staff-intensive mess then.

Re:The Internet of Things

[Anonymous Coward]

Any Sysadmin or Programmer with a hint of intelligence can smell the BS on "The Internet of Things" from 10 miles away, downwind, on a rainy day.

"How does connecting my toaster to the internet help my business?"

Good business managers can smell the BS, too.

But like any good automotive car salesmen, you need to sell something to make a living. The way you do that in the IT industry is you come up with some grand general pie in the sky bullshit and see how many skin-eating blood-sucking fly's you can attract to give you venture capital, then go off and see who you can make feel insecure enough or sucker into buying your bullshit.

In the 90's it was websites and E-everything commerce. In the 2000's it was wearable computing, tablet computing, everything through a phone line remotely, wireless internet, "Thought Leadership", "Information Silo's", etc. In the 2010's now it's the "cloud" and the "internet of things".

And when people call them on the bullshit, they start making the lies bigger, because the bigger the lie the more people tend to believe it, at least at first glance. And once you have them debating you can pick off the suckers.

Change control on SCADA is usually a nightmare waiting to happen; you back them up, put safety protocols in place so when they error out nobody dies and the damage is minimal, and run it. You run those systems on a separate network, and if they need to be hooked into automated systems, you double-encrypt the connection (stack the devices at either end) and use a 1-way proxy.

The companies that hook SCADA to the internet deserve it when some 16 year old kid goes and kills someone with a hot pot of iron or shuts their systems down and blows out power relays all over the plant for shits and giggles. They really, really do; if you can't even be bothered to put up proper signage you deserve it.

Re:The Internet of Things

[RotateLeftByte]

Have you never heard of Firewalls and VPN's?
As part of my job I login to sites all over the world via VPN (actually two VPN's). None of the systems I connect to are visible on the internet. Good job too.
Putting all sorts of devices directly on the Internet as all those IPV6 advocates are so fond of reminding us that there is plenty of address space to do it is just stupid and will eventually cost a lot of lives. Perhaps it will take a major catastrophy to wake people up to the dangers of doing this.

Having been involved with computer networks since 1974 I feel that some of these 'connect everything' advocates need to be taken outside and given a good seeing too in the hope of making them see sense.

Re:The Internet of Things

[ebno-10db]

Administration of said machine is a staff-intensive mess then.

I bet Iran wishes they'd taken that approach with their enrichment centrifuges :)

I'm old enough to remember when nobody used the Internet for remote administration. While less convenient, and slightly more expensive, it's not that big of a deal for SCADA. You have to remember that most of the boxes in a SCADA system are not like say, web servers, which are computers talking to other computers and doing only computerish stuff. SCADA controls actual physical equipment, that can't be be remotely monitored or manipulated in the same way as "pure" computers. You want somebody on-site doing this anyway, if for no other reason than to verify that the upgrade is working properly. It may also be necessary to turn off equipment that it controls, or switch to manual control or something, in order to do the upgrade.

Re:The Internet of Things

[DarkOx]

Tell me, do you split tunnel? if not do you always check the routing table before and after you connect to those VPNs? Because despite what you think those machine might very well be visible on the internet. Just takes the right malware running on your laptop.

Fundamentally you are mixing a high security domain ( the SCADA network ) with you machine which has been in a low security domain and is in a questionable security state. Even if we want to believe the VPN isolation it self is always perfect and nothing ever users you machine as a live pivot point there is still the possibility you introduce a STUXNET like worm, that has established a beach head on your machine and will hop onto any network you connect to from there. STUXNET needed no command and control to do its work.

That said I understand the need to do what you do is a very real one. I also no all the vendors of this stuff who really ought to be condemned to death like to depend on things like Teamviewer for support. The only really good enough solution to your use case I can see goes like this:

you vpnlayer internet vpnlayer -> (note the direcationality here no connections outbound allowed) firewall allowing only desktop sharing app, -> application layer firewall that makes sure only desktop sharing features are used, shuts down the connection if file transfer features show up DMZ with access server you use remotely -> firewall with minimum required ports open -> NAT translation from the DMZ to an IP on the SCADA subnet -> IPS/IDS -> SCADA network hosts, all configured with no gateway.

Re:The Internet of Things

[pcwhalen]

Is this why my lights are off? Got an email that said unless I forked over some serious Bitcoin, my X10 system would be hacked.

That and the coffee maker won't turn on, the lights are out and the TV ... damn, just a blown fuse.

Never mind.

Re:The Internet of Things

[cusco]

Pre-Internet the control systems often had their own modem and serial ports. Frequently one dialed into a serial port server, issuing commands at a prompt directly to the equipment. (A lot of the old equipment still have those RS-485/422 serial ports sticking out and only the oldest service techs know how to use them.) There were also telnet servers that would relay commands across a serial connection to the telnet server on the equipment. Starting with NT 3.51 you could set up RAS connections to standalone PCs, drop a modem on the box, and have a fairly secure terminal server (using VNC or something until RDP connections became available in Server 2000). I think you could do the same to OS/2 machines, Unix servers varied depending on the flavor and version.

Re:The Internet of Things

[thegarbz]

could someone a lot wiser than me please explain why we need to connect everything and anything to the internet?

Cost, operational expertise, remote management. There are plenty of cases. Think of an oil pipeline with pumping stations along it. Do you employ one person to sit and watch the pump 24x7? Or do you wire this all back to a control room and employ one person to watch all pumps. Now consider if your pipeline is 100s of km long, you can no longer run a simple network out.

In terms of operational expertise you can also control entire plants from central locations. Some companies do just this. They build identical plants geographically disperse across separate countries. They have a local support team that does maintenance, but actual running operations are handed over to a central office sometimes in another country. This has the benefit of being able to concentrate experience (training 10 operators in one location is easier and more effective than training 10 operators in 10 different locations). It has the benefit of removing people from blast areas (getting staff out of the danger zone). And it makes some places viable (think tiny remote gas plants where you don't want or need to build living quarters).

Obviously internet does not mean internet. None of this should be accessible to anyone outside if you have even a semi-competent network engineer, but the point is the world does use the internet for control of infrastructure. That's not going away.

Re:The Internet of Things

[Anonymous Coward]

Iran had air-gap.

The programming devices where silently infected by a thumbdrive. Later when the programming device was connected to the local network the worm spread.

Re:The Internet of Things

[Anonymous Coward]

Sorta. Some of these guys make 100+ an hour. If you can put the guy on site with the right equipment to fix things lowers your cost center cost. Or you can have that same 100+ hour guy go out and sit on site and take a reading off an a just as antique remote control reader. Phone the #'s in. Wait for someone to figure out what it is. Then wait another 4+ hours for the right part to show up hopefully paid for with another 40-80 dollar lower level employee delivery truck guy.

So you could pay a guy 500-800 dollars to go out and replace a part in 4-8 hours. Or have him show with probably the correct part in hand and is onto the next site in 20 mins. You cost goes way down. You can service your customers in mins instead of days. You do not tie up your customers time with them having to baby sit some dude all day.

Putting this on the wide whooly internet though (shudder). It should be in its own network with only allowed access and proper radius setup. Its not 'easy' to setup. But it is not exactly rocket science either. You put in the control layers and the bad security on the other end is not so 'keep me up in the middle of the night' either.

We on the software side though are quite used to just re-patch it out. These guys are a whole different world. They put something in they *DO NOT TOUCH IT*. It better run good for 10+ years. Putting a union guy on site costs mucho $'s.

There are *other* costs involved than just money. Such as customer satisfaction. Time on your end and in effect ability to support your customers. To name but 2.

Re:The Internet of Things

[Lumpy]

the "programming device" was a usb drive that was never ever looked at to be sure that it was clean.
yes even a low level non programmer can spot even an unknown worm on a usb drive by simply looking at the whole drive on a hex editor. Look at all this extra bytes, UNCLEAN, REJECT.

Re:The Internet of Things

[Lumpy]

Staff intensive as in 1 guy that has expert level knowledge and can document processes? yes. and I understand how this is unachievable in corporate america. Hiring experts or even competency is not a part of the profit plan.

Re:The Internet of Things

[fisted]

Staff intensive as in 1 guy per such machine that has expert

It does not scale.

Furthermore, depending on the importance of the machine, you might have to have someone around /all/ the time instead of just 9-5, ending you with needing at least 3 guys plus replacement

Re:The Internet of Things

[Lumpy]

I love how you add words that absolutely nobody said to prop up your straw man.

  You are simply reinforcing my statement that most IT staff are horribly under skilled in IT to begin with. As an IT person in 2014 if you can not maintain Windows, OSX and linux, you are corporate dead weight.

Even a fresh MCSE holder could maintain the setup I describe with proper documentation. And those guys are the bottom gutter of the IT world.

Note: it scales perfectly. the largest company in america has 1000 SCADA operated facilities. if that 1 person can not maintain all 1000 of the secure data gateways part time along with his other duties, then they need to hire someone that has a level of competency in Information Systems and Security.

Re:The Internet of Things

[fisted]

I love how you add words that absolutely nobody said to prop up your straw man.

Erm that was supposed to be a FTFY kind of reply, if you couldn't tell from the emphasis. It's not a straw-man, it's pointing out the flaw in your plan.

The rest of your comment doesn't really make sense to me, but I lost it at ``MCSE'' anyway.
 
Spare the reply.

Re:These systems are a product liability nightmare

[Anonymous Coward]

Updating them breaks things. Not updating them breaks things.

Re:These systems are a product liability nightmare

[Anonymous Coward]

The best thousand+ ton machinery I've seen, were running haskell code on the latest linux kernel. So cool and up to date.

Re:These systems are a product liability nightmare

[Z00L00K]

In that case I wouldn't call it a zero day vulnerability, I would call it vulnerability due to incompetence.

Hack the systems and make them go down permanently by a hard disk low level format or corresponding. That would raise the security awareness more than a slashdot article.

Only case to have an unpatched server is when you are running it standalone with no possibility to install anything new on it without opening a padlock.

The scary part

[gmuslera]

people and companies with big salaries and/or contracts still putting critical systems on the open internet. And that will keep their salaries, contracts and continuing to do so even after this is exploited.

Re:The scary part

[Lumpy]

It's because they hire management that are dumb as boxes of rocks or a small salad bar. Educated managers are not wanted, only ones that can schmooze.

article lies

[Anonymous Coward]

"researchers" are not hackers.....like it claims what a crock a shit ....
time to ask the nsa to stop pretending and fuck the hell off

No thanks

[PPH]

I'm already creeped out by how much a Nest Thermostat looks like HAL 9000.

DUH.

[Lumpy]

Almost ALL of us that have had to deal with SCADA knew this was possible. Most of the time because incredibly stupid managers DEMAND the systems be accessible from the internet.

SCADA systems need to be airgapped completely from any network other than their own. Boo Hoo to the company that needs to buy a second set of computers for the employees to get email on. the SCADA computers are to be used ONLY for SCADA systems.

100% of the security failures lie at the feet of the managers of these facilities. Until we start beating them with sacks of doorknobs nothing will change. and yes, the SCADA infection via usb drives are the fault of management. allowing the use of USB or any other device that has not been secured and low level formatted before use on a known clean machine is the fault of management.

All USB ports should be disconnected or physically inaccessible via lock and key to users.

Re:DUH.

[bill_mcgonigle]

Most of the time because incredibly stupid managers DEMAND the systems be accessible from the internet.

How does this not drive their insurance premiums through the roof? It should, and it's not, so something is broken in the process.

Do they have government protection from liability?

Re:DUH.

[zippthorne]

Why can't they do it the way that satellites do - all control operations are sent encrypted.

Re:DUH.

[dkf]

Why can't they do it the way that satellites do - all control operations are sent encrypted.

Because the SCADA vendor probably had encryption as an option that you had to pay extra for, and management wanted to chisel another few bucks off the setup costs.

Re:DUH.

[Anonymous Coward]

Where I just got hire, has a SCADA nightmare. There are 3 servers that connect to the machines and at least 50 terminals in the plant being used as control stations. All of them have internet access, all of them have a VNC service running that is not protected, they all use an unpatched (for the past 6 years), and in some cases unlicensed, version of windows XP. The self appointed 'controls' guy even set himself up WIFI points all around the plant so he can access the system from his laptop (an unpatched, windows XP laptop which is frequentlyt used to torrent stuff). Given the nature of the machines I turn EVERYTHING off before I go making any repairs because I have no idea if some kid in russia is looking at the control screen for the pendulum saw and watching the CCTV camera waiting for me to get just far enough inside.
 
Yes, there is really a pendulum saw, its a 4 foot circular saw that swings through material to cut it to length. The engineers who designed this place liked the Final Destination movies.

Re:DUH.

[Jeremi]

Why can't they do it the way that satellites do - all control operations are sent encrypted.

Or put in a data diode -- insecure machines (including the entire Internet if that's what you want) can monitor the system, but only a secure/air-gapped machine can send data to the SCADA system.

Re:DUH.

[cusco]

Do you think there is anyone in the entire insurance industry that has a clue? Having done physical security for a number of insurance company clients, as fare as I can tell the insurance industry is where IT talents go to die.

Re:DUH.

[thegarbz]

Almost ALL of us that have had to deal with SCADA knew this was possible. Most of the time because incredibly stupid managers DEMAND the systems be accessible from the internet.

SCADA systems need to be airgapped completely from any network other than their own. Boo Hoo to the company that needs to buy a second set of computers for the employees to get email on. the SCADA computers are to be used ONLY for SCADA systems.

Indeed. But in many cases they need to be done remotely.

While we're beating down on managers we should also beat down on those who advocate airgapping and then walking away as the solution to security problems. You need to embed security as a process and mentality if it is going to work. Simply airgapping will just likely get you into hot water some other way, like some Iranian cylotrons which were not connected to the internet yet got their assess handed to them.

Air-gapping isn't the solution, proper network design is.

Re:DUH.

[Lumpy]

"Indeed. But in many cases they need to be done remotely."

there is NEVER a need to do it remotely across the internet. NEVER.
If you need it remotely controlled, point to point secure encrypted T1 line. boo hoo if that is too expensive.

Re:DUH.

[omglolbah]

Costs money, different budget from any losses so the manager in charge of the expense will reject your proposal.
He has no motivation to spend the extra money to secure the system as he is not the one who is fucked if they get hacked.

Sad but true in many cases...

Re:DUH.

[thegarbz]

Or how about boohoo if it's not available or possible?

Are you denying the use case because you're incapable of doing it securely, or simply from a philosophical point of view?

The latter is quite dangerous because if competent people deny it on philosophical grounds eventually you'll end up with an incompetent person who's chasing money. THAT'S when you get open VNC connections to SCADA systems and shit like that.

Network communication is too high function

[Marrow]

Maybe these systems dont actually need all the bells and whistles of networking to communicate their state. Maybe an output-only serial communications solution would be perfect for some of these systems. They can alert when they have a problem without exposing a bi-directional communications channel through tcpip. In fact, you could even cut the pins on the serial and guarantee that nothing comes in. Its the ultimate one-way firewall.
Im not saying that all of the systems can run this way, but I bet many of them can.

Re:Network communication is too high function

[Anonymous Coward]

Nope: Scada buses are typically master/slave query/response

Re:These systems are a product liability nightmare

[I_have_a_life]

The problem isn't Windows (not sure if you are implying this or not). It's a convergence of factors which make patching systems a veritable nightmare in the process control systems.

1. The people who run the plant are trying to squeeze the maximum amount of yield from their plant. Shutting down a SCADA system so that it can be patched and tested may literally cost them millions of dollars per hour. Furthermore, the cost of upgrading is not looked upon kindly unless it's going to help you create more of product X at a lower price. You may argue that the greater good is more important than money but these guys aren't listening to that.

2. These industries are rife with rules and regulations that further inflate the cost of patching systems. In the pharmaceutical industry the cost of applying a single patch may run well into the millions of dollars because every change has to be meticulously audited.

3. IT is often outsourced to third parties in order to control costs. The downside of ceding control of your own infrastructure is that even something mundane like changing a firewall rule has a process which costs money and resources.

4. There is an old-school engineering mentality that is pervasive based on the old adage "if it ain't broke don't fix it". No person involved in the industry wants to find problems. They want the plant to produce and they expect the hardware and software they buy to produce - untouched - for 20-30 years.

I have seen crazy things at plant floors. Control systems still running on Windows NT, operators sharing credentials, copying files from one system to another using thumb drives because the network does not allow files-haring.

Re:These systems are a product liability nightmare

[Anonymous Coward]

Updating breaks now with near certainty. Not updating breaks later with a lower probability. Easy choice,

Sad, but true.

Lets over react

[AbrasiveCat]

Let get the media to over react. That will be fun, more government rules, more government oversight. I know we have multiple "SCADA" systems on my site, except most of them aren't control, they are monitoring. (Oh my! the B4-12 SquareD power meter is reading too low!! That groups power bill will be to low next month.) The other LAN connected SCADA systems on site, that I know of, would fail safe. The worst you could do is cause some experiments to fail. Part of the power of PLCs these days is having them on a LAN. (Who wants the ip of one of our PLCs, I'll give you a hint, it is on the 10. network.) Oh and do slap the folks that have true control systems open on the Internet with addressable IPs that could fail in a dangerous way.

Re:Lets over react

[some old guy]

Yup. Controls guy here.

SCADAS are just networks like any other. The controllers doing the actual work do fail safe, assuming they're fairly modern intrinsically-safe Level 4 machines. However, no matter how well Emerson, Honeywell, Allen-Bradley or Siemens secure their network architecture, somebody (usually an untrained IT SysAdmin who has no idea of the havoc an open SCADA network can wreak) will inevitably port the control network out to the whole world out of sheer ignorance. Thankfully, just getting in isn't enough to pose much danger to life or property.

Remote monitoring and data collection appliances should always be buffered away from the controls platform via an encrypted data base or other technique. If they're not, somebody needs to get fired.

Granted, no network anywhere is crack-proof given infinite cracking resources. Someone wanting to do serious harm on a Delta V, FactoryTalk, or Honeywell Distributed network is going to need the platform software and expertise in using it to do any real damage beyond, say, a temporary outage. The danger isn't from script kiddies or common criminals. The resources required for dangerous industrial hacking primarily reside with a) government and government-sponsored entities and b) nefarious controls engineers (competitors?) and there isn't much defense against these major-league threats except air walls.

It isn't as though controls engineers don't build a modicum of fault tolerance and controlled reaction to system anomalies into our systems. We do. However, if what we're chasing is iron-clad Fort Knox security in distributed SCADA systems, you might as well put a tooth under your pillow and wish for it.

Re:These systems are a product liability nightmare

[dkf]

There is an old-school engineering mentality that is pervasive based on the old adage "if it ain't broke don't fix it".

The problem with that is, by putting it on the internet, they've broken it (even if the breakage hasn't hit home yet). Nobody wants to admit that they've done that, but it's their own damn fault. A good start to fixing things would be to airgap the SCADA network from the internet, and if connecting is necessary at all, to use a good double firewall with hardened DMZ machine in between. The DMZ can be locked down hard and updated carefully, and it doesn't need to ever hold systems that need careful certifying as it should never be in the control loop; just out of band monitoring.

Re:These systems are a product liability nightmare

[frisket]

This is by no means unique to SCADA systems: I think most people here recognise the symptoms in many fields.

The people who run the plant are trying to squeeze the maximum amount of yield from their plant.

Very laudable. That's their job.

Shutting down a SCADA system so that it can be patched and tested may literally cost them millions of dollars per hour.

That cost should have been factored into the financials from Day 1. It's usually omitted by managers and accountants because with it, their projections wouldn't look as good.

Furthermore, the cost of upgrading is not looked upon kindly unless it's going to help you create more of product X at a lower price.

Bear in mind that the cost of not upgrading may be the end of the company.

In Economics 1.0, business students get taught that the primary objective of the corporation is to make a profit. Most managers believe this. Wrong. The primary objective of the corporation is to assure continuance, even if that means a couple of years of losses from time to time.

Failing to recognise this is usually among the early symptoms of eventual failure.

why are these things connected to the internet?

[csumpi]

what moron would hook these things straight to an internet connection? in the private sector, stuff like this would get you fired on the spot.

Re:why are these things connected to the internet?

[ebno-10db]

Most SCADA stuff is in the private sector.

Re:These systems are a product liability nightmare

[cusco]

Most of the endpoint devices that I've seen use either Linux (old, unpatched versions) or something akin to Tron or DOS. Management clients are often Windows, and they're unpatched and unmanaged because they're not on the normal Corp network so IT doesn't have access to them. The actual SCADA management system is normally hosted on some flavor of Unix, at least in the power and water industries.

Re:These systems are a product liability nightmare

[cusco]

Normally the SCADA systems **ARE** air-gapped from the corporate backbone, but until we start breeding better managers some idiot will occasionally pull a cable across that gap in order to produce a report or something.

Stuxnet

[Anonymous Coward]

do NOT connect SCADA systems to the internet.

That didn't help Iran against Stuxnet which jumped the air gap via USB keys. The US DoD got hit in a similar fashion with their air gap.

What you're suggesting helps, but is no guarantee.

No surprise.

[Anonymous Coward]

I worked for a company that produced industrial machines. The PLC and other hardware were pretty modern. However the control panels (U/I) were powered by Windows NT.. This was in 2008.

Re:frosty

[sd4f]

Probably is! I worked for a company manufacturing hazardous area heaters, in oz, for the oil and gas industry and many places were still using very old systems. Sure, they worked, but it didn't look like they were designed with the idea of a remote attack in mind, as they generally predated the internet.

Re:These systems are a product liability nightmare

[greenbird]

The problem isn't Windows (not sure if you are implying this or not).

Yes it is. Only idiots put any kind of embedded and/or control system on Windows. There are a whole host of reasons why but a primary one is the design of Windows makes it impossible to implement even the most basic of security.

Re:These systems are a product liability nightmare

[greenbird]

The problem with that is, by putting it on the internet, they've broken it (even if the breakage hasn't hit home yet). Nobody wants to admit that they've done that, but it's their own damn fault. A good start to fixing things would be to airgap the SCADA network from the internet, and if connecting is necessary at all, to use a good double firewall with hardened DMZ machine in between.

You know, I've never understood this predisposition towards firewalls. Secure the system such that it only listens on a specific port for specific secured encrypted messages. No need for a fire wall. A firewall just adds more complexity and points of failure. It's much more efficient to secure the system's communications than to try to secure the various access points.

Re:These systems are a product liability nightmare

[Jeremi]

Normally the SCADA systems **ARE** air-gapped from the corporate backbone, but until we start breeding better managers some idiot will occasionally pull a cable across that gap in order to produce a report or something.

This suggests a product idea -- triangular (or otherwise oddly shaped) Ethernet jacks, for use in computers that are not supposed to ever connect to the Internet. All your SCADA machines would have these, and it would be very difficult for the idiot to connect a cable to them that also connects to a non-SCADA machine.

(Until the inevitable RJ45-to-triangle adapter cable becomes widely available, anyway)

Re:These systems are a product liability nightmare

[peragrin]

here is the trick. you put them on the internet so that you can cheap out on labor.

This way you can hire on an hourly rate the programmers who make adjustments. I know a major golf equipment manufacturer, whose systems are on the net . now there is a VPN, and an air gap(the local repair technician has to physically plug the cable into the network port for updates)

However the guys who do the programing, come out to the job site for initial setup and testing, once you are up to speed. future errors are debugged over the net.

SCADA on the internet is almost a requirement. it is cheaper to hire out help on an as needed basis than paying some guy to sit on his ass 90% of the time.

Re:These systems are a product liability nightmare

[radiumsoup]

what you're describing (the port listening part) *is* a firewall - just locally installed and managed. The traditional idea of "a firewall" is exactly that, but in a centrally managed package that makes changes somewhat easier to manage and MUCH easier to scale. No difference functionally, really, except for the "listening for specific secured encrypted messages" part, which is an application-level thing anyway. Furthermore, if planned carefully, the "secured encrypted messages" part can be offloaded to a layer 6/7 switch as well, so even that's not always a restriction.

So really you just want application hardening (a good idea in most cases) and a firewall to filter the port, but you want to do that N number of times for however many hosts you have doing the same job (speaking about more complexity!) instead of centralizing it once or twice to redundant switches, etc.

Re: These systems are a product liability nightmar

[DigiShaman]

Treat them like you wood security cameras. Keep them behind their own physical switch (or VLAN) that uplinks to a dedicated firewall.

Re: These systems are a product liability nightma

[DigiShaman]

Wood (would). Damn auto correct.

Re:These systems are a product liability nightmare

[thegarbz]

No. Very few SCADA systems for plants that do anything other minor local control are "air-gapped".

Most normal SCADA systems are part of a virtual network. And that's kind of the point. Small pumping stations, local control systems that none the less need to act as part of a larger system (think power grid) require some kind of network connection.

Just because it's not the corporate backbone doesn't mean it's not the internet.

Re:These systems are a product liability nightmare

[greenbird]

what you're describing (the port listening part) *is* a firewall - just locally installed and managed.

No it's not. A firewall in every sense that I've experienced the word being used is a piece of software that monitors and filters network traffic whether installed locally or running as a gateway node on a network. What I'm saying is don't run any software that listens for network traffic except the piece of software that is using the traffic. There a huge difference. One adds complexity. You have to configure the firewall software to except the correct traffic and only the correct traffic. The other way it doesn't matter what traffic is send because there is nothing there listening to it. This make things simpler since there is nothing to misconfigure.

The traditional idea of "a firewall" is exactly that, but in a centrally managed package that makes changes somewhat easier to manage and MUCH easier to scale. No difference functionally, really, except for the "listening for specific secured encrypted messages" part, which is an application-level thing anyway. Furthermore, if planned carefully, the "secured encrypted messages" part can be offloaded to a layer 6/7 switch as well, so even that's not always a restriction.

See. this is exactly what I'm talking about. None of this is needed and contributes nothing but complexity and additional points of failure. This is an industrial control system. It's not a general network. If the only thing listening is the secure software what exactly are you going to configure the firewall software to do? What is there to scale? Your application is going to be receiving the exact same traffic if the firewall is there or not. If the application and the box it's running on are secure and running on a secure system the firewall serves no purpose. If I have a Linux/Unix box I can control exactly what is running and ensure the only thing listening for network traffic is my software. And it's trivially easy to do this if you know even basic system administration. Additionally you can configure it to ignore any additional hardware connected so some idiot plugging a USB stick in won't do anything no matter what is on it. No need for anti virus software. None of this is particularly difficult or expensive to do.

So really you just want application hardening (a good idea in most cases) and a firewall to filter the port but you want to do that N number of times for however many hosts you have doing the same job (speaking about more complexity!) instead of centralizing it once or twice to redundant switches, etc.

No. If there is no other software running why do I need software to filter the port? What you're talking about is adding pointless software.

Re:These systems are a product liability nightmare

[cusco]

The SCADA systems that I have worked with were for electrical generation and distribution and water/sewer systems, and they absolutely were air gapped. Crossing that bridge with a cable was an automatic firing offense, and yes, they canned a manager who thought that no one would notice. That utility covered an entire very large and highly-populated county and tied into the larger national electrical grid. I'll guarantee that most of the SCADA systems nationwide are air gapped, as it's required by FERC and can generate hefty fines if they're not.

Presidential Derp

[Anonymous Coward]

Thanks for to the good folks at the NSA for their pioneering work on subverting SCADA systems and encryption, and to President for firing the open shot in cyberwarfare. A bit like Saudi Arabia developing an oil-eating bacteria and using it to attack Iraq. Derp.

Re:Presidential Derp

[Anti-Social Network]

I can't believe nobody else has mentioned this. This is probably half the NSA's fault, for making proper security in any context very difficult (the other half of course is the SCADA manufacturer's fault for not building good security in from the factory). NSA wants juicy secrets from a few international groups, and thereby exposes our entire infrastructure to international malice. It's as simple as the tipping point where the engineer/manager says "Well all the security products available suck anyway, might as well save my budget and the hassle of another network middleman."

But it can't be done

[Anonymous Coward]

In a safety critical SCADA system where 99.999%+ availability is mandated, you CAN NOT just simply apply the latest O/S updates.
There is a long-winded and expensive evaluation and system regression test required for any and every such change. And the updates come thick and fast.

Where customers have demanded realtime autoupdating virus scanners and auto patching operating systems, I have told them yes we can do it, but only if the 99.999% availability requirement is waived.

Or else you risk process shutdown:
- 30,000 people trapped in the subway system because the trains have stopped.
- A Million people without power.
- Billion dollar plant out of action because the process is stopped and the goo has solidified in the pipes.
- $millions in liquidated damages.

You can only physically isolate the system, and only allow ssh for selected limited access maintenance users, preferably through a portal which is only physically switched in while the maintainer is logged in.

This isn't office-monkey IT.

Re:But it can't be done

[Anonymous Coward]

Is your 99.999% availability stuff made-up or real ?

Re:These systems are a product liability nightmare

[phantomfive]

A firewall makes port scanning harder (because by default computers respond 'port closed', so the scanner doesn't have to wait for timeout).
A separate firewall can protect against vulnerabilities in the network connection code (when it's easier to upgrade the firewall machine).
A separate firewall blocking connections out prevents malware from sending messages back home once it's installed.
A firewall lets you filter traffic, to make sure nothing strange is getting through.

These probably often outweigh the risk of adding an extra point of failure.

This wasn't the work of hackers.

[Anonymous Coward]

It was the work of crackers.

Radio Waves Shielding !

[Anonymous Coward]

One NSA slide revealed how a combination of the SOMBERKNAVE, VALIDATOR and OLYMPUS exploits can be used to extract data from Windows XP PCs that are “air-gapped”, i.e. not connected to any public networks. After taking control of a nearby wireless access point, SOMBERKNAVE is able to connect to a machine even if its embedded 802.11 device (WiFi cards are standard fare in business and consumer PCs) has been disabled.

Re:These systems are a product liability nightmare

[thegarbz]

Not all SCADA systems can sit and hum away without any external influence control or set-points. Not all SCADA systems can be set up in a way that a technician can easily travel out and download logs or trends.

The SCADA systems I have worked with are absolutely connected to the "internet". I use inverted commas since it's not connected in a way that you can just fire up it's IP address and be all happy. VPNs, firewalls, and a connection to a specific machine in a specific network only. Why? It's a pumping station. It needs a remote start command and it also needs the ability to log any local issues trips, fire deluge activations etc and report them back.

Air-gapping is not the answer in many cases. This goes especially for hazardous materials plants where the legal requirement to keep offsite data of the process may be at odds with your desire to have a stand-alone airgapped system. Though if you have the money you can always run a cable. That's what our electrical industry does. If you're going to use a helicopter to pull 6, 12 or more HV cables you may as well drag a run of fibre along while you're at it.

Re:These systems are a product liability nightmare

[Anonymous Coward]

The problem isn't Windows (not sure if you are implying this or not).

Yes it is. Only idiots put any kind of embedded and/or control system on Windows. There are a whole host of reasons why but a primary one is the design of Windows makes it impossible to implement even the most basic of security.

I can follow that for WinCE generation, but since you say blanket Windows, care to expand on what in the NT security model of Windows Embedded Standard that makes it impossible to implement even the most basic of security?

Re:These systems are a product liability nightmare

[freakyfog]

Until the ethernet card needs to be replaced and someone changes it to a normal one

Re:These systems are a product liability nightmare

[artson]

I suppose I'm an idiot, but I can't see why you can't patch and update a copy of the code, then when you are absolutely sure you haven't broken something, you just do a swap.

This message

[Anonymous Coward]

will always hold true.

http://www.youtube.com/watch?v=rjigODNy3jk

Re:These systems are a product liability nightmare

[Kasar]

Government regulations keep changing. The local hydro system here was so antiquated that they used simplex 1200 baud modem communication on the SCADA system. In modernizing, they initially had an isolated network, but the government wanted monitoring capabilities, since they have rules like no more than 1/2 inch of downstream water height variance (because natural rivers never fluctuate) and assorted other lunacy. I don't know which way the wind has blown with regulators lately, but it seemed to be a mess only exacerbated by federal dabbling.

Re:These systems are a product liability nightmare

[Anonymous Coward]

That cost should have been factored into the financials from Day 1.
It probably was. 3 managers ago.

Re:These systems are a product liability nightmare

[DigitalSorceress]

I was personally involved in a project to collect and analyze data for a plant floor at my previous job.

Plain and simple - QA and process engineers are asking for more and more data which simply can't work with an Air gap unless the entirety of the colleciotn and analysis systems are inside the Air Gapped network.

I know the company I was working for could not afford the cost of "doing it right" so I had to put routers in each production line's Industrial Ethernet internal network to NAT it out so I could get the data collection servers to gather data.

I made sure the router only allowed external requests coming from the specific data collection system's address - but I was unable to convince them of the need to set it up with a DMZ, so in theory, if you could break into our LAN and get to the correct server, you could use that to jump the air gap.

However, even then, the NAT I set up was for specific port that only allowed queries for settings/data, not for control, and there were far more juicy targets than a plastics extrusion line's controls, so even though it was a risk, the $9million / year they ended up realizing in savings due to the analysis of the data more than made up for the risk that someone would take the time to dig in to damage/control the extrusion lines.

As others have said, there are HUGE disincentives to taking down time to patch these systems... the Data Must Flow is the operational mantra, and they don't want to risk losing production time - even if the very real risk is of a break-in or even just break-down causing potential down time.

Re:These systems are a product liability nightmare

[cusco]

There's a big difference between being on a private network and being on the Internet. The utility's systems were networked, but unless you have physical access to the hardware you're not getting on that network. It's a SCADA **system**, not some stand-alone hardware controls. Agreed, you could break into the power dam control room, unplug something from the router, spoof that device's MAC address, plug into that same port, and then get on that network but don't you agree that's just a wee bit different than being able to attack something from the comfort of your living room? The SCADA system was air-gapped, not the individual devices.

Re:These systems are a product liability nightmare

[omglolbah]

Now look at this system:

DC1/DC2, handles SMB shares for users and general data storage for the engineering staff
DB1/DB2/DB3, has 50+ services running that handles everything from antivirus updates to OPC data
OPC1/OPC2/OPC3/OPC.. /OPC12, handles routing for MMS traffic between database servers and equipment/controllers
History logger, runs an oracle DB for logging every single action in the plant, required by law in this field.
BACKUP1/2, SMB shares on raid for backups of all servers and clients.

How exactly are you going to do what you propose without firewalling or air-gapping this from the rest of the networks?

In a perfect world you can limit everything to just the secure messages... in the real world you end up with DCOM communication set up to allow anything on the network to start and stop processes on anything on the domain.. *cringe*

Oh... and this is the top level of an oil rig control system, fancy that *wimper*

Re:These systems are a product liability nightmare

[omglolbah]

If a full copy of the environment would cost tens of millions of dollars, it is hard to justify it.. sadly.

Re: These systems are a product liability nightmar

[Anonymous Coward]

My company helps critical infrastructure owners meet data sharing requirements with govt agencies. If you use certain industrial communication protocols that were established pre-internet you may be in luck. In particular, we have a unique connection that is one way, only allows the data you choose to share, and does not require any sharing of your network with the outside world or feds. To be precise, your network and the govt network come within feet of each other and our unique device creates a restricted "bridge" that only passes MB data over serial. Read only.

Re:These systems are a product liability nightmare

[greenbird]

DC1/DC2, handles SMB shares for users and general data storage for the engineering staff DB1/DB2/DB3, has 50+ services running that handles everything from antivirus updates to OPC data OPC1/OPC2/OPC3/OPC.. /OPC12, handles routing for MMS traffic between database servers and equipment/controllers History logger, runs an oracle DB for logging every single action in the plant, required by law in this field. BACKUP1/2, SMB shares on raid for backups of all servers and clients.

If you're using SMB on a control system your application is broke beyond the point of idiocy already. And that's my point. If you start with a broken design you're going to end up with a broken system no matter how much money and effort you put towards isolating and securing it. If you design correctly it's MUCH cheaper in along run. You're not spending massive amounts of money on what are essentially people plugging their fingers into holes in the dyke and there are a lot more than one or 2 holes.

Re:These systems are a product liability nightmare

[greenbird]

I can follow that for WinCE generation, but since you say blanket Windows, care to expand on what in the NT security model of Windows Embedded Standard that makes it impossible to implement even the most basic of security?

Being able to simple and with certainty determine exactly what is running on the box especially with regards to external communications.

Re:These systems are a product liability nightmare

[greenbird]

A firewall makes port scanning harder (because by default computers respond 'port closed', so the scanner doesn't have to wait for timeout).

All that does is slow it down a bit. I'm not buying that it would be worth the greatly added complexity.

A separate firewall can protect against vulnerabilities in the network connection code (when it's easier to upgrade the firewall machine).

This is the one place you can make an argument and I thought about that after I made my post. But thinking it through security is about mitigating risk. Vulnerabilities in drivers are pretty rare. Especially in something as old and standardized as network drivers. And malware attacking at this level is even more rare. Anything above the driver level (layer 5 and above) is going through your firewall anyway with encrypted traffic unless you write some form of propriety firewall software that does deep packet inspection. The other point is from a security perspective I would make the base assumption that the network my control system is plugged into is unsecured anyway. I want to secure what I can control. I can't control what is on the network my devices are functioning on. There are just too many stupid people in the world to assume otherwise.

A separate firewall blocking connections out prevents malware from sending messages back home once it's installed.

Again I'd rather secure this on my system. It should scream bloody murder or even shut down if a port is opened. Now arguable you're getting into the realm of local firewall and or IDS software here though.

A firewall lets you filter traffic, to make sure nothing strange is getting through.

That's not really a firewall. That's more in the realm of an IDS. I wouldn't argue against having an IDS installed. But in that case I would rather install a true IDS/IPS rather than a firewall that may provide some IDS functionality.

Re:These systems are a product liability nightmare

[Anonymous Coward]

I can follow that for WinCE generation, but since you say blanket Windows, care to expand on what in the NT security model of Windows Embedded Standard that makes it impossible to implement even the most basic of security?

Being able to simple and with certainty determine exactly what is running on the box especially with regards to external communications.

So, when you _implement_ a security model (what you first talked about) the NT security *model* is as good if not better than Unix/Linux. If you when admin'ing a system prefer the Unix/Linux way of monitoring processes, that is a matter of taste and what you are used to I guess.

Re:These systems are a product liability nightmare

[omglolbah]

If only I wasnt under multiple NDAs I'd love to describe how insane the offshore oil business really is when it comes to security....

Some examples:

We have people accessing the secure clients from onshore using RDP, the security for that is implemented as read-only users on the domain offshore... so it assumes there are no flaws in the RDP client for an unpatched Windows 2003 server... yay.....

They gave access to the raw OPC servers for a data logging service that is managed from a 3rd party office on shore... With no access control implemented so that they could save 5000 dollars... this on a rig that produces 50 million USD worth of product -a day-.

Nobody get security at these companies, nobody. It is painful to watch your audit get marginalized because any fix will cost money.
Especially if the whole security upgrade to patch up at least 20 serious issues cost less than 10 minutes of downtime... sigh.

These rigs tend to have a top-level operator system based on windows, with limited patching and a variety of issues. Why?
Building a custom system is expensive, and any losses from breaches are gambled on by managers who are not personally responsible for anything. All they care about is short term goals and their next bonus...

I stopped feeling bad for them years ago when yet another security flaw was reported and ignored. It will bite them in the ass eventually, until then, they wont learn a thing.

Re:These systems are a product liability nightmare

[phantomfive]

Again I'd rather secure this on my system. It should scream bloody murder or even shut down if a port is opened. Now arguable you're getting into the realm of local firewall and or IDS software here though.

Generally I trust my own system enough to deal with it too (and IPTables will do its thing to slow down port scans, too). But if I had a stable of various machines from WinNT to Win8 and even various embedded OSes, and you're not sure which ports are opening when on what computers by which software, then an extra hardware firewall makes sense because you can deal with all of that in one place.

Re:These systems are a product liability nightmare

[Anonymous Coward]

A firewall lets you filter traffic, to make sure nothing strange is getting through.

Dude, I've done a lot of medical related stuff another field where you'd think security would be at least a consideration. Believe me I could tell some stories too. It's what comes from hiring non-technical PHBs for management from 1st level to CTO. And it won't change until that does.

Re:These systems are a product liability nightmare

[greenbird]

If you when admin'ing a system prefer the Unix/Linux way of monitoring processes, that is a matter of taste and what you are used to I guess.

It has nothing to do with monitoring processes. It has to do with being in complete control of what processes are running at all levels. You don't have to monitor processes.

Re:These systems are a product liability nightmare

[thegarbz]

Define "internet"? Is a secure VPN tunnel through the internet, "internet"? Some people here say yes, other's say no.

If you define a VPN tunnel to be a private network as I do then I fully agree with you a SCADA system should never be connected to the internet. Backhaul the network over the internet yes, but not publicly accessible.

Re:These systems are a product liability nightmare

[cusco]

Since the utility owned all the poles they pulled fiber everywhere, so there wasn't any need for a VPN. Yes, I would agree that a (properly configured) VPN could be part of a private network. The only issue that I had with the security for their SCADA implementation was a link between an offshore island and the mainland that used Ethernet over Power Lines for comms, but since it just ran from one substation to another substation with no feeders anywhere else any risk was minimal.

Wireless SCADA ???

[Anonymous Coward]

I have not heard anyone mention this problem. The County I live in uses RF from different pumping stations and water plants. The wireless vulnerability can't over stated.

Exempt - Heck no

[Anonymous Coward]

Over a year ago I mentioned to some manufacturing types the issue of SCADA vulnerabilities and they seem to be clueless to the hazards.

Re:frosty

[Anonymous Coward]

Mods on crack. Clearly a referense to: http://yro.slashdot.org/story/14/01/11/0248244/australian-teen-reports-sql-injection-vulnerability-company-calls-police