Evaluating Or Testing Utility SCADA Security?

EncryptedBit writes "I am a local elected official involved in bringing new water and waste water treatment plants online in a small town. The new plants will incorporate SCADA, which can be used to change operational aspects at the plants, up to forcing a shutdown or changing operational parameters. Can any Slashdotters recommend ways to make sure it is secure? Any testing recommendations? The operational engineers are oblivious to security and SCADA is a new factor, so this concerns me. Any pointers would be appreciated."

Comments

Don't put it on the Internet!

[Anonymous Coward]

Seriously keep it on it's own separate network.

Re:Don't put it on the Internet!

[j35ter]

Just keep away from Simatic and Wonderware! Better yet, keep away from ANY kind of winXP implementation. Oh yeah, don't forget to *physically* lock all USB ports and external media. Of course, you should make sure that there is no network connectivity with the outer world. And I mean none! No multihomed machines, no VLAN's etc. And, of course, if you see all alarms going off at once...run :)

Re:Don't put it on the Internet!

[Bigjeff5]

Good safe practice for separating a process control network from the internet is something like: internet > corporate network > buffer network > process network. Completely separating it is not advisable, because it can actually make it harder to administer and protect (updates, antivirus, etc). It's an option though if you are diligent with sneakernet updates and whatnot.

The network your SCADA system runs on should never, ever have direct access to your corporate network or the internet, your buffer network should never have direct access to the internet, and your corporate network should never have direct access to your process network. Be stingy about what you allow through the firewalls at each layer.

Basically when you need SCADA data outside the process network, you send it to the buffer network, and from there it is accessed from the corporate network. All controls should only be managed from the SCADA network (i.e. don't set something up so that it can be managed from the corporate network).

Separation is key. As others have said, SCADA networks need a lot of open access to the various systems they control in order to function efficiently, so within the network things have to be practically wide open. The only real option is to protect yourself with layers to make it nearly impossible for anything you don't want to access the system.

Re:Don't put it on the Internet!

[Anonymous Coward]

As a security expert who has audited SCADA systems, I must amend the above remark. If you ask them, "I'm here for security, is it connected to the Internet?", they will say "no". If you instead ask them, because you are concerned, "in an emergency, how an administrator can access it remotely", then will tell you the series of systems that will allow you to connect in- firewalls, vpns, and usually a last hop Citrix remote desktop session to the SCADA software itself, which is often Siemens, and is run in a VM. When you tell them to take it off the Internet, they will put your request in change control, and find a way to get rid of you, or if you're a politician, to buy you out. Usually the people running these things have no ability to think adversarially, so they consider something that is several levels removed from the Internet to not be Internet connected, even though it is. They may even tell you that it has its own leased network, run over Frame Relay leased from a telco, which is quite common. This is also internet connected, as the ISP can get pwned, and the frame relay stuff has a management network that is on the ISP's LAN. I've done security for a Fortune 50 ISP as well.

The short answer is, every SCADA system in the Americas is Internet connected, and no one has the balls to tell them to stop. They will only hire people to audit them who will put on kid gloves and play by their rules, and they refuse to take advice from their vendors, who they pay to be compliant. A security consultant lecturing a SCADA client on security measures is like a temp secretary lecturing a CEO on spelling. It's an event that always ends in a raised eyebrow and a prompt firing.

Every nuclear reactor in the united states is internet connected. I've seen them. I'm certain. Being a whitehat pen tester these days is like being a turned out whore- you know you're not helping anyone but it's too late to go back.

Posted anonymously for obvious reasons.

Re:Don't put it on the Internet!

[Anonymous Coward]

I almost forgot. Don't ask them, "is your network secure", because they will say yes, even though there's no such thing as a secure network. The appropriate question is, "how defensible is your network" and "what is your dwell time", i.e., what do you have in place to stop intrusions and how long do hackers usually last on your network. But the best question to start with is "what's your incident response budget", which they will brag about, i.e. how much money they spend on getting hackers removed when they're owned. Start there, get them to tell stories about the last time they got hacked. Then you don't have to listen when they start telling you about how they're "secure".

Re:Don't put it on the Internet!

[Mysteray]

Completely separating it is not advisable, because it can actually make it harder to administer and protect (updates, antivirus, etc).

Yeah, it might make it a little harder to work with sometimes.

TOO FLIPPING BAD!

Get your lazy ass off out of your chair and maintain the low-level infrastructure like it needs to be. Sometimes the infrastructure needs a guy with a truck, a wrench, and even a firmware update to match.

There's absolutely no reason industrial control processes should be accessible to the same web browser that can play Facebook games. Ever. There's little or no security isolation between the systems, regardless of how many proxies you put in place. The web just was never designed to work that way.

They really should have as few interoperable ports (e.g. USB) as possible.

Don't believe me? Just ask the Iranians right now.

Re:Don't put it on the Internet!

[j35ter]

IMHO the standard multi-layered network security approach does not always satisfy the security needs of a process management system.

you really don't want your windows machines to update themselves: NO NO NO NEVER!!!
Just imagine the operators' dumb face when WinXP goes into a restart during some critical operation, this could seriously ruin a plant managers Tuesday!

One important aspect is the internal security, people are lazy, and if you allow the to use a USB drive somewhere (management demands reports in electronic form!) you can be sure that someday some admin will need an aspirin and a new job!

The best approach is to have a 2 tiered network: PLC's and SCADA ... and basta!

Re:Don't put it on the Internet!

[noidentity]

Usually the people running these things have no ability to think adversarially, so they consider something that is several levels removed from the Internet to not be Internet connected, even though it is.

Indeed, everything is connected to the Internet. It's always just an Ethernet cable away. Send the right commands over a phone line, and that cable will be connected for you.

Re:Don't put it on the Internet!

[KUHurdler]

Seriously, the original poster needs to hire someone that is an industry expert in Utility Communications. Disclaimer: I work for this division, but we do this stuff every day and it's not the kind of thing you want to try doing on your own. In the worst instances, there are no second chances.

Re:Don't put it on the Internet!

[thegarbz]

Typical slashdot answer. When most of the worlds popular SCADA implementations are run on a windows based system, and many which are not currently are slowly moving that way, what are you left with? Keep it on it's own network completely isolated depending on the plant may also run afoul of laws requiring off-site historical databases. It'll also anger ops engineers no end when they can't easily aggregate and manipulate the historian data on there computers which gives you another problem, do you let an external party directly onto the process control network, and allow them to bring and install the software they need?

My advice for the OP, talk to the manufacturer. Many SCADA implementations I have seen from reputable companies come with a very hardened network design. - Only ever allow one way communication from the control network outside. - Have a computer on the outside historize the data so other's have limited access to it. - On the process network itself ensure the boxes are under lock and key. If the operators have access to a physical mouse and keyboard then make sure they can't leave the application. No USB sticks, nothing! - For shits and giggles and time of need put a kill switch on the firewall that lets the data out of the control network.

Re:Don't put it on the Internet!

[Mysteray]

While there are some truly compelling advantages to KISS/dumb systems, there's usually no reason that the system has to have a painful interface. The UIs for medical equipment, for example, are highly polished and tested for ease-of-use. It's just IMHO dangerously stupid to put plug-n-play networking, USB, or Wifi on a drug infusion pump.

If plug A fits into socket B, somebody's going to plug it in. Drop some infected USB drives in the parking lot, somebody's going to stick one in a USB port behind your firewall. Have an open USB port, somebody's going to charge their MP3 player from it. If it has a web browser and connectivity, somebody's going to surf with it.

Power-grid-like critical systems need to export their data from a DVD burner, not over the network. This can happen even several times a day. If this causes problems due to the latency it introduces into some spreadsheet-based workflows, then the system needs to be fixed. It's horribly broken if desktop office applications have been allowed to creep into the control loop!

Just my 2c, I don't expect everyone to agree with it.

Re:From what I understand

[postbigbang]

"Bastion hosts" are an oxymoron. Every device on a network needs the best self-protection set at the highest possible standard. Layers of security are only incrementally effective. It takes only one bot to bring you down. Firewalls, although they sound impressive, are ineffective when users plug in infected flash drives, or other media. You have to distrust every device on your network without exception-- even routers and switches can be broken into using fuzzing techniques. Continually hack yourself, ethically, using the latest techniques. Then do it again. Be afraid, be very afraid. If you're not afraid, then I'm deeply afraid.

Attacking and Defending the Grid

[andyverbunt]

I recently went to a Belgian OWASP meeting where Justin Searle talked about "Attacking and Defending the Grid".
This guy knows what he's talking about. Among other things, he also mentioned SCADA vulnerabilities.

I recommend you contact him or his company for professional advice (I am in no way affiliated with him or his company - just thought you might be interested)

If you google for the subject of this reply in combination with "OWASP", you'll find more info about the talk.

SCADA and Security are not yet integrated

[PureFiction]

SCADA systems are not designed, implemented, or operated with network and application level security concerns in mind.
  (Usually. The exceptions know who they are :)

Your compensating control is physical security to limit access to SCADA elements and programming. It costs more, but you have no sane alternative.

And before you get too cocky about that restricted air gap, consider Stuxnet turning such a strength into a weakness for exploit. At some point SCADA systems will be security conscious; that day is not today...

Re:SCADA and Security are not yet integrated

[ScrewMaster]

And before you get too cocky about that restricted air gap

This is the fundamental problem right here. The kind of mind that thinks airgap and be done with it is usually the kind of mind that doesn't think about security, and will fall for exactly the likes of Stuxnet. This illusion that everything is fine because there's no internet can often be worse than a well designed remote access system built from the ground up with security in mind. They do exist and the ability to remotely see what is going on can pull you out of the shit, or put you in the shit depending on how it's implemented.

The fundamental problem is that good security is a process, not a state, and you cannot have good security by any fire-and-forget solution.

Re:VPN

[jeff4747]

we use a VPN to provide the illusion of a secure private network.

FTFY

Re:From what I understand

[NewbieProgrammerMan]

There isn't much to do with SCADA regarding security - The systems themselves are inherently insecure...

As somebody that worked at a SCADA software company for a few years, and saw (1) the skill level of the core development team and (2) what customers did with our systems, I heartily endorse this viewpoint.

Re:Check with Ahmadinijad?

[mystik]

Remember Suxtnet? Not too long ago?

It spread by usb drives, which Gleefully jump the "air gap".

It's slightly more complicated than simply keeping an air gap, and probably requires the consultation of someone who's had experience securing these types of networks.

I too work with SCADA systems

[jake-itguy]

The town I work for has a SCADA system for it water treatment plants and lift stations. A lift station pumps sewage to the wastewater treatment plant. SCADA (at least in my town) has two main components. The first component is a control station which is a Windows XP SP2 PC. A read-only monitoring terminal is also a Windows XP SP2 PC. The second components are several small boxes inside cabinets to which various sensors and radio links are attached. Each lift station, water treatment plant, pump house, etc. has a SCADA cabinet with the small box in it. The sensors are usually RS-232 or RS-483 and connect via RJ45 adapters to the designated ports. A Radio link is in each cabinet. Each SCADA box has an ethernet jack to connect it to a network. The lift stations and pump houses don't have a network connection back to the town's network so those stations are fairly secure.

When I started at this job the SCADA system was on the same network as the town's PCs. I fixed this by moving SCADA to its own network with no internet access. It took several days and alot of cabling (the remote terminal was the hard part) but I did it. Two weeks later there was a problem and the head water person could no longer remotely access the system to fix it. I compromised and allowed VPN access to the SCADA system. A totally non-Internet accessible SCADA system is impossible. Even if you have someone monitoring the system 24/7 on site so no VPN access is needed, your Frame Relay, T1, or other connectivity options are certainly internet accessible.

Next we have 9600 baud radio links from each remote station back to the main SCADA control station. I have no way of knowing if the information over these links is encrypted or not. Siemens says the information is encrypted but I can't verify it.

Siemens also loves XP SP2 and refuses to support us if I install XP SP3, patches, or anti-virus on the system. I can't even turn on the windows XP SP2 firewall. Siemens also seems to love Symantec PCAnywhere. Every PC that has Siemens software has Symantec PCAnywhere installed it. Versions range from 10 to 12. We just had a third SCADA PC installed and it is still XP SP2 and PCAnywhere 12, not even 12.5.

The best I could do was to physically isolate the SCADA systems to their own network. Allow VPN access only to the control station. I installed RealVNC server on the control station and put a password on it. I setup a laptop with VPN client software and the RealVNC client. So the user connects to the VPN enters a username and password (password changes every 6 months and it is at least 10 characters long, mixcase alphanumeric password), launches the VNC client and enters the VNC password (not the same password as above but uses similar specs).

I look forward to reading the rest of the posts.