Slashdot Mirror
Evaluating Or Testing Utility SCADA Security?
EncryptedBit writes "I am a local elected official involved in bringing new water and waste water treatment plants online in a small town. The new plants will incorporate SCADA, which can be used to change operational aspects at the plants, up to forcing a shutdown or changing operational parameters. Can any Slashdotters recommend ways to make sure it is secure? Any testing recommendations? The operational engineers are oblivious to security and SCADA is a new factor, so this concerns me. Any pointers would be appreciated."
Seriously keep it on it's own separate network.
Seriously though, heard of "air gap"?
You most certainly should be concerned about that. Incompetent/oblivious mgmt? In the 21st Century?!?
"Tongue tied and twisted, just an Earth bound misfit
There isn't much to do with SCADA regarding security - The systems themselves are inherently insecure, the extent of it reaching only so far as default passwords that are scarcely ever changed and the requirement to have a compatible console. If you're connecting these devices to the internet in any way, then you're opening yourself up for a world of hurt. The best security is physical security, with no link to the outside world except in closed, site-to-site communications. I'm by no means an expert, but having heard experts speak about the subject and with some limited experience of my own, there really doesn't seem to be any better way the way things are.
Screw the rules, I have green hair!
0x00000047, 0x000008C4, and 0x00000A08 All hold awful vulnerabilities, make sure no one has access to them.
...I will say that training is going to be your biggest hurdle. But I'm sure you know this. Water and waste water technicians are, in my experience, terrified of technology. This manifests as an unwillingness to use technology and a hostility when forced to do so.
Other than that, treat it as you would any other service that could potentially kill people, or more importantly, lose your next election ( I know how politicians think ); Lock down the subnet, do not provide access to this remotely on equipment you do not own ( ie: no home systems connecting up for scada work ). VPN with two factor ( RSA keyfobs are pretty easy to implement against an ASA anymore ), with municipality provided laptops.
But again, it comes back to training. Good luck with that, I never was able to get that worked out ( although my hands were tied by those in charge, so maybe there's a lesson in there for you ).
Mod me down with all of your hatred and your journey towards the dark side will be complete!
It's simple......
Do NOT, under any circumstances, connect the SCADA systems, including workstations which can control or monitor them, to anything which touches or has access to the Internet. Make SURE that your control and monitor workstations have current AV in place. Do NOT connect them to the net to update the AV, figure out how to do it with sneakernet.
Further, make SURE you use RFC 1918 addressing for the SCADA systems so that they are not readily routable to the 'net.
Map the interfaces, and have a AAA (Authentication, Authorization and Accountability) strategy for each. Log EVERYTHING.
If you use a carrier to link remote sites into a WAN, make them prove to you that their pipes are clean and secure.
Have Fun......
Red...
This answer assumes you're in the US. If not, replace the relevant government agency with your own government.
Since your engineers are 'oblivious about security', you're going to have to hire somebody to evaluate your system. NSA should be able to point you towards government contractors that do this kind of thing. You could try DHS since they're supposed to help civilian infrastructure such as your operation, but their "cyber" operation isn't really set up yet.
No, it won't be cheap. You'll have to balance the cost vs. the possible repair costs after an attack.
You'll certainly might get some good tips here on slashdot for free, but really, you might want to think about getting local expertise... your next post "our town's waste and water services have been maiciuously hacked, please help, we're up to our ears in ****" might not filter to the front page.
Your internal systems need to be on their own network as others have said. Otherwise, you'll be owned. However, if you have a need to share data "publicly", you can create a data diode to a public server. It involves either a very expensive piece of hardware, or soldering a switch so there is no way to communicate to the main plant computer. Then the plant server communicates to the public server via UDP, and you can use OPC (or whatever you like) to retrieve data. If you have some idiot that wants to control stuff from home, follow the Republican Motto: Just say no.
The operational engineers are oblivious to security
Its like saying Our bus drivers are oblivious to road safety. If they don't know how to do their jobs then you are screwed.
Put an air gap between your SCADA system and the internet.
http://michaelsmith.id.au
I personally witnessed Samba root level shares on SCADA boxes at an oil refinery in Brisbane. As far as I could tell the SCADA boxes were on an intependant network but were fully reliant on no internal security.
Posting anon for obvious reasons.
Seriously scarey.
I might be repeating what others have said, but I found this looking to find out what SCADA is.
http://www.theregister.co.uk/2010/11/02/scada_search_engine_warning/
A search engine that indexes servers and other internet devices is helping hackers to find industrial control systems that are vulnerable to tampering, the US Computer Emergency Readiness Team has warned.
The year-old site known as Shodan makes it easy to locate internet-facing SCADA, or supervisory control and data acquisition, systems used to control equipment at gasoline refineries, power plants and other industrial facilities. As white-hat hacker and Errata Security CEO Robert Graham explains, the search engine can also be used to identify systems with known vulnerabilities.
According to the Industrial Control Systems division of US CERT, that's exactly what some people are doing to discover poorly configured SCADA gear.
Is anyone else terrified by the fact that this question even had to be asked?
A latent existence
For more direct advice:
1) discrete network firewalled, ideally air gapped, from the "corporate" or normal network. This is a single function network.
2) strict controls on media usage as well as protocols on how to use
3) strict config management and change control
4) physical protections to "local" and "remote" systems (RTUs, PLCs, and IEDs (note: IED = intelligent electrical device), you really don't want to build a secure control room and then get back hacked from a field device!)
To actually learn more, Idaho National Labs has the National SCADA Test Bed Program (http://www.inl.gov/scada/) and they also have control system security workshops/training programs. Their Advanced SCADA Security Training is pretty eye opening, and that's coming from the perspective of an IT security guy. Your normal operators and operational engineers will likely be blown away by it.
Like I mentioned, coming from the electric sector, I know what your facing (technical as well as cultural issues) and feel you pain. Good luck, and know you are not alone out there... just a minority! ;)
I recently went to a Belgian OWASP meeting where Justin Searle talked about "Attacking and Defending the Grid".
This guy knows what he's talking about. Among other things, he also mentioned SCADA vulnerabilities.
I recommend you contact him or his company for professional advice (I am in no way affiliated with him or his company - just thought you might be interested)
If you google for the subject of this reply in combination with "OWASP", you'll find more info about the talk.
I'm working with an international firm on Scada - we use a VPN to provide a secure private network.
SCADA systems are not designed, implemented, or operated with network and application level security concerns in mind. :)
(Usually. The exceptions know who they are
Your compensating control is physical security to limit access to SCADA elements and programming. It costs more, but you have no sane alternative.
And before you get too cocky about that restricted air gap, consider Stuxnet turning such a strength into a weakness for exploit. At some point SCADA systems will be security conscious; that day is not today...
FTFY
invite Russian, Asian, and Middle eastern hackers to attempt to bring down the network and system then find all the points they attacked and fix em
There are many different types of SCADA implementations and several options for remotely accessing them - RF (VHF or UHF radio telemetry), cellular, network (wired, or wireless bridged), etc. The answer is simple - if a disaster would result from unauthorized access you need to find a consultant that knows enough about the implementation to keep the bad guys out. Spend the money and get it done right. I've often wondered about some of the SCADA implementations over VHF or UHF radio telemetry and what the person was thinking when the system was put into place, that's as insecure as a CB radio and probably controls some critical infrastructure that people depend on. This whole topic is going to keep resurfacing for years to come, and recent events such as Stuxnet proved this is an area to be concerned with.
That doesnt really help when the client gets rootkitted, does it....
Here is the most important thing to know: security is a process not a project. No systems ever "achieve" security. Rather, better-protected systems beat the attackers by having well-thought-out-and-executed security processes.
Anybody want a peanut?
On the regulatory side, for networks the NERC Reliability Standards for the Bulk Electric Systems of North America address similar concerns (including cyber security) in electrical grids. For highly integrated systems MILS kernels are an engineering solution e.g. to keep actuators and monitoring subsystems apart.
Seriously ? Were all doomed, man ...
There are a variety of good posts here (among the chaff). The post by @bigjeff5 and the anonymous coward post amendment. For standards and an understanding of the risk metrics Sandia labs has a great set of documents for SCADA security http://www.sandia.gov/ccss/ , never mind all the FUD. You'll have to decide on whether you want a best in class, good enough, or what you can afford and wherever the three vectors meet at a solution. Technically there is no reason for SCADA to be a risk. Experience though tells us there are plenty of reasons to push the SCADA operational component into the risk category. Not being able to afford to keep the utility operational engineers employed because the technical SCADA solution cost three times your budget is the risk I usually see. What you'll need is an experienced person to act as a trusted third party and there are a lot of them out there in the real world. Be wary of people who talk about security, technical issues, operating systems, and other elements in black and white terms. They rarely have the real world experience to understand real world issues in implementation. Since you appear to be talking about water and in the United States (pardon if not) you are likely highly regulated. You will also need to balance the new requirements and regulations for implementing SCADA devices too.
--- Location Unknown
I work for a company who designs SCADA networks for water/wastewater clients. We rarely connect the SCADA network to the internet, but when we do, it takes a lot of time and money to do it right. Hire a firm who specializes in SCADA security, you can't do it on your own.
Let me guess, they recommend all SEL equipment?
Fix Your Own TV - RiddledTV.com Avoid the Landfill
If you don't understand the issues well enough to do an audit yourself, insist on an outside audit by a company that does, *and* does not sell any security products or services themselves.
Asking Slashdot is just going to get you a whole lot of uninformed suggestions from mostly well-meaning people with the occasional good suggestion buried in the noise. But you don't know enough about the subject to know which ones are the good suggestions and which are not.
Have someone perform a risk assessment on the system - and focus on the quantitative aspects (ie what the cost to the community will be if it fails). Make sure that the contract has compensatory and insurance options in excess of those amounts, so that it is in the vendors 'hip pocket' best interests to ensure it does not fail. And of course make sure that the contract has provisions for review, should the potential impacts change or the vendor changes company name, is bought out, etc :-) (yes - i've seen that happen)
You could also have someone do a thorough risk analysis of the system (google up the NIST SP800-30 document) as well as have them supply a complete inventory of hardware, software, and services they will be using to deliver the solution. Again, NIST have an online database where you can look up what vulnerabilities are known for some IT products.
Have the vendor perform a detailed risk analysis of the system - see what they think are problems, and what are not. Where you see gaps - ask them and see what color their faces turn.
Have a look around to see what failures or disasters you have seen in SCADA systems, refer those scenarios to the vendor, and ask them what technical measures they have taken to ensure that a similar act could not happen to them
You should also have your own people clarify and document their own roles and responsibilities with the system - don't assume that you have the resources on hand to manage your side of the situation responsibly - again a risk analysis will help out there.
And of course get it all in writing.
First advice: do not connect the network that runs the SCADA systems to the Internet or to any network that connects to the Internet. Leave an air gap between your control systems and the outside world. It's OK to network them, but make it an isolated, stand-alone network. It's much harder to attack a network when getting access to it requires physically going to one of your offices or plants. It may make it less convenient, but remember you're making it less convenient for the bad guys too and the consequences of a successful attack probably outweigh any inconvenience in having to bring in updates via USB drive or the like. And don't let the vendors cow you. It's your system, after all, and you that's going to be responsible if an attacker causes a problem. If they insist their stuff absolutely needs access to the Internet, ask them point-blank if they'll be willing to sign a binding contract taking full, 100% responsibility for any and all costs stemming from a successful attack on the system from the Internet. If they won't, ask yourself whether you want to be their fall-guy.
It's possible to firewall things thoroughly enough to have indirect connection (eg. the SCADA network connects to your office network, which in turn connects to the Internet, with firewalls at both boundaries), but it's dangerous. You'll need an expert network engineering and design team to do it, and the first thing they're going to ask you is why you need that connection. If you don't have a really good answer for them, you probably don't need even that indirect connection.
Air gap, it should not be connect to the internet in any form or fashion. it should only be networked with other scada computers and nothing else. Physically seperate networks than the rest of your systems. The Department of Homeland security will come out and interrogate you about it if you ask them. google SCADA site:dhs.gov for there guidelines.
There, I fixed it for you...
Delta-Mike November Bravo Tango
Not as good as they think it is, but a little better than nothing. An appliance can't secure the network, but as a gateway to the SEL network it might make sense.
One useful consequence of the Stuxnet Trojan is to provide a concrete example of how SCADA networks can be exploited. Knowing the details of how Stuxnet works can inform you of how to perform a comparable variety of penetration tests on your own network.
You will then have a measure of how vulnerable your network is to Stuxnet in particular. It's only a circumstantial indicator of how vulnerable your network is generally, which is why I'm not in favor of pen testing except as part of a more comprehensive security initiative. But it can be politically useful at the outset to have some pen testing results to share with management, and politically useful at the conclusion to show that something was measurably improved.
Protective measures directed against Stuxnet alone will not improve your security in general, but if you develop very good general security processes, you should observe that they effectively protect against Stuxnet among others. Security is hard because you can't prove the nonexistence of vulnerabilities. But you can certainly put measures in place, and you can test their effectiveness against known threats.
As an example, you've seen advice here to put some kind of strong isolation between your corporate network and your SCADA network. The extreme example is to have no connection of any kind, whether through firewalls or bastion hosts or whatever, between the networks.
That's good advice. You'd be crazy to ignore it. But you'd also be crazy to believe it sufficient. Stuxnet is a Trojan Horse. Among other strategies for getting onto your SCADA network, it rides on USB sticks. Your network could be completely isolated and yet it would be still vulnerable to attack. A slightly more sophisticated variant of Stuxnet would actively try to build a connection from your SCADA network outward. Since most network design is not geared to protecting against access from within, and since systems on your SCADA network have to be maintained at least occasionally, chances are that the network will be compromised.
So, it's no joke. This is a tough problem, and probably the toughest part is protecting against inside workers from defeating your security measures for the sake of convenience. Get professional help with this, and if I were you I wouldn't trust a single security contractor to get it all right. When you write up your RFP, do it in such a way that it doesn't bind you to selecting one winning bid.
Parity: What to do when the weekend comes.
Keep all computers off that network that are allowed on ANY OTHER NETWORK. The ONLY possible exception should be a single billing type system, where it pushes data from scada to the billing system (with the pushing system not allowed to have anything else on it). Also, all systems on the SCADA network should not have a single wifi on them.
I prefer the "u" in honour as it seems to be missing these days.
SEL does make encrypting transcievers (302x). Depending on your SCADA system, I would not yet recommend a SEL device, such as the 3332, especially for untrained engineers. What SCADA system is your company planning to employ? Have you looked into adding a dedicated phone line to keep your information off the network? Have you contacted a local power company who very often employs secured SCADA systems?
The U.S. Government fully understands the need for isolation and just how impossible it really it. There are niche companies out there that make systems that comply with specific DCID 6/3 requirements to make the system match a Protection Level. They use mandatory access control with Solaris 10 Containers, Trusted Solaris/Irix before that, and SELinux nowadays.
Here's their problem though. In order to be effective, an organisation must clearly know what must come in or out, network wise. It is difficult, technically speaking, and managing such an interface point is a speciality either run by expensive people or by cheap, clueless dimwits.
As Bruce Schneier has pointed out, liability laws need to be in place because the market will not apply the proper controls, if for nothing else, then for cost alone. Folks may complain about PCI or SOX compliance and how it doesn't really make things safer and I agree because it just forces compliance but doesn't make them want to be compliant. Companies that are able to equate vulnerability with a decrease in stock price will find themselves motivated to make it right. The fear of lawyers can be pretty good motivation to do the right thing.
Here's my recommendation. Provide an incentive for passing an inspection. Provide an incentive for the inspector. Then clearly set the rules of the competition. The incentives are not based upon a "failure to hijack," but upon an ability to control an intrusion. The inspector does not get incentive for penetration, he gets incentive for control after he's in. The integrators need to pride themselves in limiting the damage that can be done. If they keep the installation simple and easy to understand, then it's harder to find sneaky ways in.
Meanwhile, light one up and pass it over 'cause I'm not holding my breath.
The town I work for has a SCADA system for it water treatment plants and lift stations. A lift station pumps sewage to the wastewater treatment plant. SCADA (at least in my town) has two main components. The first component is a control station which is a Windows XP SP2 PC. A read-only monitoring terminal is also a Windows XP SP2 PC. The second components are several small boxes inside cabinets to which various sensors and radio links are attached. Each lift station, water treatment plant, pump house, etc. has a SCADA cabinet with the small box in it. The sensors are usually RS-232 or RS-483 and connect via RJ45 adapters to the designated ports. A Radio link is in each cabinet. Each SCADA box has an ethernet jack to connect it to a network. The lift stations and pump houses don't have a network connection back to the town's network so those stations are fairly secure.
When I started at this job the SCADA system was on the same network as the town's PCs. I fixed this by moving SCADA to its own network with no internet access. It took several days and alot of cabling (the remote terminal was the hard part) but I did it. Two weeks later there was a problem and the head water person could no longer remotely access the system to fix it. I compromised and allowed VPN access to the SCADA system. A totally non-Internet accessible SCADA system is impossible. Even if you have someone monitoring the system 24/7 on site so no VPN access is needed, your Frame Relay, T1, or other connectivity options are certainly internet accessible.
Next we have 9600 baud radio links from each remote station back to the main SCADA control station. I have no way of knowing if the information over these links is encrypted or not. Siemens says the information is encrypted but I can't verify it.
Siemens also loves XP SP2 and refuses to support us if I install XP SP3, patches, or anti-virus on the system. I can't even turn on the windows XP SP2 firewall. Siemens also seems to love Symantec PCAnywhere. Every PC that has Siemens software has Symantec PCAnywhere installed it. Versions range from 10 to 12. We just had a third SCADA PC installed and it is still XP SP2 and PCAnywhere 12, not even 12.5.
The best I could do was to physically isolate the SCADA systems to their own network. Allow VPN access only to the control station. I installed RealVNC server on the control station and put a password on it. I setup a laptop with VPN client software and the RealVNC client. So the user connects to the VPN enters a username and password (password changes every 6 months and it is at least 10 characters long, mixcase alphanumeric password), launches the VNC client and enters the VNC password (not the same password as above but uses similar specs).
I look forward to reading the rest of the posts.
You need to hire a control system or SCADA engineer to assist you to evaluate the overall requirement.
Depending on your operation you might just need a isolated network to more sophisticated requirement. You mention that you are in a small town and runs a small water treatment plant, so that tells me that a. you have a smaller size facility with only a few components (versus big facility for a big city), and b. seems that this is your only facility to operate, or one of very few. The requirement depends greatly if you are operating within the facility (inside the yard), versus if you are hiring an external company to monitor and respond to the day-to-day operational needs, which a lot of small utility district tends to do today.
I can also tell you that you should look for the outfit that are more specialize in working with water utilities instead of specialize in other industry. Different sector have different requirements. I can tell you that your safety and cost-effectiveness requirement will not be the same as the oil and gas sector, or manufacturing sector. There are plenty of consulting company or control integrator who are familiar with the physical as well as legal requirements you would need. I can also recommend you to look for someone with an PE license of your state as well.
Lastly, your network security does not trump or supersedes your safety requirement of the facility and of the general public. In fact, the whole reason network security is of any factor is to protect your facility from the potential harms that can not only take down the operation of the facility, but also cause harm to general public. Automating your facility with control and SCADA system can enhance the operational safety. It does sounds like your "operational engineer" is not a control guru, so you need someone who is knowledgeable of the subject matter.
Before you begin, revisit the mission statement for the utility. Decide now what kind of disruptions you're willing to tolerate and how much expense you're willing to fund before anyone brings you any kind of technology. Make sure the costs of disruptions and impact to the community is considered in making that decision. Check policy and local/state/provincial laws for potential impacts from outages, and include those impacts in your analysis.
When the analysis is complete, check with public relations to see what they think the response will be. you may need to revisit that decision. When you have a safe way to proceed, loudly and repeatedly publish your decision on how you intend to proceed and what your goals and tolerances are. Make sure the utility people know it and can repeat it in their sleep. Make sure the vendors know. Leave no stone unturned.
When you're about to start buying things and collecting bids, be very, very careful. If you expect the solutions you buy to do something in particular, make sure you word it that way in your requests for proposals. Don't say something like "must be able to log to our log servers" when you mean to say "must log to our log servers." Vendors will seize upon the former and, when you see they're not doing what you want they tend to respond, "well, it's *able* to do that but it'll cost an additional 27 bazillion dollars for a pro services engagement. It doesn't do that out of the box." Small utilities have a hard enough time with finding money without having to put up with that kind of silliness.
When things begin, you will hear from two types of people: Control Systems people will repeat whatever the vendor tells them about what good security is (if they mention it at all), and the vendor will want to connect a lot of stuff to the Internet. They may also want to remotely access your control systems for support and maintenance. There are sometimes reasons for this, but it's almost always an extraordinarily bad idea. Vendor back doors and B2B connections into control systems usually lead to unauthorized changes and unexpected behavior. The next thing you know, you're dependent on the vendor for everything and they can start setting the contract renewal terms. This is bad and becomes costly.
Vendors and Control Systems folks also tend to focus entirely on encryption (and sometimes access controls) as the extent of security, totally ignoring data integrity and often hand-waving availability. Software vulnerabilities that lead to denials of service are an availability issue. Poor authentication and unencrypted protocols are a potential data integrity issue. Don't let the vendors or the Control Systems people slide on those- if they have integrity, encryption, and authentication controls, make the vendor implement them.
Make the vendor come on site to do maintenance and put your foot down with the Control Systems people who want remote access- if it's important enough for need their attention in the middle of the night or on Thanksgiving, it's important enough for them to drive to the site where the equipment is and deal with an issue. Utilities can usually pay for maintenance and overtime, but would rather not pay fines for an outage or a safety issue.
IT Security people will want to do all kinds of intrusive and disruptive scans on the systems, and they like to keep things patched. This is important, but sometimes patching can break the systems that the Control Systems people make. The reasonableness threshold for patching is usually tied to the most recent version of the software and patches that are available from the SCADA vendors. Absolutely keep the vendor's software and firmware patched to the most recent levels, and don't patch the OS beyond what's supported by the vendor. That sounds counter-intuitive to a lot of IT Security folks, but that's the way it is (hence the tedium about isolation.)
As for the collection of Control Systems folks and IT people, they'll need some way to
OK. Try doing that 600 lines with ice cube relays instead... and hope nothing changes.
This is Slashdot. There are many self styled experts here. Some know what they're talking about. Many do not. Tread with care.
I am a registered professional engineer with 25 years of experience integrating, fixing, and designing several generations of SCADA and plant control systems for a large water and sewer utility. I not only design and build these things, I live with my creations through the entire life-cycle. If it does anything unexpected, they call me; no matter how old it is. I have worked on every aspect from the field to the operations control center and every single detail in between. I am not just an engineer, I write software, including protocol drivers, embedded firmware, and system management scripts. I designed the networks we use and I often show our IT department some of the finer points of network design. I'm not a consultant. I'm not selling anything.
As with safety, just as there are no perfectly safe systems, there are no perfectly secure systems. We do not have testing procedures or magic text books that we can throw at this problem. However, for suggestions, I recommend ISA-99 or NIST SP 800, if you do any work on behalf of the Federal Government in the US. In particular, see NIST 800-82. I should warn you, NIST 800-82 is a smorgasbord of suggestions. I think if someone tried to implement all of these notions you'd have a nearly unusable system. At some point, you have to stop securifying and safetying everything and just educate staff what the risks are and what to look for.
If you're really trying to do SCADA, stay away from Modbus. While Modbus is a good DCS protocol, it is a poor SCADA protocol. Too many engineers still specify Modbus for SCADA because they know nothing else. DNP3 is an event oriented, near-real-time protocol. It is a far better SCADA protocol. If you don't understand the distinction, find a consulting engineer who can explain it to you. If your present consulting engineer has difficulty understanding this nuance, find another one. DNP has been in wide use in the electrical power industry in North American for over a decade. Water utilities in Australian and the UK are using DNP. It also has new features for secure authentication. See IEC 62351-5 for details.
On the other side, Office IT is not the same as a real time or near real time IT. I know of at least two books explaining this difference. If your IT staff do not understand that this application is truly different, find someone who does.
Understand that control engineering is a very broad field, embedded IT is a very broad field, and IT security is a very broad field. If you put the experts in the same room, make sure you define the problem for them very carefully. These fields tend to make prima-donna characters out of otherwise mild mannered people. I can guarantee a very heated argument after a short period of time and even a fist fight is not entirely out of the question. We do not have room for this kind of behavior.
The IT security people need to understand that while IT systems can be restored from backups, physical systems can not. They also need to understand that we do not spray patches at the field on a whim. We do not have broadband to a manhole in the middle of the street. And we use common carriers with extreme caution and evaluation. Likewise, the engineers need to stop thinking about building things and start thinking about how to break them. Too many engineers assume that nobody in their right mind would mess with their wonderful creations because it would clearly break something. That's like designing cars without considering what they'll do when they crash.
I suggest getting familiar with the various standards efforts. Also, I suggest keeping track of what is going on with SCADA security. Note some of the following
(Shameless plug alert)
I am a co founder of SCADASEC, an email list for SCADA and ICS security issues. I am also a member and contributor to the DNP3 and ISA-99 standards committees. These are many popular efforts. The NIST SP 80
Nearly fifty percent of all graduates come from the bottom half of the class!
http://www.bcit.ca/appliedresearch/gait/projects/scada.shtml
give these guys a phone call, if their tool sucks for you, I'm sure they can give you some pointers
enjoy
-judging another only defines yourself
The University of Idaho has a research lab dedicated to this. They have SCADA systems set in a lab. There's a grad class in the subject you can take via video. And there are quite a few papers they have helped published. Search the IEEE document library for some good info.
Howdy, The only way to secure it is with an Air Gap. A PC with a CDROM drive as the final "non-connection" to the internet. The problem with that is that the vendor will not like it, since they like to keep their grubby fingers inside your system remotely, so that they can see what you are doing and upload fixes easily. The last thing they want to deal with is a air gap CDROM.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
Did you ask them about their implementation? VPN is like any system as secure as you wish to make it.
Every computer system that was ever designed, every software that was written was done to share information, not to secure it. That is until recently. It wasn't until we built all these systems and got them all working that we then went 'aw CRAP, what about security...'
The internet was designed by engineers. Software was written by engineers. They both shared the same flaw. Engineers build things. Nearly every engineer I've met operates on the principle of "If it ain't broke, don't take it apart and find out why!"
I am a security auditor - I've audited SCADA networks and have gone head-to-head with the engineers that designed and implemented them. To a person, they nearly all take it personal when I critique the security aspects of the network. Don't get me wrong - the networks are built flawlessly, but they are all glass houses. I call the SCADA networks "Crystal Palace" because nobody should be throwing rocks anywhere near them. Keep them as protected as possible from anyone who wishes to throw rocks... Isolate the heck out of them, then isolate them again.
If you enter into the project with security first and foremost in mind - you are WAY ahead of the game.
I suggest the following:
1) Have a security expert on hand during design and consult with them on installation. It may be expensive, but it is far cheaper than building it and then securing it after the fact (which often means rebuilding it because it is VERY difficult and expensive to change a system that requires 100% uptime).
2) At every step of design and implementation, ask the question "What about security?" - It will be annoying, but you'll be glad you asked.
3) The SCADA network is a glass house - protect it from anyone and anything. If possible, maintain an 'air gap' between the SCADA network and the rest of the world. Engineers that need to access that network from the 'outside world' must VPN into it - then use remote desktop or some similar technology to administer it. By 'outside world', I mean any computer that has internet access, or rather access to the internet - irrespective if it is behind the firewall on your own network.
4) Provide security in layers - never allow the SCADA network to access the internet directly or indirectly, ever.
You'll want to consult with a security expert - one with experience in SCADA networks.
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
use a dialup connection. use ssh encryption. block IP addresses after x unsuccessful attempts to login. Provide alarms to operators if many unsucessful attempts are made to login. use a cable modem from a local isp that's off the town network with a strong dedicated linux firewall. Use two factor authentication. (any activex object can be called from wonderware's solution, and these can include smart card protocols.) How much is enough? Well, how often are there break-ins in the wild with these operations? What are they doing? In any case, get a 3rd party to look at security, but don't get crazy about it. Don't waste tons of money! Use probability to your advantage. What is the allowable risk you can take? is one break in per 10 years acceptable, 100, 1000? How does each layer of security affect that goal? Unplug net connection if not in use? So many ways.... each adds some security. 1 in 1000000 that they know your dialup (a magicjack number in timbuktu?)? 1 in 10,000,000 that breaks strong ssh2 with keys? 1 in 10 that they get through a router? 1 in 100 that they crack the database where tag information is kept? 1 in 1000 that the alerted operator does nothing? Life is risky. Just stack some probability.
Air gapping (as others have mentioned) is a great idea, but not always feasible. Remote access to plants is sometimes needed for emergencies.
Have a look at the Computer Security Resource Centre. NIST IR 7628 covers cyber security for the smart grid, and much of that is applicable to water and sewerage plants.
The report, 21 Steps to Improve Cyber Security of SCADA Networks, is also worth a read.
My opinions, based on power station and substation SCADA are: don't use one vendor for everything, have two levels of firewall from different vendors at the remote site, turn off or block services that are not needed from every device, have reporting and audit trails that are reviewed, and if management want reporting, do it through a one way connection to an intermediate system (one way Ethernet, RS232, RS485 or read only shared storage).
Simply put..
A. Do not under any circumstances connect this thing to the internet. no if, ands or buts about it.
B. If outside access is required it is by call-back ONLY.
C. EVERYTHING is logged, 100% no exceptions. Every keystroke, every entry, every response, every everything, and the logging is hard connected and encrypted.
D. Nothing is interconnected to your main business network. Place secure terminals that boot from the SCADA network that have NO media devices in them, no USB, No DVD/CD drives, no serial ports etc. in places where they are needed and the are hardwired back to the SCADA network.
The stuff above is a starting point and it needs more from there.
Hey KID! Yeah you, get the fuck off my lawn!
With a VPN, you are connected both to the Internet and the VPN. That means you're vulnerable.
When you're managing SCADA systems, you are inherently managing a massive target. Acceptable security limitations in the commercial world are not acceptable with SCADA.
As the design life of these systems is usually 20+ years whether by being compromised or for some other reason at some point the SCADA/automatic control system will fail/become erratic, so instead I'd suggest you ask how they expect to cope with such failures.
To start you off
* Is there a UPS backed independent hardwired telemetry system to alert operators to control system failure, instrument failures and hazardous conditions?
* Is any chemical dosing controlled and logged by separate hardwired systems with physical security?
* Is there a hardwired fallback mode of control? (using float switches, level probes, timers and relay logic, etc to enact simple control)
* Are there local operator controls to run the plant in hand when the automatic control system has failed?
* In the event to automatic and hardwired control failure are the plant operators trained to be able to run the plant in manual?
* How quickly can you restore the PLCs?
* How quickly can you restore the system's last known good setpoints?
* How quickly can you restore the SCADA?
* Is there a gen-set, does it have fuel, it is regularly exercised, does it have automatic incomer switching, it that regularly tested?
* Are there exercised spares and someone to fit them?
So I ask you again, did you ask about the implementation of the VPN?
Our SCADA access occurs the following way: A server receives data through a strictly one way connection from the control network. That server outside the control network can serve a remove monitoring display via Citrix from the company intranet only and is accessed via a VPN. It is read only not only by security setup, but by network design as well since no data is allowed back onto the process network. I as end user engineer who needs access to this data from home on occasion first need to log into our company network via another VPN, the client software of which does all sorts of security checks (such as not allowing a connection if my virus defs are more than a week old), and also plays with the windows routing table so now ALL data goes through the VPN. As a result I can't print to the printer on my home LAN anymore while I'm connected.
The VPN is encrypted, and the endpoints use certificates which are hardcoded. If they change, I'm shitouttaluck without a patch for the VPN client. You seem to think that the term VPN means, set it up in windows networking and be done with it.
So start with 5 points and tell me your attack vector. Subtract 1 point for every convoluted hack you propose that nothing but a hacker genius with inside information can figure out.
Security is a process, a culture. It's not a list of dos and don'ts. Oh and SCADA IS the commercial world. Number 2, 3, 4, and 6 of the top ten Fortune 500 companies are users and suppliers of SCADA equipment, and two of them have no problem accessing their control systems via some convoluted VPN. Fell free to go and hack them.
K, you absolutely sure nobody on your intranet could ever get infected with malware?
If you answered "yes", you should try again.
Until someone alters that setup...or walks a USB drive over to the SCADA network. Have you disabled all the external ports on the SCADA systems, or do you take extraordinary efforts to make sure the janitorial and maintenance staff are happy?
And while you are probably not going to get infected with anything, do you trust the families/children of all your remote users?
If you answered "yes", you should try again.
Well, you've now guaranteed that your network is protected against people trying to steal banking credentials. Someone who's interested in attacking your network will have designed and implemented their own malware, which will not show up in AV def files because they didn't infect your AV vendor's honeypots.
Sweet! Now I don't have to figure out which network to use. You've saved me a few minutes, since everything I do will be sent to the target network.
Would be an issue if I was trying to brute-force your VPN, but it's much easier to exploit your end-user's love of lolcats or farmville. Or more likely, embarrassing porn. It's not only an infection vector, it's also great for blackmailing them into revealing insider information.
You seem to think "VPN" means instant, magic security, where no client machines could ever be compromised before connecting. VPN just gives you a safe network connection. It does nothing to protect the nodes on each side of the connection.
Because Stuxnet doesn't exist.
You seem to be forgetting that by it's very nature SCADA systems are wonderful targets. Hackers in their parent's basement aren't the threat. Your security is designed to prevent their attacks. But they're throwing a large net trying to get bank passwords and such, and they don't have much in the way of resources. Up to this point, this has been the threat for most computer networks.
SCADA means you're now at the level of nation-states for your adversaries. From our perspective, they have infinite resources. Attacks that are 'too hard' for the kid in the basement are pretty easy when your budget has more than nine figures and you measure computing power in acres (or hectares). Which means if there are any theoretical holes in your security, they will be exploited.
Start with ICS-CERT (Industrial Control Systems - Cyber Emergency Response Team). Note: While they use the "us-cert.gov" domain, they are NOT a part of US-CERT.
Specifically, take a look at their "Recommended Practices: section: http://www.us-cert.gov/control_systems/practices/Recommended_Practices.html
Bump.
- scout
K, you absolutely sure nobody on your intranet could ever get infected with malware?
We did, twice since I've started working where I do. Once was Stuxnet. It was a grand no event. Every network infected except the control network. Why? Because the network was designed from the ground up with security in mind. Naturally assuming VPN doesn't play a role in SCADA systems and everyone who implements it is your biggest downfall here. It's just part of a system which encrypts external traffic.
Until someone alters that setup...or walks a USB drive over to the SCADA network. Have you disabled all the external ports on the SCADA systems, or do you take extraordinary efforts to make sure the janitorial and maintenance staff are happy?
Yes on the disabled external ports. Don't care about janatorial staff. All the machines are spread around three locations on site and are all in locked cabinets. Only 1 person has the keys to the cabinet and he gives them out to a select few people if needed. Operator consoles are under lock and key, they don't have normal keyboards, but rather customised game pad style things so they can't start playing with the OS. Also we give them a separate computer which they can screw with as much as they like since it's connected to the standard intranet. Again you're assuming that you're smarter than everyone.
Well, you've now guaranteed that your network is protected against people trying to steal banking credentials. Someone who's interested in attacking your network will have designed and implemented their own malware, which will not show up in AV def files because they didn't infect your AV vendor's honeypots.
We got stuxnet before anyone knew of stuxnet. Mcafee took only a few hours to come up with a virus def file for us. Again it infected the intranet, but not the control network. Even if we ran a Siemens solution rather than a Honeywell system no one was overly worried, except the IT dept who had to pull an allnighter to get rid of the infection. When you're talking companies with competent IT departments, infections are actually quite quickly noticed.
And while you are probably not going to get infected with anything, do you trust the families/children of all your remote users?
What the hell kind of a question is that? What are we talking about again? SCADA systems. Do feel free to come over, I'll be more than happy to log you in and let you play with the graphics. After all there's a big difference between seeing data and manipulating the plant from outside (something which I haven't seen anyone give access too). I'm more worried about Dr Evil coming to my computer while it's logged on and deleting my documents, than him screwing with a read only control system window which doesn't actually connect to the control network anyway. In this case I'll just call the IT dept and get them to load the most recent backups.
You're problem is that you assume VPN is the only part in play here. Yes it only provides end to end encryption, but this is a big part of accessing a SCADA system. Still you use the term "acceptable security in the commercial world". I still say SCADA is part of the commercial world.
Please call again when Exxon/Mobil is in the news because their systems got infiltrated. Actually please call again when you hear if Stuxnet actually did anything nefarious to the intended target, because so far the biggest scaremongerer of them all has done nothing to it's supposedly Iranian target. Even if the Iranian operators were stupid enough to have a system vulnerable to it there are quite a lot of competent people in the world who are happily using VPN to access SCADA systems remotely as part of their security solution.
Sure, but that route has a tendency to get your agents caught. Much better to pay daddy a couple grand for the information and/or data.
Yes, which is what I already said. My entire point was that a VPN alone can not protect your network. In fact, the VPN is the least-important security feature.
Helpful to know I have 3x as many options for compromise.
Oh no! Not a lock!! That's not been routinely defeated for literally centuries.
Do you place it conveniently close to the SCADA systems?
No, there are plenty of people who can kick my butt intellectually. But they don't start with claiming a VPN is the root of their security, and then start talking about it as a small piece of the puzzle.
(Take a look back up-thread. You'll find that the post I originally responded to mentioned a VPN as the source of their security)
Heh. I'm sure Google, Adobe, Dow and the others will be happy to hear their IT staff is incompetent, since it took them 2 years to find the attacks announced at the beginning of this year.
You'd probably be better off not lying about stuff that can be refuted with quick Google search. Researchers in Belarus had it for quite a while before McAfee.
Well, you're the one who seemed to be claiming that VPN'ing from home to the SCADA system was ok. Home machines aren't going to be nearly as well protected because they will have uneducated users in the form of small humans who trust easily.
My point here is the security comes from the read-only part. Not the VPN. Well, at least the read-only part makes it more difficult.
SCADA is implemented in the commercial world, but needs significantly higher levels of security. Not standard commercial-world security. You yourself agree with this, since your SCADA network is more secure than your intranet.
Why on earth would it make the news? Who, exactly, would be willing to talk to reporters? The security admins who failed? Perhaps the executives looking to drop their stock price?
Stuxnet is just a widely-known example. Why would an Iranian-targeted malware have far more installs outside Iran than inside? That's the big clue that it wasn't developed by pros.
I agree with you almost completely. The only exception that I take is assuming that a competent IT staff is available.
This may be a given in industrial settings, but municipal government is an area where you often don't even have an IT function -- the role just doesn't exist in the organization. I would guess that this is even more likely to happen in sewer and water districts, which are often quasi-independent government entities on their own that are dominated by a combination of appointed political hacks who need help tying their shoes and civil engineers who think in terms of pipes and backhoes.
Conformity is the jailer of freedom and enemy of growth. -JFK
They are a UK based organisation (I'm assuming that you are in the US), but they have produced lots of useful papers on risk assessment and risk management for SCADA. Full disclosure - I used to work there. A good starting point is here www.cpni.gov.uk/protectingyourassets/scada.aspx