Slashdot Mirror
Evaluating Or Testing Utility SCADA Security?
EncryptedBit writes "I am a local elected official involved in bringing new water and waste water treatment plants online in a small town. The new plants will incorporate SCADA, which can be used to change operational aspects at the plants, up to forcing a shutdown or changing operational parameters. Can any Slashdotters recommend ways to make sure it is secure? Any testing recommendations? The operational engineers are oblivious to security and SCADA is a new factor, so this concerns me. Any pointers would be appreciated."
If your reactor starts glowing blue, you may be infected.
Seriously keep it on it's own separate network.
I would recommend giving these guys a l.ook They helped us with a complete installation at an electric utility I worked at.
http://www.selinc.com/engineeringservices/SCADA/
Seriously though, heard of "air gap"?
You most certainly should be concerned about that. Incompetent/oblivious mgmt? In the 21st Century?!?
"Tongue tied and twisted, just an Earth bound misfit
http://en.wikipedia.org/wiki/SCADA#Security_issues
There isn't much to do with SCADA regarding security - The systems themselves are inherently insecure, the extent of it reaching only so far as default passwords that are scarcely ever changed and the requirement to have a compatible console. If you're connecting these devices to the internet in any way, then you're opening yourself up for a world of hurt. The best security is physical security, with no link to the outside world except in closed, site-to-site communications. I'm by no means an expert, but having heard experts speak about the subject and with some limited experience of my own, there really doesn't seem to be any better way the way things are.
Screw the rules, I have green hair!
0x00000047, 0x000008C4, and 0x00000A08 All hold awful vulnerabilities, make sure no one has access to them.
Air gap, it should not be connect to the internet in any form or fashion. it should only be networked with other scada computers and nothing else. Physically seperate networks than the rest of your systems. The Department of Homeland security will come out and speak to you about it if you ask them. google SCADA site:dhs.gov for there guidelines.
...I will say that training is going to be your biggest hurdle. But I'm sure you know this. Water and waste water technicians are, in my experience, terrified of technology. This manifests as an unwillingness to use technology and a hostility when forced to do so.
Other than that, treat it as you would any other service that could potentially kill people, or more importantly, lose your next election ( I know how politicians think ); Lock down the subnet, do not provide access to this remotely on equipment you do not own ( ie: no home systems connecting up for scada work ). VPN with two factor ( RSA keyfobs are pretty easy to implement against an ASA anymore ), with municipality provided laptops.
But again, it comes back to training. Good luck with that, I never was able to get that worked out ( although my hands were tied by those in charge, so maybe there's a lesson in there for you ).
Mod me down with all of your hatred and your journey towards the dark side will be complete!
I worked at a glass fabrication plant with SCADA controllers.
Those guys bought an interface card for a PC, plugged it into the SCADA network, and used the same computer for controlling a tempering furnace and surfing porn. Big mistake.
Keep the control systems completely dedicated to controlling stuff, don't install extra software on them. Don't connect them to a network. And you're fine.
if your engineers are oblivious, it sounds like you need to fire them.
It's simple......
Do NOT, under any circumstances, connect the SCADA systems, including workstations which can control or monitor them, to anything which touches or has access to the Internet. Make SURE that your control and monitor workstations have current AV in place. Do NOT connect them to the net to update the AV, figure out how to do it with sneakernet.
Further, make SURE you use RFC 1918 addressing for the SCADA systems so that they are not readily routable to the 'net.
Map the interfaces, and have a AAA (Authentication, Authorization and Accountability) strategy for each. Log EVERYTHING.
If you use a carrier to link remote sites into a WAN, make them prove to you that their pipes are clean and secure.
Have Fun......
Red...
One of the first known attacks against a SCADA system was basically against exactly the same type of systems you are trying to protect
http://csrc.nist.gov/groups/SMA/fisma/ics/documents/Maroochy-Water-Services-Case-Study_report.pdf
D.
This answer assumes you're in the US. If not, replace the relevant government agency with your own government.
Since your engineers are 'oblivious about security', you're going to have to hire somebody to evaluate your system. NSA should be able to point you towards government contractors that do this kind of thing. You could try DHS since they're supposed to help civilian infrastructure such as your operation, but their "cyber" operation isn't really set up yet.
No, it won't be cheap. You'll have to balance the cost vs. the possible repair costs after an attack.
You'll certainly might get some good tips here on slashdot for free, but really, you might want to think about getting local expertise... your next post "our town's waste and water services have been maiciuously hacked, please help, we're up to our ears in ****" might not filter to the front page.
The usual way to "secure" a SCADA network without having to become an expert in the specifics of the typically undocumented or proprietary protocols it uses, is to have the plant network be connected to a hardware VPN router; any cheapo Cisco will do. This way, the only thing that can "dial in" is a PC on the other end running the VPN client; hopefully your _real_ IT dept controls this PC, and can make sure it has antivirus, updates, etc. installed ( + a sane security policy).
Your internal systems need to be on their own network as others have said. Otherwise, you'll be owned. However, if you have a need to share data "publicly", you can create a data diode to a public server. It involves either a very expensive piece of hardware, or soldering a switch so there is no way to communicate to the main plant computer. Then the plant server communicates to the public server via UDP, and you can use OPC (or whatever you like) to retrieve data. If you have some idiot that wants to control stuff from home, follow the Republican Motto: Just say no.
The operational engineers are oblivious to security
Its like saying Our bus drivers are oblivious to road safety. If they don't know how to do their jobs then you are screwed.
Put an air gap between your SCADA system and the internet.
http://michaelsmith.id.au
I personally witnessed Samba root level shares on SCADA boxes at an oil refinery in Brisbane. As far as I could tell the SCADA boxes were on an intependant network but were fully reliant on no internal security.
Posting anon for obvious reasons.
Seriously scarey.
I might be repeating what others have said, but I found this looking to find out what SCADA is.
http://www.theregister.co.uk/2010/11/02/scada_search_engine_warning/
A search engine that indexes servers and other internet devices is helping hackers to find industrial control systems that are vulnerable to tampering, the US Computer Emergency Readiness Team has warned.
The year-old site known as Shodan makes it easy to locate internet-facing SCADA, or supervisory control and data acquisition, systems used to control equipment at gasoline refineries, power plants and other industrial facilities. As white-hat hacker and Errata Security CEO Robert Graham explains, the search engine can also be used to identify systems with known vulnerabilities.
According to the Industrial Control Systems division of US CERT, that's exactly what some people are doing to discover poorly configured SCADA gear.
Is anyone else terrified by the fact that this question even had to be asked?
A latent existence
For more direct advice:
1) discrete network firewalled, ideally air gapped, from the "corporate" or normal network. This is a single function network.
2) strict controls on media usage as well as protocols on how to use
3) strict config management and change control
4) physical protections to "local" and "remote" systems (RTUs, PLCs, and IEDs (note: IED = intelligent electrical device), you really don't want to build a secure control room and then get back hacked from a field device!)
To actually learn more, Idaho National Labs has the National SCADA Test Bed Program (http://www.inl.gov/scada/) and they also have control system security workshops/training programs. Their Advanced SCADA Security Training is pretty eye opening, and that's coming from the perspective of an IT security guy. Your normal operators and operational engineers will likely be blown away by it.
Like I mentioned, coming from the electric sector, I know what your facing (technical as well as cultural issues) and feel you pain. Good luck, and know you are not alone out there... just a minority! ;)
Don't connect any important computers/networks to the Internet. Problem 80% solved, duh.
I recently went to a Belgian OWASP meeting where Justin Searle talked about "Attacking and Defending the Grid".
This guy knows what he's talking about. Among other things, he also mentioned SCADA vulnerabilities.
I recommend you contact him or his company for professional advice (I am in no way affiliated with him or his company - just thought you might be interested)
If you google for the subject of this reply in combination with "OWASP", you'll find more info about the talk.
Contact other municipalities using SCADA. Ask them what they did. Ask them for further advice if they seem knowledgeable.
I'm working with an international firm on Scada - we use a VPN to provide a secure private network.
SCADA systems are not designed, implemented, or operated with network and application level security concerns in mind. :)
(Usually. The exceptions know who they are
Your compensating control is physical security to limit access to SCADA elements and programming. It costs more, but you have no sane alternative.
And before you get too cocky about that restricted air gap, consider Stuxnet turning such a strength into a weakness for exploit. At some point SCADA systems will be security conscious; that day is not today...
www.ics-cert.org
That is all.
invite Russian, Asian, and Middle eastern hackers to attempt to bring down the network and system then find all the points they attacked and fix em
There are many different types of SCADA implementations and several options for remotely accessing them - RF (VHF or UHF radio telemetry), cellular, network (wired, or wireless bridged), etc. The answer is simple - if a disaster would result from unauthorized access you need to find a consultant that knows enough about the implementation to keep the bad guys out. Spend the money and get it done right. I've often wondered about some of the SCADA implementations over VHF or UHF radio telemetry and what the person was thinking when the system was put into place, that's as insecure as a CB radio and probably controls some critical infrastructure that people depend on. This whole topic is going to keep resurfacing for years to come, and recent events such as Stuxnet proved this is an area to be concerned with.
Hire a competent auditing and security assessment firm.
For a rate (probably around $8-$10k per week), you can have a skilled team of security assessors come in, do an analysis, penetration test, etc, and provide a report on the likely weaknesses in any existing design.
For a similar rate, you can hire a different team to come in an consult with you on building/fixing the system.
It's expensive, but it's hard to beat the level of expertise that comes with a group like that. Just don't go hire some guy in his basement, be sure it's a reputable firm. Ask about the variety of the consultant's experience and how many other SCADA systems they've worked with, etc.
Good luck!
I love that one of the reasons ppl use SCADA is that is so much easier... say that again it thirty years. I have seen offers where ppl "guarantee" that the computers will last that long. Yeah right. Go for a PCL set up and hire some grease moneys that can crawl around and fix stuff. You will be much happier porting PLC code than trying to reconstruct some pre-made "modules" in software you don't have access to the source to. I have seen one module produce 600+ lines of code! Those flatscreens are pretty now but you will probably have to replace the whole system in 10-15 years. (unless something breaks... You did buy spareparts for 30years in the project, right?)
Here is the most important thing to know: security is a process not a project. No systems ever "achieve" security. Rather, better-protected systems beat the attackers by having well-thought-out-and-executed security processes.
Anybody want a peanut?
On the regulatory side, for networks the NERC Reliability Standards for the Bulk Electric Systems of North America address similar concerns (including cyber security) in electrical grids. For highly integrated systems MILS kernels are an engineering solution e.g. to keep actuators and monitoring subsystems apart.
Seriously ? Were all doomed, man ...
There are a variety of good posts here (among the chaff). The post by @bigjeff5 and the anonymous coward post amendment. For standards and an understanding of the risk metrics Sandia labs has a great set of documents for SCADA security http://www.sandia.gov/ccss/ , never mind all the FUD. You'll have to decide on whether you want a best in class, good enough, or what you can afford and wherever the three vectors meet at a solution. Technically there is no reason for SCADA to be a risk. Experience though tells us there are plenty of reasons to push the SCADA operational component into the risk category. Not being able to afford to keep the utility operational engineers employed because the technical SCADA solution cost three times your budget is the risk I usually see. What you'll need is an experienced person to act as a trusted third party and there are a lot of them out there in the real world. Be wary of people who talk about security, technical issues, operating systems, and other elements in black and white terms. They rarely have the real world experience to understand real world issues in implementation. Since you appear to be talking about water and in the United States (pardon if not) you are likely highly regulated. You will also need to balance the new requirements and regulations for implementing SCADA devices too.
--- Location Unknown
I work for a company who designs SCADA networks for water/wastewater clients. We rarely connect the SCADA network to the internet, but when we do, it takes a lot of time and money to do it right. Hire a firm who specializes in SCADA security, you can't do it on your own.
Nice to see my dark expectations of the IT security industry confirmed. Not that it was hard to guess, mind. But still. As for the original question, well, there's these three envelopes that you probably found in your desk drawer....
Anyway, one of the big questions of our time is HOW THE HELL DO WE SECURE ALL THOSE SYSTEMS?
And we won't have an answer for at least another decade. Best get cracking then. I'm getting the feeling the only long term solution is to basically rewrite everything from scratch and this time care about securing and input handling, and take building the rest --networks, use environments, the works-- from there. That, and learning how to secure our policies and procedures and rewrite those from scratch, too.
Another big question of our time is HOW THE HELL DO WE PROTECT OUR PRIVACY?
Which in a certain light is the same as the previous question. Try it. Now consider that some think privacy and security to be diametrically opposed, and taste the delicious irony. Yeah, they're wrong, so wrong they don't know how wrong they are. Franklin, however, was right. But he's dead and we aren't, so it's up to us now.
As an elected official, you have standing that the average citizen does not. Contact your local power company (if it's a co-op, contact their supplier) and tell them who you are. All major power companies use SCADA and you'd better believe security is a high priority with them. They can point you in the right direction quicker than anyone else.
Can any Slashdotters recommend ways to make sure it is secure?
Don't connect it to the Internet.
The operational engineers are oblivious to security
Fire them and hire someone competent. :P
And don't install antivirus on the secure network(Not a joke!). To have an effective anti-virus you need regular updates. Those updates need connectivity. That connectivity is exactly what everybody says you have to limit (or connectivity should not exists according to others). And don't forget AV somethimes give false positives. You never want your mission critical software affected by AV.
Besides that, dont go for any automated update on your mission critical network. For any updates you might need apply them to a test system first.
I've recently been through this in a mid-sized town, some recommendations from experience.
1) Put the workstations and primary logic controllers on their own fiber ring. Make sure there are no gateways to other vlans.
2) Disable all optical drives and usb drives through a group policy (a MUST) on the workstations.
3) If remote access is manditory, use a device like the BlueTree cellular modems connected to the network via an IPSec/GRE tunnel which can provide remote access via dedicated mobile devices (again disable all nics, wireless adapaters, optical drives, usb drives and use a cellular modem included in the tunnel).
Hope this info helps.
If you don't understand the issues well enough to do an audit yourself, insist on an outside audit by a company that does, *and* does not sell any security products or services themselves.
Asking Slashdot is just going to get you a whole lot of uninformed suggestions from mostly well-meaning people with the occasional good suggestion buried in the noise. But you don't know enough about the subject to know which ones are the good suggestions and which are not.
I think treating security as a checklist item with regard to control systems is a bad idea, and it's been my experience that when someone starts crying for it it's really a people problem.
It's been my experience that SCADA system operators are very no-nonsense people who prefer simple and effective measures to more complex ones with better long-term maintainability. There's something about this approach that drives security-minded people who aren't used to the industry right up the wall, me included. I used to believe that was the wrong approach until I started to have to work with the customers directly, then I began to realize that the sort of security usually found in large networks actually reduces overall system reliability in most cases. I've seen some pretty spectacular disasters resulting from simple things like PKI certificates expiring and time synchronization breaking Kerberos.
I've thought a lot about this and over time have realized that sharing anything -- whether it be HMI control or a bank account or even a book -- increases in complexity as trust between the parties decreases. It's a fundamental engineering principle that if you eliminate a need for a system component it can't fail, and there's no reason why that can't be true for a SCADA system as long as you have good perimeter security. If you can't trust your system operators, what you really need is better management tools like auditing, metrics, and training, not a vague "security" requirement. One terrible thing about treating it as a checklist item instead of part of the overall plan is that you often don't get the funding required to train the staff how not to screw up the computers and have spare machines they can screw around on so they don't break production machines.
One of the nice things about the industrial control industry is that there's often very good job security. Many of our customers know who we are personally and it's very unusual to talk to someone at a customer site that you don't know. Sometimes the control operators are a few fries short of happy meal, but quite honestly the only time we've ever had security problems has been when someone misconfigured a firewall and put some operator terminal out on the Internet. The sites where the line staff are managed well and trust each other enough to simply share the admin passwords with each other have never -- NEVER -- had any kind of network security problem in several decades.
Like I said, there's something about that which drives people with traditional network security backgrounds completely bonkers, so I'll leave you with the following point: the technicians who might get that admin password may make a few mistakes early on and it's going to be annoying. You then have a choice. You can either lock them out and deny them the opportunity to learn from the experience, or you can take heart in the fact that they're the ones who are going to have to drive out and fix whatever unnecessary screwup they caused, ensuring they're never going to want to repeat the experience. There aren't very many things you can do with a SCADA system that are actually going to destroy equipment if your control engineers did their job right, so my advice is to just give the technicians administrative access, airgap the network, and spend your testing budget on seeing if the staff does well at complying with procedures to resist social engineering. Throwing more money at security products has always ended badly for us.
Control systems need to be on a completely isolated network with extremely strict controls over how data and programs get onto the network.
Do not enable any USB devices, no firewire, no floppy disks and definitely NO EMAIL. There show not be any way for an average user to bring any data in or out of the systems, PERIOD.
Physical security controls are critical too. The systems need to be in controlled access locations - no following.
The physical systems need to be locked shut so someone with a screwdriver can't open them and add another HDD without authorization too.
Oh, and don't run Microsoft operating systems. If you don't run Windows, then almost none of your people will try to violate data security to load the latest "Elf Bowling" game. http://en.wikipedia.org/wiki/Elf_Bowling
After the first person violates the rules, FIRE HIM/HER. You'll only need to do it once.
Have someone perform a risk assessment on the system - and focus on the quantitative aspects (ie what the cost to the community will be if it fails). Make sure that the contract has compensatory and insurance options in excess of those amounts, so that it is in the vendors 'hip pocket' best interests to ensure it does not fail. And of course make sure that the contract has provisions for review, should the potential impacts change or the vendor changes company name, is bought out, etc :-) (yes - i've seen that happen)
You could also have someone do a thorough risk analysis of the system (google up the NIST SP800-30 document) as well as have them supply a complete inventory of hardware, software, and services they will be using to deliver the solution. Again, NIST have an online database where you can look up what vulnerabilities are known for some IT products.
Have the vendor perform a detailed risk analysis of the system - see what they think are problems, and what are not. Where you see gaps - ask them and see what color their faces turn.
Have a look around to see what failures or disasters you have seen in SCADA systems, refer those scenarios to the vendor, and ask them what technical measures they have taken to ensure that a similar act could not happen to them
You should also have your own people clarify and document their own roles and responsibilities with the system - don't assume that you have the resources on hand to manage your side of the situation responsibly - again a risk analysis will help out there.
And of course get it all in writing.
First advice: do not connect the network that runs the SCADA systems to the Internet or to any network that connects to the Internet. Leave an air gap between your control systems and the outside world. It's OK to network them, but make it an isolated, stand-alone network. It's much harder to attack a network when getting access to it requires physically going to one of your offices or plants. It may make it less convenient, but remember you're making it less convenient for the bad guys too and the consequences of a successful attack probably outweigh any inconvenience in having to bring in updates via USB drive or the like. And don't let the vendors cow you. It's your system, after all, and you that's going to be responsible if an attacker causes a problem. If they insist their stuff absolutely needs access to the Internet, ask them point-blank if they'll be willing to sign a binding contract taking full, 100% responsibility for any and all costs stemming from a successful attack on the system from the Internet. If they won't, ask yourself whether you want to be their fall-guy.
It's possible to firewall things thoroughly enough to have indirect connection (eg. the SCADA network connects to your office network, which in turn connects to the Internet, with firewalls at both boundaries), but it's dangerous. You'll need an expert network engineering and design team to do it, and the first thing they're going to ask you is why you need that connection. If you don't have a really good answer for them, you probably don't need even that indirect connection.
One useful consequence of the Stuxnet Trojan is to provide a concrete example of how SCADA networks can be exploited. Knowing the details of how Stuxnet works can inform you of how to perform a comparable variety of penetration tests on your own network.
You will then have a measure of how vulnerable your network is to Stuxnet in particular. It's only a circumstantial indicator of how vulnerable your network is generally, which is why I'm not in favor of pen testing except as part of a more comprehensive security initiative. But it can be politically useful at the outset to have some pen testing results to share with management, and politically useful at the conclusion to show that something was measurably improved.
Protective measures directed against Stuxnet alone will not improve your security in general, but if you develop very good general security processes, you should observe that they effectively protect against Stuxnet among others. Security is hard because you can't prove the nonexistence of vulnerabilities. But you can certainly put measures in place, and you can test their effectiveness against known threats.
As an example, you've seen advice here to put some kind of strong isolation between your corporate network and your SCADA network. The extreme example is to have no connection of any kind, whether through firewalls or bastion hosts or whatever, between the networks.
That's good advice. You'd be crazy to ignore it. But you'd also be crazy to believe it sufficient. Stuxnet is a Trojan Horse. Among other strategies for getting onto your SCADA network, it rides on USB sticks. Your network could be completely isolated and yet it would be still vulnerable to attack. A slightly more sophisticated variant of Stuxnet would actively try to build a connection from your SCADA network outward. Since most network design is not geared to protecting against access from within, and since systems on your SCADA network have to be maintained at least occasionally, chances are that the network will be compromised.
So, it's no joke. This is a tough problem, and probably the toughest part is protecting against inside workers from defeating your security measures for the sake of convenience. Get professional help with this, and if I were you I wouldn't trust a single security contractor to get it all right. When you write up your RFP, do it in such a way that it doesn't bind you to selecting one winning bid.
Parity: What to do when the weekend comes.
Keep all computers off that network that are allowed on ANY OTHER NETWORK. The ONLY possible exception should be a single billing type system, where it pushes data from scada to the billing system (with the pushing system not allowed to have anything else on it). Also, all systems on the SCADA network should not have a single wifi on them.
I prefer the "u" in honour as it seems to be missing these days.
As to threats specific to SCADA, look beyond me
A smattering of ideas:
A. Check out OReilly's books, Bruce Schneier's material (schneier.com), and the National Security agency among other resources.
B. Physical security first: Preventing the next StuxNet is pointless if someone can hop the fence and pour toxins in the treated reserve for residents. Same thing with someone stealing the computers and hardware.
C. Realistic threat assessment and appropriate countermeasures.
Although possibly not actual security
i. When I worked tech support for rural convenience stores, the most common related issue we had
was thunderstorms damaging equipment.
Heavy duty surge suppresors, NOT necessarily backup batteries, but something to limit power surges.
(In my case, I didn't care about outages or a couple corrupt files, only that the system would work when the storm was over.
Your case may be different: Does the system shutdown gracefully? Or will it allow the treated and untreated water to mix?
ii & iii. internal threats & external threats: an employee padding their own hours
or a jealous person padding someone else's hours so bosses come down on them.
to the second: ensure, each person has access only to what they need and no more.
D. Considering that completely cutting off the internet is often unrealistic:
What about two connected networks. One with no or limited internet access. Then restricting access between the two to a degree?
E. regular audits: computer security and financial.
F. It almost goes without saying, but antivirus/ security suite.
Check Consumer Reports for the most effective. They do this crazy idea of actually testing the software for effectiveness against various known and unknown threats. (Last I heard, the percentages, were disappointingly low)
G. Default/ blank passwords are evil.
Sensible password policy: required changes every so often. A certain complexity- single word passwords are too easy to crack. I personally, take a sentence and use the first letter of each word in the sentence. Then throw in some digits and a punction mark or two. Ideally in the middle somewhere.
H. If feasible, and certainly something to consider over the long run:
Consider migrating to a Unix/ Linux/ and/or MacOS environment.
Likewise, avoid Internet Explorer. Firefox is much less problematic. Of course other options are avaiable.
IMHO,
In reality, if it's a small enough city, terrorism, is probably the smallest worry. They may want larger targets.
It's incredibly common to overestimate some risks and underestimate others
-Nick H.
SEL does make encrypting transcievers (302x). Depending on your SCADA system, I would not yet recommend a SEL device, such as the 3332, especially for untrained engineers. What SCADA system is your company planning to employ? Have you looked into adding a dedicated phone line to keep your information off the network? Have you contacted a local power company who very often employs secured SCADA systems?
The U.S. Government fully understands the need for isolation and just how impossible it really it. There are niche companies out there that make systems that comply with specific DCID 6/3 requirements to make the system match a Protection Level. They use mandatory access control with Solaris 10 Containers, Trusted Solaris/Irix before that, and SELinux nowadays.
Here's their problem though. In order to be effective, an organisation must clearly know what must come in or out, network wise. It is difficult, technically speaking, and managing such an interface point is a speciality either run by expensive people or by cheap, clueless dimwits.
As Bruce Schneier has pointed out, liability laws need to be in place because the market will not apply the proper controls, if for nothing else, then for cost alone. Folks may complain about PCI or SOX compliance and how it doesn't really make things safer and I agree because it just forces compliance but doesn't make them want to be compliant. Companies that are able to equate vulnerability with a decrease in stock price will find themselves motivated to make it right. The fear of lawyers can be pretty good motivation to do the right thing.
Here's my recommendation. Provide an incentive for passing an inspection. Provide an incentive for the inspector. Then clearly set the rules of the competition. The incentives are not based upon a "failure to hijack," but upon an ability to control an intrusion. The inspector does not get incentive for penetration, he gets incentive for control after he's in. The integrators need to pride themselves in limiting the damage that can be done. If they keep the installation simple and easy to understand, then it's harder to find sneaky ways in.
Meanwhile, light one up and pass it over 'cause I'm not holding my breath.
The town I work for has a SCADA system for it water treatment plants and lift stations. A lift station pumps sewage to the wastewater treatment plant. SCADA (at least in my town) has two main components. The first component is a control station which is a Windows XP SP2 PC. A read-only monitoring terminal is also a Windows XP SP2 PC. The second components are several small boxes inside cabinets to which various sensors and radio links are attached. Each lift station, water treatment plant, pump house, etc. has a SCADA cabinet with the small box in it. The sensors are usually RS-232 or RS-483 and connect via RJ45 adapters to the designated ports. A Radio link is in each cabinet. Each SCADA box has an ethernet jack to connect it to a network. The lift stations and pump houses don't have a network connection back to the town's network so those stations are fairly secure.
When I started at this job the SCADA system was on the same network as the town's PCs. I fixed this by moving SCADA to its own network with no internet access. It took several days and alot of cabling (the remote terminal was the hard part) but I did it. Two weeks later there was a problem and the head water person could no longer remotely access the system to fix it. I compromised and allowed VPN access to the SCADA system. A totally non-Internet accessible SCADA system is impossible. Even if you have someone monitoring the system 24/7 on site so no VPN access is needed, your Frame Relay, T1, or other connectivity options are certainly internet accessible.
Next we have 9600 baud radio links from each remote station back to the main SCADA control station. I have no way of knowing if the information over these links is encrypted or not. Siemens says the information is encrypted but I can't verify it.
Siemens also loves XP SP2 and refuses to support us if I install XP SP3, patches, or anti-virus on the system. I can't even turn on the windows XP SP2 firewall. Siemens also seems to love Symantec PCAnywhere. Every PC that has Siemens software has Symantec PCAnywhere installed it. Versions range from 10 to 12. We just had a third SCADA PC installed and it is still XP SP2 and PCAnywhere 12, not even 12.5.
The best I could do was to physically isolate the SCADA systems to their own network. Allow VPN access only to the control station. I installed RealVNC server on the control station and put a password on it. I setup a laptop with VPN client software and the RealVNC client. So the user connects to the VPN enters a username and password (password changes every 6 months and it is at least 10 characters long, mixcase alphanumeric password), launches the VNC client and enters the VNC password (not the same password as above but uses similar specs).
I look forward to reading the rest of the posts.
I am not a professional security technician. I have no formal security training whatsoever. I do not consider myself a hacker (even though friends and colleagues would likely think otherwise). I have been using computers since I was 8 (I'm now 30), and I read a lot, including security discussions, books, papers, and most anything else I can get my hands on. My personal, non professional advice is the following:
1. *Anything* with an external connection can likely be hacked. Deal with it. A totally secure system *can not* have a connection to the internet, period.
2. If you're dealing with more then 1 location, the "suits" will require they can talk to each other for multiple reasons including monitoring, etc. This inherently violates rule #1. Get used to it; they aren't changing any time soon.
2a - If you really, really have to do it, send up *1* tunnel, restricted by IP address, requiring 4096 bit key handshakes and any other thing you can possibly think of to be *damn* sure you're talking to the right equipment.
2b - *Anything* else, including the head sysadmin "who just has to be able to get in from home when needed" is denied. Flat out, period, denied. If it is that critical that an issue can be fixed that quickly, you need to have an employee on site at all times to do it themselves. Otherwise, quite frankly, it can wait the 30 min for someone to drive in. Even if its an inconvenience.
2c - Make sure remote connections are limited to the absolute minimum functions needed, regardless of who logs in. This is kind of weird, but even once they "authenticate" themselves, assume they are untrusted. I don't care if root, a wheel account, allah or the company president logs in under their login. If a remote connection is needed for monitoring, make sure they can only do read only access. Restrict *everything* by the absolute, bare minimum function needed for that specific task and connection and assume your own machines are untrusted. Eventually, they might be. This requires a *lot* of foresight and planning if done properly.
2d - Make sure the local routers have source and destination filters in place. If you absolutely have to set up a tunnel between two locations; make sure each one only has a single point to talk to each other. Routers on both sides should be configured to automatically drop *all* packets with an unknown src or dst IP address. These particular routers should let *absolutely* no other traffic through.
3. As previously stated, use internal only non-routable IP addresses. There's no reason whatsoever for any kind of internal command and control interface to have wan-routable IPs. This is a no brainer, but it has to be said anyway.
4. All generic logins are required to be changed on a periodic basis, no more then 30 days. Passwords should have a minimum strength algorithm automatically applied to them when entered, and rejected if needed. No, "p4ssw0rd" is not a good password. I don't care how many checkers it passes. "Gr3If%&3F" is a good password. Passwords can not match any of the last few passwords used. This is to help mitigate damage in the event of an old password getting out. Specifically employees are accountable to changing the passwords every x days (preferably 30, no more then 90) (and documenting it), and changes should be automatically logged to a remote server (see below). A general password reset for all critical systems should be required any time an employee leaves the company, especially anyone in an IT role. Notification systems should be put into place so the IT staff is automatically notified when someone leaves the company and minimum response times implemented for their logins being disabled and general passwords being changed. There's no excuse for an employee terminated 7 days ago to still know valid logins aside from laziness or incompetence.
5. All logins, password changes, configuration changes, etc get logged to a remote logging server, that allows *no* remote logins period. Ideally, this will be locked very securely in the he
You need to hire a control system or SCADA engineer to assist you to evaluate the overall requirement.
Depending on your operation you might just need a isolated network to more sophisticated requirement. You mention that you are in a small town and runs a small water treatment plant, so that tells me that a. you have a smaller size facility with only a few components (versus big facility for a big city), and b. seems that this is your only facility to operate, or one of very few. The requirement depends greatly if you are operating within the facility (inside the yard), versus if you are hiring an external company to monitor and respond to the day-to-day operational needs, which a lot of small utility district tends to do today.
I can also tell you that you should look for the outfit that are more specialize in working with water utilities instead of specialize in other industry. Different sector have different requirements. I can tell you that your safety and cost-effectiveness requirement will not be the same as the oil and gas sector, or manufacturing sector. There are plenty of consulting company or control integrator who are familiar with the physical as well as legal requirements you would need. I can also recommend you to look for someone with an PE license of your state as well.
Lastly, your network security does not trump or supersedes your safety requirement of the facility and of the general public. In fact, the whole reason network security is of any factor is to protect your facility from the potential harms that can not only take down the operation of the facility, but also cause harm to general public. Automating your facility with control and SCADA system can enhance the operational safety. It does sounds like your "operational engineer" is not a control guru, so you need someone who is knowledgeable of the subject matter.
Before you begin, revisit the mission statement for the utility. Decide now what kind of disruptions you're willing to tolerate and how much expense you're willing to fund before anyone brings you any kind of technology. Make sure the costs of disruptions and impact to the community is considered in making that decision. Check policy and local/state/provincial laws for potential impacts from outages, and include those impacts in your analysis.
When the analysis is complete, check with public relations to see what they think the response will be. you may need to revisit that decision. When you have a safe way to proceed, loudly and repeatedly publish your decision on how you intend to proceed and what your goals and tolerances are. Make sure the utility people know it and can repeat it in their sleep. Make sure the vendors know. Leave no stone unturned.
When you're about to start buying things and collecting bids, be very, very careful. If you expect the solutions you buy to do something in particular, make sure you word it that way in your requests for proposals. Don't say something like "must be able to log to our log servers" when you mean to say "must log to our log servers." Vendors will seize upon the former and, when you see they're not doing what you want they tend to respond, "well, it's *able* to do that but it'll cost an additional 27 bazillion dollars for a pro services engagement. It doesn't do that out of the box." Small utilities have a hard enough time with finding money without having to put up with that kind of silliness.
When things begin, you will hear from two types of people: Control Systems people will repeat whatever the vendor tells them about what good security is (if they mention it at all), and the vendor will want to connect a lot of stuff to the Internet. They may also want to remotely access your control systems for support and maintenance. There are sometimes reasons for this, but it's almost always an extraordinarily bad idea. Vendor back doors and B2B connections into control systems usually lead to unauthorized changes and unexpected behavior. The next thing you know, you're dependent on the vendor for everything and they can start setting the contract renewal terms. This is bad and becomes costly.
Vendors and Control Systems folks also tend to focus entirely on encryption (and sometimes access controls) as the extent of security, totally ignoring data integrity and often hand-waving availability. Software vulnerabilities that lead to denials of service are an availability issue. Poor authentication and unencrypted protocols are a potential data integrity issue. Don't let the vendors or the Control Systems people slide on those- if they have integrity, encryption, and authentication controls, make the vendor implement them.
Make the vendor come on site to do maintenance and put your foot down with the Control Systems people who want remote access- if it's important enough for need their attention in the middle of the night or on Thanksgiving, it's important enough for them to drive to the site where the equipment is and deal with an issue. Utilities can usually pay for maintenance and overtime, but would rather not pay fines for an outage or a safety issue.
IT Security people will want to do all kinds of intrusive and disruptive scans on the systems, and they like to keep things patched. This is important, but sometimes patching can break the systems that the Control Systems people make. The reasonableness threshold for patching is usually tied to the most recent version of the software and patches that are available from the SCADA vendors. Absolutely keep the vendor's software and firmware patched to the most recent levels, and don't patch the OS beyond what's supported by the vendor. That sounds counter-intuitive to a lot of IT Security folks, but that's the way it is (hence the tedium about isolation.)
As for the collection of Control Systems folks and IT people, they'll need some way to
The NERC CIP regulations, which may or may not govern your security requirements, are a good place to start learning.
This is Slashdot. There are many self styled experts here. Some know what they're talking about. Many do not. Tread with care.
I am a registered professional engineer with 25 years of experience integrating, fixing, and designing several generations of SCADA and plant control systems for a large water and sewer utility. I not only design and build these things, I live with my creations through the entire life-cycle. If it does anything unexpected, they call me; no matter how old it is. I have worked on every aspect from the field to the operations control center and every single detail in between. I am not just an engineer, I write software, including protocol drivers, embedded firmware, and system management scripts. I designed the networks we use and I often show our IT department some of the finer points of network design. I'm not a consultant. I'm not selling anything.
As with safety, just as there are no perfectly safe systems, there are no perfectly secure systems. We do not have testing procedures or magic text books that we can throw at this problem. However, for suggestions, I recommend ISA-99 or NIST SP 800, if you do any work on behalf of the Federal Government in the US. In particular, see NIST 800-82. I should warn you, NIST 800-82 is a smorgasbord of suggestions. I think if someone tried to implement all of these notions you'd have a nearly unusable system. At some point, you have to stop securifying and safetying everything and just educate staff what the risks are and what to look for.
If you're really trying to do SCADA, stay away from Modbus. While Modbus is a good DCS protocol, it is a poor SCADA protocol. Too many engineers still specify Modbus for SCADA because they know nothing else. DNP3 is an event oriented, near-real-time protocol. It is a far better SCADA protocol. If you don't understand the distinction, find a consulting engineer who can explain it to you. If your present consulting engineer has difficulty understanding this nuance, find another one. DNP has been in wide use in the electrical power industry in North American for over a decade. Water utilities in Australian and the UK are using DNP. It also has new features for secure authentication. See IEC 62351-5 for details.
On the other side, Office IT is not the same as a real time or near real time IT. I know of at least two books explaining this difference. If your IT staff do not understand that this application is truly different, find someone who does.
Understand that control engineering is a very broad field, embedded IT is a very broad field, and IT security is a very broad field. If you put the experts in the same room, make sure you define the problem for them very carefully. These fields tend to make prima-donna characters out of otherwise mild mannered people. I can guarantee a very heated argument after a short period of time and even a fist fight is not entirely out of the question. We do not have room for this kind of behavior.
The IT security people need to understand that while IT systems can be restored from backups, physical systems can not. They also need to understand that we do not spray patches at the field on a whim. We do not have broadband to a manhole in the middle of the street. And we use common carriers with extreme caution and evaluation. Likewise, the engineers need to stop thinking about building things and start thinking about how to break them. Too many engineers assume that nobody in their right mind would mess with their wonderful creations because it would clearly break something. That's like designing cars without considering what they'll do when they crash.
I suggest getting familiar with the various standards efforts. Also, I suggest keeping track of what is going on with SCADA security. Note some of the following
(Shameless plug alert)
I am a co founder of SCADASEC, an email list for SCADA and ICS security issues. I am also a member and contributor to the DNP3 and ISA-99 standards committees. These are many popular efforts. The NIST SP 80
Nearly fifty percent of all graduates come from the bottom half of the class!
http://www.bcit.ca/appliedresearch/gait/projects/scada.shtml
give these guys a phone call, if their tool sucks for you, I'm sure they can give you some pointers
enjoy
-judging another only defines yourself
The University of Idaho has a research lab dedicated to this. They have SCADA systems set in a lab. There's a grad class in the subject you can take via video. And there are quite a few papers they have helped published. Search the IEEE document library for some good info.
As to threats specific to SCADA, look elsewhere.
A smattering of ideas:
A. Check out OReilly's books, Bruce Schneier's material (schneier.com), and the National Security agency among other resources. Schneir's
B. Physical security first: Preventing the next StuxNet is pointless if someone can hop the fence and pour toxins in the treated reserve for residents. Same thing with someone stealing the computers and hardware.
C. Realistic threat assessment and appropriate countermeasures.
Although possibly not actual security
i. When I worked tech support for rural convenience stores, the most common related issue we had
was thunderstorms damaging equipment.
Heavy duty surge suppresors, NOT necessarily backup batteries, but something to limit power fluctuations.
(In my case, I didn't care about outages or a couple corrupt files, only that the system would work when the storm was over.
Your case may be different: Does the system shutdown gracefully? Or will it allow the treated and untreated water to mix?
ii & iii. internal threats & external threats: an employee padding their own hours
or a jealous person padding someone else's hours so bosses come down on them.
to the second: ensure, each person has access only to what they need and no more.
D. Considering that completely cutting off the internet is often unrealistic:
What about two connected networks. One with no or limited internet access. Then restricting access between the two to a degree?
E. regular audits: computer security and financial.
F. It almost goes without saying, but antivirus/ security suite.
Check Consumer Reports for the most effective. They do this crazy idea of actually testing the software for effectiveness against various known and unknown threats. (Last I heard, the percentages, were disappointingly low)
G. Default/ blank passwords are evil.
Sensible password policy: required changes every so often. A certain complexity- single word passwords are too easy to crack. I personally, take a sentence and use the first letter of each word in the sentence. Then throw in some digits and a punction mark or two. Ideally in the middle somewhere.
H. If feasible, and certainly something to consider over the long run:
Consider migrating to a Unix/ Linux/ and/or MacOS environment.
Likewise, avoid Internet Explorer. Firefox is much less problematic. Of course other options are avaiable.
IMHO,
In reality, if it's a small enough city, terrorism, is probably the smallest worry. They may want larger targets.
It's incredibly common to overestimate some risks and underestimate others.
Howdy, The only way to secure it is with an Air Gap. A PC with a CDROM drive as the final "non-connection" to the internet. The problem with that is that the vendor will not like it, since they like to keep their grubby fingers inside your system remotely, so that they can see what you are doing and upload fixes easily. The last thing they want to deal with is a air gap CDROM.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
Every computer system that was ever designed, every software that was written was done to share information, not to secure it. That is until recently. It wasn't until we built all these systems and got them all working that we then went 'aw CRAP, what about security...'
The internet was designed by engineers. Software was written by engineers. They both shared the same flaw. Engineers build things. Nearly every engineer I've met operates on the principle of "If it ain't broke, don't take it apart and find out why!"
I am a security auditor - I've audited SCADA networks and have gone head-to-head with the engineers that designed and implemented them. To a person, they nearly all take it personal when I critique the security aspects of the network. Don't get me wrong - the networks are built flawlessly, but they are all glass houses. I call the SCADA networks "Crystal Palace" because nobody should be throwing rocks anywhere near them. Keep them as protected as possible from anyone who wishes to throw rocks... Isolate the heck out of them, then isolate them again.
If you enter into the project with security first and foremost in mind - you are WAY ahead of the game.
I suggest the following:
1) Have a security expert on hand during design and consult with them on installation. It may be expensive, but it is far cheaper than building it and then securing it after the fact (which often means rebuilding it because it is VERY difficult and expensive to change a system that requires 100% uptime).
2) At every step of design and implementation, ask the question "What about security?" - It will be annoying, but you'll be glad you asked.
3) The SCADA network is a glass house - protect it from anyone and anything. If possible, maintain an 'air gap' between the SCADA network and the rest of the world. Engineers that need to access that network from the 'outside world' must VPN into it - then use remote desktop or some similar technology to administer it. By 'outside world', I mean any computer that has internet access, or rather access to the internet - irrespective if it is behind the firewall on your own network.
4) Provide security in layers - never allow the SCADA network to access the internet directly or indirectly, ever.
You'll want to consult with a security expert - one with experience in SCADA networks.
Good security is based upon reality and common sense. Common sense is a function of having common knowledge.
I was involved (heavily) in updating a 911 system that used SCADA to send emergency reports (rip and run reports) to Fire Halls and Ambulance Stations. It would also activate PA systems, radios (they would get and alert and the message over the radio at the station which would be sent across the PA system at the station), station lights, raise bay doors, and could also unlock man doors for people who wanted access to the station while the crew was out and the doors were locked. We used local dedicated leased phone lines to send the data, and had a local protocol analyser to track the data signals sent. I had to wire up a lot of connectors as stations were added, as well as running wire, and then modifying software in the dispatch system and database. The easiest way to keep it secure (as already pointed out) was to keep it completely off the internet. We has a local control system that used a version of the Server Message Block protocol (SMB) to connect to the system. The network people would scratch their heads when I said it was for a unix system to monitor fire halls and not local files on a microsoft network. *IF* you have to connect it to the internet (and I *STRONGLY* recommend that you don't use the net), then encrypt the signals, use a long forward-error-corrected string for each command, and use automated traffic checking to make sure that data doesn't get messed with. Making the system redundant (a second net connection or better, a microwave broadcast), would be a very good idea.
use a dialup connection. use ssh encryption. block IP addresses after x unsuccessful attempts to login. Provide alarms to operators if many unsucessful attempts are made to login. use a cable modem from a local isp that's off the town network with a strong dedicated linux firewall. Use two factor authentication. (any activex object can be called from wonderware's solution, and these can include smart card protocols.) How much is enough? Well, how often are there break-ins in the wild with these operations? What are they doing? In any case, get a 3rd party to look at security, but don't get crazy about it. Don't waste tons of money! Use probability to your advantage. What is the allowable risk you can take? is one break in per 10 years acceptable, 100, 1000? How does each layer of security affect that goal? Unplug net connection if not in use? So many ways.... each adds some security. 1 in 1000000 that they know your dialup (a magicjack number in timbuktu?)? 1 in 10,000,000 that breaks strong ssh2 with keys? 1 in 10 that they get through a router? 1 in 100 that they crack the database where tag information is kept? 1 in 1000 that the alerted operator does nothing? Life is risky. Just stack some probability.
First of all, you'd better find out of you have to follow NERC/CIP regulations. If you are in the US, you most likely do.
Even if you are not, they are a good set of standards to follow: http://www.nerc.com/page.php?cid=2|20
Second, I think the keys are following procedure, 100% of the time, no exceptions.
One procedure we have added is that all DVDs and USB ports are disabled. We use tamper-evident labels. All network ports not connected to operational computers are shut down. There is literally no way to get software into our network without passing through two sets of firewalls. The first set of firewall scans and only allow access to a secured host where we drop off files. That host is a hardened Linux machine running secure ftp in a chroot setup that also has real-time AV (to additionally check). From there we download the software into our test network and write up the exact procedures to follow to install. We then run our tests to see exactly what changed and make sure it doesn't add any new ports or services (and/or verify this is the expected behavior and document). Then we revert our test machines back to matching production state and follow the step-by-step procedure and no others steps/changes allowed. Then we test again that the same results occur. Then we let it sit and bake and observe that there is no extra connections going out from the box/network or extra processes that expected running (a ton of "good" software insists on phoning home and/or checking for CA cert revokes). Finally, if it passes all of this, then another team follows are exact step-by-step process to install this on our production SCADA primary system (after first switching to our backup SCADA system), then we do our verifications to make sure all is functioning, then we switch to running to our primary SCADA system for the bulk of the day, and if all is well then we follow the same procedure on our backup SCADA system.
We never ran into problems with Stuxnet, but just as the Arizon utility that did run into it, they caught it due to odd behaviors that their SIEMS equipment detected. We'd catch the same thing. The only way we'd miss something is if it somehow ran so quick that our SIEMS equipment never had anything to detect and it "slept" for a long time, longer that we let things bake in a test environment (typically a week or more). But then, if something was really that dormant, I don't know how it would call itself up. Plus, we have Tripwire watching all crucial system files and registry settings, and A/V protecting Tripwire. I'm don't think there is a way to foil all of this.
Air gapping (as others have mentioned) is a great idea, but not always feasible. Remote access to plants is sometimes needed for emergencies.
Have a look at the Computer Security Resource Centre. NIST IR 7628 covers cyber security for the smart grid, and much of that is applicable to water and sewerage plants.
The report, 21 Steps to Improve Cyber Security of SCADA Networks, is also worth a read.
My opinions, based on power station and substation SCADA are: don't use one vendor for everything, have two levels of firewall from different vendors at the remote site, turn off or block services that are not needed from every device, have reporting and audit trails that are reviewed, and if management want reporting, do it through a one way connection to an intermediate system (one way Ethernet, RS232, RS485 or read only shared storage).
Simply put..
A. Do not under any circumstances connect this thing to the internet. no if, ands or buts about it.
B. If outside access is required it is by call-back ONLY.
C. EVERYTHING is logged, 100% no exceptions. Every keystroke, every entry, every response, every everything, and the logging is hard connected and encrypted.
D. Nothing is interconnected to your main business network. Place secure terminals that boot from the SCADA network that have NO media devices in them, no USB, No DVD/CD drives, no serial ports etc. in places where they are needed and the are hardwired back to the SCADA network.
The stuff above is a starting point and it needs more from there.
Hey KID! Yeah you, get the fuck off my lawn!
Build safety into the hardware. Make it impossible for the SCADA system to cause a dangerous condition. Anything that can go wrong, will. Murphy's law.
Use an isolated dedicated safety controller like http://www.ab.com/en/epub/catalogs/12762/2181376/2416247/7448535/
Also a simple safety loop circuit can work wonders; any break in the loop should cause a fail-safe shutdown.
Also for the love of god hire somebody who knows what the fuck they are doing because you obviously don't.
Document everything on how the system is built, what it's supposed to do, how it does it, who's allowed to access what posts and who's in charge. Go for posts (chief code wrangler, deputy system administrator) rather than names, but put those names in an appendix to the documentation and make it very clear that whoever's in charge has to make sure those posts are filled by trained staff with a suitable number of backups for when someone gets run over or comes down with rockin' pneumonia or the boogie-woogie 'flu.
Make it someone's responsibility to keep the documentation up to date so that when the auditors come in there's no embarrassed running around trying to find who the current chief code wrangler is.
Make it a rule that all changes to the system - hardware, OS, supporting software - are subject to formal change control. Nothing - absolutely nothing - gets changed on the system without going through that process, EXCEPT if it's a class-one triple grade-A emergency AND there's no time to get everyone's agreement. In those cases, the change control people, and the pereon in charge of the system, go through a formal debriefing afterwards to make sure the emergency call was justified.
The change control process includes a review of previous changes to confirm they were done on time and worked, or if not why there was a delay and was the system "rolled back". It may be likely there's a good reason for delays, they're not automatically a failure and could highlight the change system needs changing.
If any updates can be supported by one or more CRCs for each file, so much the better, it makes it harder for an altered program to be slipped in.
SCADA:
Supervisory Control = Being controlled by a supervisor. You cannot expect a supervisor to be stationed directly over the process. Therefore it is usually controlled remotely.
and Data Acquisition = being able to acquire data about your process. Collecting local data is silly, just look at your process. Collecting data remotely useful.
If you do not want security issues, do not install a system that was designed to be accessed remotely (S.C.A.D.A.). Just install a PLC that is totally isolated.
Bottom line, SCADA is rendered useless without an outside world connection. For security, drop the "S ADA", and stick with the "C" only.
Talk to these guys and take this class; better yet, drag your more enthusiastic SCADA guys with you to it. http://www.inl.gov/scada/training/intermediate_scada.shtml
It'll eat several days of your time, but you'll see things from multiple perspectives. You'll get suggestions on tools and references/websites to further your understanding of security.
I know the guy whose contact info is on the above page, and many of his coworkers. They're ramping up a CERT program for SCADA called ICS-CERT. They teach at SANS, they teach this class, they work together with tier-1 manufacturers of SCADA gear, and it has an current/former staff list that include some damn talented hackers.
And ignore Joe Weiss (and one of the posters on this page). The SCADA situation is steadily improving, yet if you ask Joe, the sky has been falling with no security improvements whatsoever for more than a decade. Reality would disagree. The situation is no less and no more grim than high-profile computer security (spearfishing and Shikata ga nai make my blood run cold).
http://openvpn.net/ would be ideal.
As the design life of these systems is usually 20+ years whether by being compromised or for some other reason at some point the SCADA/automatic control system will fail/become erratic, so instead I'd suggest you ask how they expect to cope with such failures.
To start you off
* Is there a UPS backed independent hardwired telemetry system to alert operators to control system failure, instrument failures and hazardous conditions?
* Is any chemical dosing controlled and logged by separate hardwired systems with physical security?
* Is there a hardwired fallback mode of control? (using float switches, level probes, timers and relay logic, etc to enact simple control)
* Are there local operator controls to run the plant in hand when the automatic control system has failed?
* In the event to automatic and hardwired control failure are the plant operators trained to be able to run the plant in manual?
* How quickly can you restore the PLCs?
* How quickly can you restore the system's last known good setpoints?
* How quickly can you restore the SCADA?
* Is there a gen-set, does it have fuel, it is regularly exercised, does it have automatic incomer switching, it that regularly tested?
* Are there exercised spares and someone to fit them?
You should take a look at the Trustworthy Cyber Infrastructure for the Power Grid (TCIPG) project (formerly TCIP). Although the research is mainly focused on the power grid, there is a healthy amount of research being done on securing and assessing the security of SCADA. For example, one activity TCIPG is working on is a proprietary protocol fuzzer called LZFuzz which is built specifically with SCADA in mind. I would suggest talking a look at their website and publications at http://www.iti.illinois.edu/content/tcip-trustworthy-cyber-infrastructure-power-grid
I think many here have the wrong idea. In just a few more short years it wont matter any more what we consider secure at this point. Things will change, so your system should not be so hardwired that it can't adapt. Even in nature those who can't adapt to change, will die.
What you should ask yourself is what do I need to do to get in? Then, how did I get the information required to get in?
Hackers use tools that we often overlook, it's called social engineering. If you need a code to get in how did you acquire it? and if you forgot it how would you recover it?
As for internet, well that is a boat all it's own. Just make sure there is at least two live people there 24/7 who can unplug or shutdown the system in case of suspicions activity.
Make sure everything is challenged and assume that no one is who they say they are even after being verified 5 times, make sure someone checks that all requests, and if anything looks wrong fall back on the safest option. Make sure you are getting very annoyed when trying to get in, the more annoying it is for you the less likely a hacker will want to continue their attempt to get in. In security annoying is better than efficient, if someone is complaining about how much security they have to go through to get in, then you probably have the minimum you need.
Then the final question what is there that someone would want to hack, most hackers just want to show off, but there is the occasional one that would want to do harm. For security reasons I personally would set up an alternate path something that looks complicated and has a high degree of security, and looks like the real deal at least digitally, and log all input, if someone get in and changes something then there will be a good probability that they will get into the wrong system, and any changes can be detected without harm.
And lastly have a fail-safe if someone does get in, and causes a worst case scenario, make sure you have a system in place to handle it.
Start with ICS-CERT (Industrial Control Systems - Cyber Emergency Response Team). Note: While they use the "us-cert.gov" domain, they are NOT a part of US-CERT.
Specifically, take a look at their "Recommended Practices: section: http://www.us-cert.gov/control_systems/practices/Recommended_Practices.html
A company called Tofino makes purpose-built firewalls for PLCs. They basically inspect traffic in and out of the controller and block traffic it is not programmed to accept. I saw a live demo and they seem to work pretty good, but I do question the effectiveness of it, since, AFAIK, most PLCs will ignore bad inputs anyway.
http://www.tofinosecurity.com/
Also, the priorities of a SCADA engineer are very different than an IT security analyst. The SCADA guy is looking for maximum availability and intergrity, while confidentiality is a distant 3rd.
Don't listen to random Assholes on the internet. How many of these people on Slashdot do you think are experienced in this kind of work? Being an IT turd is about as high up as most Slashdotters get. Go talk to professionals in the industry that work on security and deal with these problems every day. NERC, Seimens, Schweitzer, other Utilities. Sorry you seem to think you'll be capable of ensuring anything as an elected official. This is the pathetic adult version of "please do my homework for me... for free" go get some consultants in the industry if your core employ are incapable of handling the situation. DON'T just start asking around the internet. SCADA systems being internet connected is a requirement, anyone that thinks that SCADA is useful to anyone without being able to talk outside it's little box doesn't understand the first thing about utility systems.
Put it on the Internet and take steps to secure it. You will have to throw money at this because you don't have the in house resources to deal with it. You knew this already but you wanted to post online to make people think you were a good little politico saving the people money and going to the "experts". Slashdot is not the experts, though, and I hope to god I don't live where this installation will be servicing me.
They are a UK based organisation (I'm assuming that you are in the US), but they have produced lots of useful papers on risk assessment and risk management for SCADA. Full disclosure - I used to work there. A good starting point is here www.cpni.gov.uk/protectingyourassets/scada.aspx