Slashdot Mirror
US Electricity Grid Reportedly Penetrated By Spies
phantomfive worries about a report in the Wall Street Journal ("Makes me want to move to the country and dig a well") that in recent years a number of cyber attacks against US infrastructure have been launched over the Internet: "Cyberspies have penetrated the US electrical grid and left behind software programs that could be used to disrupt the system, according to current and former national-security officials. The spies came from China, Russia, and other countries, these officials said, and were believed to be on a mission to navigate the US electrical system and its controls. The intruders haven't sought to damage the power grid or other key infrastructure, but officials warned they could try during a crisis or war."
...you must live in perpetual fear. Whenever you're starting to focus on the reality of life, new fear WILL be injected into it to distract you. This is how the natural order sustains itself.
I'm sure China and Russia are having the same kind of problem.
"Some officials" come forward and warn about threats from China, Russia, Iran and North Korea. "Ya know, Sir, we need funding for enhancing national security, so please make sure you get your budget right."
They must have the CIP module !
Aren't these people just admitting that they were incompetent? That's refreshingly honest of them.
http://twitter.com/onion2k
I thought mission critical computers should not be reachable from the Internet. So the spies walked to those computers and planted the software there???
Colorless green Cthulhu waits dreaming furiously.
So, the week before a review is due looking into whether or not they should increase the flow from the money pump, "current and former national-security officials" have come forward to draw attention to a network of spies in the power grid.
Look, I'm not saying that cyber-attacks don't happen, or that there isn't a risk, but bloody hell, this article reads like a well-crafted piece of BS, designed to put the N back into FUDing.
'If Christ had tweeted the sermon on the mount, it might have lasted until nightfall.' - John Perry Barlow
The systems I work on are typically airgapped, but there is a constant push from users for some access to the internet. A user might need to access meteorological information, and the simplest way is to go online to get the data. Another user might need to refer to work instructions on the corporate intranet, but the intranet gets you to the internet anyway. Like it or not, the internet is working its way into many types of work and many people are starting to expect it to be available.
http://michaelsmith.id.au
China, Russia, and other countries,
So you mean there are people capable of hacking the US energy grid but who can't start the attacks from a hacked box in Madagascar?
"Who's attacking us?"
"Sir, the attacks come from half a million infected machines all around the world."
"From all coutries?"
"Yes, sir."
"So China and Russia too?"
"Hmm, Yes, of course, sir"
"Damn commies... We should've nuked them a long time ago."
Then I'd suggest they need two PCs.
Please read my Canon EOS tech blog at http://www.everyothershot.com
Trust me folks, it's coming. It won't be pretty, either. The power to disrupt a nation's economy via information warfare measures represents a much clearer threat than people trying to get something through airport security.
There's a reason the military is starting to get mighty interested in nerdy types, although most programs designed to leverage these skills are in their infancy. We need to get serious about this fast; other nations certainly are.
512 MB RAM, 20 GB disk, 200 GB transfer, five datacenters. $19.95/month.
AFAIK the whole remotely controlled stuff is not on internet or anything but on modem and similar box (can't remember their name) to which you have to directly dial in (non routable), and is separately powered from the power grid. If not I would fire the ass of the guy in responsibility: who in their right mind would put the control structure for a power grid, on something which can only be accessed when the same power grid is functioning. Also there are local control which override any possible remote control anyway.
C. Sagan : A demon haunted world:
http://www.amazon.com/gp/product/0345409469/
visit randi.org
The systems I work on are typically airgapped, but there is a constant push from users for some access to the internet. A user might need to access meteorological information, and the simplest way is to go online to get the data. Another user might need to refer to work instructions on the corporate intranet, but the intranet gets you to the internet anyway. Like it or not, the internet is working its way into many types of work and many people are starting to expect it to be available.
Then your users need two PCs and a KVM (or even two completely separate PCs - ideally on opposite ends of the desk - to properly drive the point home).
There are some situations where security MUST override convenience.
I actually do work with these exact systems. I have yet to install a system in a control room that had net access to the operator consoles or even the operational servers. These computers - yes, running Server 2003/8 or XP Pro - are patched to the latest and greatest before they leave our shop, but once on-site should never, ever, ever interact with the Internet.
That being said, the PI data servers are designed to be a go-between for the internal secure network, and the rest of the world so the data logging can reach those who need it. Not only does the PI server have security protocols built in, but is required to be installed in a DMZ with full firewall protections, and in some cases a dedicated leased hard line to an off-site office.
So, to summarize, no, the Op stations, the Op servers, should NEVER be connected to the Internet, and we do out best to disable any way of the operators even getting to the OS level, but there are times and reasons that you need to hook the internal network (through full security measures) to the outside world.
Everyone wants money for their projects. Part of getting it is knowing what to sell in your given field. Well, as of late with federal government dollars, national security has been the name of the game. Was more narrow to anti-terror but they are kind losing focus on that. So, it is also no surprise that is what people use to try and get the money, even if what they want really has fuck all to do with it.
For example Consolidated Edison wants to install a super conducting core in for New York's power grid. Reason is the existing grid has load problems and this looks like the best way to handle it, rather than massive amounts of more copper. This is expensive, of course. To the best of my knowledge when this is deployed, it'll be the first super conductor used for commercial power delivery. Means plenty of R&D in addition to the actual costs. Well, sure would be nice if the government would help pay for that... So they got them to.
How? Well they sold it to DHS as an "anti-terror" deal. No idea how this is supposed to be more terror resistant, but DHS bought it and that's what's important. They gave ConEd something like half the money they need for the project.
Now you know that ConEd isn't really doing this as an anti-terror measure, they are doing it as a "grid is overloaded" measure. However, they put that spin on it to get government funding, and it worked. I'm betting this is a similar money grab.
On one they're controlling the power station, on the other they're reading slashdot.
Unless typing 'FIRST POST! LOL' on the wrong box causes a reactor meltdown, I think we'll be ok :)
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
From TFA:
But protecting the electrical grid and other infrastructure is a key part of the Obama administration's cybersecurity review, which is to be completed next week.
Under the Bush administration, Congress approved $17 billion in secret funds to protect government networks, according to people familiar with the budget.
The Obama administration is weighing whether to expand the program to address vulnerabilities in private computer networks, which would cost billions of dollars more.
A senior Pentagon official said Tuesday the Pentagon has spent $100 million in the past six months repairing cyber damage.
Sounds a lot like someone is making up excuses and drumming up support to ask for more government money.
Mit der Dummheit kämpfen Götter selbst vergebens
There are some situations where security MUST override convenience.
Tell that to the union. Remember power industry operational environments are blue collar work places. I have seen people in similar environments go to any length to get a system they don't want to see shut down. They will play totally dumb, like not noticing they are using the wrong keyboard for hours at a time. Assume that your users are hostile to you. Then design a solution.
http://michaelsmith.id.au
Spy sappin' my generator.
Any system that needs to be secure should never be allowed access to any network that has public access. If remote systems "need" to communicate it should be done via dedicated leased lines. Even better since we are talking utility type SCADA systems here, why not have the utility lay fiber, line of sight microwave or what have you (as long as it is properly encrypted)? This way if the telco gets ganked, the leased lines can't be traversed.
"Nobody shoots anybody in the face unless you're a hit man or a video gamer"- Jack Thompson
Blowing all my moderation to reply to this.
Let me make this clear. Putting a critical system on to the internet is pure, stupid, incompetence.
ALL of your "situations" can be solved with a second $399.95 DELL sitting next to the critical workstation. Anyone saying that that is not practical is a blathering moron. I have seen MANY water filtration plants that the Supervisors in charge of the whole operation are so incompetent they put the entire plant's operation system on the corporate or city network. Then we have the low quality SCADA software called WonderWare that is so badly written that the company requests they have direct access to the machines so they can issue fixes faster.
If any mission critical machines are on anything but a sealed private non connected network, the person that designed it is a incompetent idiot that should take the fall for any failures. Gitmo time for whoever approved or asked for interconnection.
I have been appalled at the amount of interconnection I see in really important SCADA systems. I have seen this stupidity in major infrastructure control systems for 14 years now. Typically put here by some asshole manager that wants to "keep an eye" on his guys while he is at home. he get's a workstation (typically the one in his office) set up with a second network card and Pc anywhere or another Remote control system to interconnect the secure to the un-secure. and does it with a stupid windows box. Then the idiot uses it to check email, surf the net,etc... All installed by your friendly company IT slackie After the SCADA installation guys go home.
Every system I looked at that was "secure" typically had one of these bridging computers on it the only way to find the is to do a hard audit of every computer, the rate of finding these security breaches goes up as the age of the installation increases.
Do not look at laser with remaining good eye.
The module that allows outsiders to do this is called the CIP device. I hacks into the governments firewall. Who knew they had just one layer of protection over every bit of US infrastructure, that it's all linked together, and that it could be so easily circumvented by a genius hostage in a matter of hours?
But it's okay. A man by the name of Jack Bauer has been alerted to the situation. And knowing his previous record I'm confident that he will deal with the crisis, because all of the bad people operate within driving distance to him.
It is rather stupid to keep crying wolf, when there is little to nothing to raise the alarm about. Or, alternatively, it is very clever, if you want people to not take security warnings seriously; only, I can't see why anybody in America would wan't to achieve that.
Don't we hear these allegations all too often? It's "the Chinese and Russians" they say, and apparently it comes from the CIA or something, so we can't get to see any documentation. Perhaps some would like to think they can poison China's or Russia's reputations with this kind of stories, but as I point out, all they achieve is to weaken America's defence by undermining public trust in the agencies that are supposed to help protect them - it seems idiotic to me.
And objectively, why should China or Russia want to harm America? Like it or not, they are no longer likely to be enemies of America in a future, global conflict, which will probably be between the industrialised and developing nations. To my mind it seems more believable that the culprits are international criminal gangs; multinational companies have grown to almost nation-like power, and it seems almost unthinkable that international gangs haven't grown proportionally, especially since the introduction of the internet. They would certainly have an interest in staking out as much of the public infrastructure as they can. And, of course they might also see an interest in people not believing public security warnings.
ALL of your "situations" can be solved with a second $399.95 DELL sitting next to the critical workstation. Anyone saying that that is not practical is a blathering moron.
In all the control room environments I have worked in this approach is just not acceptable. The users expect to get a single, integrated UI environment.
http://michaelsmith.id.au
I'm afraid not, that was 20 years ago: I no longer have the originals. There were a set of published security updates for telnet and sendmail at the time, which the Morris Worm probably exploited on my systems: the vendors had not revealed all the exploit details. (Few vendors do.) We frankly didn't bother to do extensive analysis at the time, we had critical work to do and a lot of systems to rebuild, very painfully, from bootstrap systems that hadn't been tested in years and backup policies that I'd also written about as being badly scheduled and incomplete.
Having the "I told you so" documents on paper can be critical: they have much more power than mere verbal testimony. The fact that I'd kept them under lock and key and wouldn't let the originals out of my hands were an interesting source of internal strife, and revealed some other bureaucratic issues when other documents were somehow "lost" by the people assessing the situation.
If you were the designer, then you did not do your job educating them as to why they are not supposed to do that, and the repercussions for not following them
It is the SCADA system designers job to inform the customer as to the incredible danger of their desire to be convenient.
If you were a employee that worked at one of those stations, why did you never voice your concern about it? One word to the regulators and your bosses would have been screamed at and fined heavily for having an integrated UI for internet, SCADA, and email. Most regulatory commissions REQUIRE security and system separation.
Do not look at laser with remaining good eye.
The WSJ article was apparently triggered by a letter sent by NERC (North American Electric Reliability Council) to its members. I think it shows a healthy development of security digging down to yet another layer of depth.
Forget the major computers in the major control centers. That's what everyone thinks of first. At that level it is becoming like the Indians and athropologists in the Grand Canyon. For every utility cyber worker there seems to be 30 government gumshoes and overseers looking over their shoulders. One would expect no aspects of security to be neglected at that level.
The NERC letter refers to devices at a lower level. Primarily, what the industry calls "protective relays" in substations. From 1888 to a few years ago these functions were really done with electromechanical relays. Now, many of them have been replaced by digital equivalents on a one-by-one basis. In a household analogy, it is like the difference between a central electric control computer for the house, as compared to a "smart" digital LED light bulb. One worries about the central computer being hacked, but at first blush, not the light bulb.
The problem is that the engineers who deal with this level of equipment aren't used to thinking of these devices like the light bulb instead of like computers in a network. They have not identified many of these low-level devices as "cyber critical". The NERC letter urges utilities to change that culture.
This is an industry that owns and maintains hundreds of millions of diverse pieces of equipment. Every day, some fraction of them are converted to digital. No single study, no single policy can change this infrastructure overnight. I think they are approaching cybersecurity thoroughly and methodically, but it will take time.
Remember Y2K? Roughly the same collection of hundreds of millions of devices were threatened by a common-mode failure (Y2K). It was very analogous to an external cyber attack. The utility industry tackled Y2K, thoroughly reviewed all those devices, and performed flawlessly on the morning of 1/1/2000.
My point? Sure we should worry about cyber attacks on critical infrastructure, but don't jump to the conclusion that no security exists or that nothing competent is being done about it.
Interesting timing for this report to come out right as Obama is asking for draconian emergency powers to be able to shut off the internet and other private networks at will without regard for any law. http://www.tomshardware.com/news/obama-shut-down-internet-legislation,7478.html
Or rather, "Spy sappin' mah power grid!"
The solution is oversight. Congress passes a law noting that major pieces of infrastructure are critical to national security. An oversight body is created to set policies for administration of such intrastructure. Violation of these policies carries criminal penalties.
Then you have the Feds start busting control rooms. Manager in charge gets sent to prison.
Let's see how fast those managers can arrange to have competent people on-duty 24x7 and not need to use pcAnywhere or whatever to get in.
As much as I'm not a fan of a lot of military culture this is one thing they REALLY get right. The mission comes first. Just think about it - they manage to work out every process to something that some 20-year-old with two years experience can supervise with 18-year-olds doing the grunt work. The officers then stay on top of things. The captain of the ship sleeps on the ship and can be woken up at any time should the situation require it. Even the president can be woken up if the chain of command truly requires it.
Manager too lazy to come in to work to see what is going on - no problem, just hire one for each shift.
Not every business needs to be run like a ship. However, the power grid isn't just any business - it requires a much higher level of rigor.
Some have pointed out labor relations issues. These sorts of issues should not impact national security - just look at the Air Traffic Controller strike. By all means the workers should be given proper time to complete their jobs in a secure way - if two computers slow them down then hire a few more people and give them time to do the job right. The solution isn't to cut corners.
I think that software bug was unpatched windows machines in Ohio. But I was too close to it all and may just be making an erroneous jump for correlation to causation. The network storm caused by that virus was pretty horrendous.
As the story unfolded the early reports said the machines were unpatched. Then that story seemed to be brushed for reasons I can only guess with tinfoil hat securely fastened.
I imagine there were many factors that met on that day contributing to the blackout. And I doubt the virus was designed to take down the grid. But the lesson I took from it is that there are many critical machines that are hooked up to the internet or networks that hook up to the internet that aren't properly maintained and these sort of events will be more common. Also that if a non-specific virus can do that much harm I shudder to think what a well designed attack would unleash.
I am a control systems engineer, a member of ISA-99, and a contributor to several other standards on industrial control system cyber security.
The parent post is what SHOULD be done in a recently installed system. I can tell you from experience of dealing with other infrastructure (not the electric grid) that it isn't always that way. There were many systems installed around Y2k that are still in service. And most of you will remember that back then very few people took security seriously. Back then it was all about compatibility. Security wasn't even an issue. The big issue was SHARING the data.
Control systems and SCADA have long working lives ranging from ten to twenty years. The reason for this is because the field I/O validation cost is significant. It dwarfs the cost of the software, the control center, and all that lovely flashy stuff you're so used to seeing. Updating a configuration is very expensive, not just in validation costs, but also training costs, for miscellaneous costs such as review of operating procedures, control system narratives, and so forth. This is why many are forced to keep their systems isolated in the hope that by doing so, things will somehow stay secure.
But these days, that's no easy feat. Nearly every company has a contingent of data surfing desk jockeys with enough authority and enough dream-weaving synergy talk to push for interconnections. That's when things get very ugly.
The problem isn't that they want the data. The problem is that they want the data IN REAL TIME. Most of the time these idiots say the term though they do not understand the implications or even what it means. And that's how the exploits get started.
There are solutions. There are relatively secure methods for moving data in and out of a SCADA system. But they need careful review by people who know both the industrial side of things (to identify what is at risk) and the IT side of things (to know what the potential vectors could be). And the number of people with that kind of expertise is extremely small. We're talking about hundreds or maybe a thousand such people world-wide.
There simply aren't enough people to train the trainers who will train the trainers. And so, we're stuck with the status quo until we can build a community of cross trained people who understand industrial processes, control systems, and IT large enough to handle this situation.
I know many of you probably think you have it bad in the office IT business. And it is. Just know that there is far more truth in the Homer Simpson character than you'd ever dream of...
Nearly fifty percent of all graduates come from the bottom half of the class!
Color codes can help a lot. Blue network is scada, green is public. Scada network has blue ports, blue cables and blue stripes on the devices. Public internet has same deal but in green.
Plugging anything in the wrong color is a firing offense. Specially designated and signed off gateway machines might have a blue port and a green port and special markings that it is OK. Otherwise, any color mis-match or mixing is to be reported immediately.
For extra paranoia, all blue network devices get the high octet set to non-zero (on the card's flash, not just setting it by the OS). The wrong MAC seen on either network is an emergency.
Watch the union guys cheer when said asshole manager is escorted from the building for plugging a green cable into a blue workstation.
Such products exist. The problem is that data often does need to go both ways.
For example, load shed, distribution system models, and demand forecasts often go to servers and clients outside the distribution control center.
These sorts of operations are near-real time processes.
Likewise the outputs include run times, certain transient events, and hourly/daily total meter data often go in the other direction.
As I said before, with careful consideration given to a DMZ between the office network and the control systems, with a sacrificial historian server, and with careful monitoring and alarming, it should be possible to safely set up a portal to the office network.
People have written books on this subject, and I expect to be doing so before long. It is not something I can fit in to a nice pithy message here.
Nearly fifty percent of all graduates come from the bottom half of the class!