Slashdot Mirror


Communications Protocol Leaves Power Grid Vulnerable

mspohr writes "The NY Times has an interesting story about a pair of researchers who 'discovered that they could freeze, or crash, the software that monitors a [power] substation, thereby blinding control center operators from the power grid.' These two engineers wrote software to test for vulnerabilities in the control systems of electrical power grids which use a protocol called DNP3 to communicate with sub-stations. They first tested an open source implementation of the protocol and didn't find any problems. They were worried that their software test wasn't adequate so they started testing proprietary systems. The broke every single one of the 16 proprietary systems they tested initially and found nine more systems vulnerable in later testing. They were able to install malware and also found firewalls ineffective. The pair reported this to the Department of Homeland Security's Industrial Control Systems Cyber Emergency Response Team, I.C.S.-C.E.R.T. and didn't get much of a response. It's scary that our electrical grid is so vulnerable and there doesn't seem to be much urgency to get it fixed. A few patches have been issued, but who knows if the systems have been updated?"

48 of 68 comments (clear)

  1. They had to dislodge other code first by bob_super · · Score: 1

    What are the odds that our best friends already have botnets ready to take our grid down on command?

    Excuse me while I got get a few solar panels.

    1. Re:They had to dislodge other code first by cusco · · Score: 1

      "Our best friends" - you mean like the friendly folks that helped write Stuxnet? Pretty much guaranteed. Having worked in the utility industry for a time I can pretty much guarantee as well that the fixes they mentioned haven't been deployed, as no one wants to take down a substation that controls, for example, a Navy base and an aircraft factory to update software.

      --
      "Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
    2. Re:They had to dislodge other code first by bob_super · · Score: 1

      You mean like my neighbor, whose panels were running his AC full blast when we lost the grid a few weeks back?
      They're not cheap, but AC disconnects are kinda useful. And if your city doesn't allow them, vote them into the 21 century.

    3. Re:They had to dislodge other code first by sjames · · Score: 1

      Much better to have an enemy shut it down when it most suits them.

    4. Re:They had to dislodge other code first by HiThere · · Score: 1

      AC disconnects are only useful if you have a very large number of high powered batteries. Which can easily double the cost of the installation. Yeah, they're quite useful. Useful enough? Maybe not.

      --

      I think we've pushed this "anyone can grow up to be president" thing too far.
    5. Re:They had to dislodge other code first by HiThere · · Score: 1

      IIUC, this wouldn't depend on a botnet. This isn't a DDOS attack, this is a code vulnerability. So a lone malicious hacker could take down the grid. (Yeah, some code vulnerabilities need a botnet to set things up. IIUC this isn't one of them.)

      --

      I think we've pushed this "anyone can grow up to be president" thing too far.
  2. Subject Discussed Years Ago: FIRE THEM! by BoRegardless · · Score: 3, Insightful

    It is not like this is a new issue. Fire all IT managers who were responsible for not doing penetration testing, including the ones at Homeland Security.

    If you do NOT hold managers responsible then they are just lifers waiting for their pension!!

  3. One of my former bosses knew this. by digsbo · · Score: 2, Interesting

    I worked for a fellow who'd previously done some work on power grids. He was aware of these problems in 2005 or earlier. I'm pretty sure these problems were also published in the 9/11 comission's report. But I don't think patching holes in power grid controls provides enough theater to keep people scared, so it hasn't been done.

    1. Re:One of my former bosses knew this. by BoRegardless · · Score: 1

      ...And...as I said, fire all those responsible for ignoring penetration testing and then implementation or it will continue.

    2. Re:One of my former bosses knew this. by Minupla · · Score: 2

      Yep, I saw this talk:

      http://www.internetnews.com/infra/article.php/3831956/Black+Hat+Exposes+Smart+Grid+Security+Risks.htm at blackhat in 2009, so what, at least 4 years?

      Min

      --
      On the whole, I find that I prefer Slashdot posts to twitter ones because I don't get limited to 140 chars before
    3. Re:One of my former bosses knew this. by icebike · · Score: 1

      How do you know it hasn't been done?

      Maybe if your were a little closer to the actual work than "knowing a guy that used to" you would realize that most of these places installed off the shelf VPN routers (about $69 bucks each) years ago, and aren't exposing their SCADA controllers to world plus dog.

      --
      Sig Battery depleted. Reverting to safe mode.
  4. DHS? by reboot246 · · Score: 2, Insightful

    Their first mistake was assuming that the Department of Homeland Security actually cares about homeland security. Department of Homeland Control would be a better, more accurate name.

  5. Scary Lack of Urgency by CanHasDIY · · Score: 1, Interesting

    It's scary that our electrical grid is so vulnerable and there doesn't seem to be much urgency to get it fixed.

    Sure - scary to you, scary to me, scary to the old lady down the road.

    You know who it's not scary to? The NSA, CIA, and all other clandestine TLAs that profit from allowing harm to come to American citizens.

    Remember: the CIA had solid intel about the 9/11/2001 terrorists, but did nothing to stop them; same goes for the Boston Bombers. The more Americans that they can allow to be injured by "terrorists," the fatter their budgets grow.

    Stopping terrorist attacks is the last thing anyone in the federal government wants to have happen. THAT is fucking scary.

    --
    An enigma, wrapped in a riddle, shrouded in bacon and cheese
    1. Re:Scary Lack of Urgency by clarkkent09 · · Score: 2

      So a post saying CIA wanted 9/11 to happen so it's budget would be increased gets modded "Interesting". You mods really need to hang out more on conspiracy nut message boards, you'll find a lot more "interesting" stuff there.

      --
      Negative moral value of force outweighs the positive value of good intentions.
    2. Re:Scary Lack of Urgency by CanHasDIY · · Score: 2

      So a post saying CIA wanted 9/11 to happen so it's budget would be increased gets modded "Interesting". You mods really need to hang out more on conspiracy nut message boards, you'll find a lot more "interesting" stuff there.

      There's nothing nutty about it - it's a proven fact that the government had good, solid intel that a group of mostly Saudi men were planning on hijacking planes and crashing them into buildings. It's also a proven fact that our government did nothing to stop them, and that the budgets and powers of various TLAs see explosive growth (no pun intended) when shit like that is allowed to happen. Contrary to what a lot of people seem to want to believe, the people who run these agencies are not inept, incompetent fools who can't tell their asses from their heads; guys like Patraeus and Clapper got to where they are by being very, very good at what they do.

      What I find nutty is how so many people deny the truth, even when it has been covered, extensively, by multiple media outlets.

      I guess some folks will believe anything... so long as it's a government agent giving the narrative.

      --
      An enigma, wrapped in a riddle, shrouded in bacon and cheese
    3. Re:Scary Lack of Urgency by mlts · · Score: 1

      I hate so state this, but you are actually right.

      Consider a grid down scenario done by some intruder. There would be laws passed by Congress, but I would be genuinely surprised if any of what they passed actually did anything for genuine security.

      Instead, it would likely be laws for expanded surveillance 24/7 on US citizens, mandatory DRM stacks in all hardware accessing the Internet, trying to make it illegal to be anonymous to websites, and things that wouldn't prevent another power loss, but lowering the bar for arrests and seizures, a la SOPA/PIPA. We'd see far more curious teenagers being hauled in front of judges than we would ever see true blackhats trying to attack the power grid.

    4. Re:Scary Lack of Urgency by HiThere · · Score: 1

      Now there's no proof that they wanted it to happen. I'll admit that there is proof that they knew about it, and about some of the participants, ahead of time, but that's separate from what their desires and goals were.

      If you were to ask me what I guessed, then I would agree with you, but I don't know where the decision came from, and I tend to believe that the decision was a bit higher. That, however, is also just a guess.

      P.S.: We also don't know just exactly how much they knew ahead of time, and how specific their information was. It's even possible that they had actual good reasons. I doubt this judging by the actions taken right after the events, but I must acknowledge the possibility.

      --

      I think we've pushed this "anyone can grow up to be president" thing too far.
    5. Re:Scary Lack of Urgency by gtall · · Score: 1

      Care post a link to this proof?

    6. Re:Scary Lack of Urgency by SpaceLifeForm · · Score: 1
      http://www.washingtonpost.com/wp-dyn/content/article/2006/09/30/AR2006093000282.html

      On July 10, 2001, two months before the attacks on the World Trade Center and the Pentagon, then-CIA Director George J. Tenet met with his counterterrorism chief, J. Cofer Black, at CIA headquarters to review the latest on Osama bin Laden and his al-Qaeda terrorist organization. Black laid out the case, consisting of communications intercepts and other top-secret intelligence showing the increasing likelihood that al-Qaeda would soon attack the United States. It was a mass of fragments and dots that nonetheless made a compelling case, so compelling to Tenet that he decided he and Black should go to the White House immediately.

      --
      You are being MICROattacked, from various angles, in a SOFT manner.
    7. Re:Scary Lack of Urgency by gtall · · Score: 1

      Okay, they were going to attack the U.S. How were they to do this? What specifically was the U.S. to protect against? You might have noticed that the U.S. is a large country with a lot of infrastructure. Right now your evidence is more along the lines of the aliens are visiting earth.

    8. Re:Scary Lack of Urgency by mlts · · Score: 1

      Don't you know, those things amplify the mind control waves? Oh wait... they require brains to work. I'm safe.

      The reason for the concern is that we have had shenanigans in government before. Had it not been for multiple whistleblowers, ACTA would be the law of the land in US and Europe, a treaty that would never have seen the light of day until it became ratified. This would have mandated DRM stacks and expanded monitoring.

  6. Re:So... by c-A-d · · Score: 1

    Except that open source... oh, I see what you did there....

    --
    some karma... and kinda lukewarm about it.
  7. Re:Subject Discussed Years Ago: FIRE THEM! by jc42 · · Score: 3, Insightful

    If history is any guide, the managers of these systems are trying to find ways to prosecute the researchers for their actions. It's fairly standard to classify security testing methods as attacks (since that's in effect what they are), and publishing the problems is generally considered telling the "terrorists" how to attack the systems.

    But this is about what should be expected for systems that depend on "security by obscurity". And the managers of such systems rarely reward someone who demonstrates how they've failed.

    --
    Those who do study history are doomed to stand helplessly by while everyone else repeats it.
  8. Re:Private networks? by compro01 · · Score: 1

    No. That would require the investment of money better spent on executive bonuses and shareholder dividends.

    --
    upon the advice of my lawyer, i have no sig at this time
  9. Re:Subject Discussed Years Ago: FIRE THEM! by E-Rock · · Score: 1

    If you want to go after someone, it probably should be the vendor that sold the crappy implementation.

    I'm not a fan of more government, but since the power grid really goes beyond the company owning it, you should have regulations requiring the testing and remediation of any technical/physical security issues. That takes care of your hypothetical lazy IT Manager, the boss who blocks the good manager because it's expensive and not required, and the company who wants to keep selling equipment.

  10. Re:Private networks? by wiredlogic · · Score: 1

    Don't electric utilities maintain private communications networks for their critical infrastructure?

    They do, but nowadays many SCADA systems have internet connectivity for service and support. All it takes is one unsecured internet connection left open by bad system design or a forgetful technician to let the wolves in.

    --
    I am becoming gerund, destroyer of verbs.
  11. Re:So... by ThatAblaze · · Score: 1

    Wait, there was no response to a report about a vulnerability in our energy structure? Gee, I wonder why.. Perhaps they should try submitting the report when the office that will ultimately respond is.. I don't know.. open maybe?

  12. Re:Subject Discussed Years Ago: FIRE THEM! by icebike · · Score: 2

    It is not like this is a new issue. Fire all IT managers who were responsible for not doing penetration testing, including the ones at Homeland Security.

    If you do NOT hold managers responsible then they are just lifers waiting for their pension!!

    Before you loop that noose over the tree branch, perhaps you should check if this report actually reflects the real world.

    TFA simply says the tested software from vendors, not real world installations. This software is in actual use, but that doesn't necessarily mean its running naked on the internet. Most often this is run on private circuits, as most of these installations predate the availability of internet. Even when on the internet, most of these installations use VPN between plants and control centers.

    Even those foolish enough to put SCADA directly on the net have already been notified by their trade associations (if not the DHS) to start using off the shelf VPN routers immediately, and that happened months ago.

    Contrary to the rantings of Slashdot Experts, these places aren't run by total idiots. Nor do they have the luxury of replacing every SCADA controller in their plants. But they do know enough to use common off the shelf technology to provide reasonable level of security, and probably accomplished this a long time ago simply to make management of their network easier.

    Sure, you can scan the net and find some SCADA controllers small water pumps in East Podunk Oklahoma. But they don't control big city plants.

    --
    Sig Battery depleted. Reverting to safe mode.
  13. Re:Private networks? by icebike · · Score: 1

    Most big ones do, because they have been in business for far longer than the internet was up and running.
    Those who don't use VPN routers so that they can have all their plants on the same IP subnet.

    So the story is designed to enrage slash dot nerds, but it never actually says they penetrated systems in the
    wild, simply that the software was vulnerable.

    --
    Sig Battery depleted. Reverting to safe mode.
  14. Re:Private networks? by Cramer · · Score: 1

    They were supposed to, but they've been busy selling off those massive fiber networks for a good long while now. (after first trying to be ISPs, and failing rather comically.)

  15. Protocol != Implimentation by jklovanc · · Score: 1

    Is the problem with the protocol or the implementation of that protocol?

    Mr. Crain ran his security test on his open-source DNP3 program and didn't find anything wrong. Frustrated, he tested a third-party vendor’s program to make sure his software was working. The first program he targeted belonged to Triangle MicroWorks, a Raleigh, North Carolina based company that sells source code to large vendors of S.C.A.D.A. systems. It broke instantly.

    If the vulnerability is not in an open source implementation but is in third party vendor implementation then it looks like an implementation problem not a protocol problem.

    1. Re:Protocol != Implimentation by Darinbob · · Score: 1

      Also note that this is not a protocol that you just trivially tap into. These are not on the internet in general (though never trust your local utility to do the smart thing) and the individual end point devices often don't even have operating systems. Sure, they could put malware onto PCs running in a utilitie's back office, but at that point it is irrelevant what protocol is being used.

      Overall this sounds like a typical Timothy "omg smart grid are evil!" article except that it wasn't from Timothy.

  16. Re:Too important to "fix" by AndroSyn · · Score: 1

    Not to mention the power systems for the entire DC area are too important to allow any outages no matter how short.

    Bahaha...the power goes out in the DC area all the freaking time. Pepco is notorious for power outages in DC. They blame the "dense tree canopy of the city" or something retarded. Ask anyone who's lived there for a while.

    The DC metro area has suffered major outages, the remnants of Hurricane Isabel knock out most of the power and water in Fairfax County, Virginia as well.

    Anything important in the DC metro area and well everywhere else, is going to have both battery and generator backup power, knowing that grid power can and does fail all the time.

    Too important my ass...the power reliability in DC was like living in a third world country.

  17. Re:Subject Discussed Years Ago: FIRE THEM! by HiThere · · Score: 1

    Well, as for private networks...
    Do you remember a few years back when a nuclear plant that was only on a private network was taken over by a virus. (Nothing major happened that time.) This was because in a different building on the network a contractor plugged in his laptop to the private network. I believe that this was by accident. I think he was trying to go on the web. But his laptop had an active infection.

    What with wifi becomming increasingly common, I don't think private networks count as security unless they are QUITE strictly controlled.

    --

    I think we've pushed this "anyone can grow up to be president" thing too far.
  18. Re:So... by BrentNewland · · Score: 1

    I think you're trying (and failing) to make a pun. "Open Sores" = vulnerable proprietary systems. Lame.

  19. Re:The real deal by thebigmacd · · Score: 2

    DNP3 functionality will soon (5-10 years) be embedded in grid-tie solar inverters in Canada so the local power company can control them at will on a per-second basis (I'm working with a local college developing this technology right now). Pretty easy access to the communications channel if you ask me. And no, no one seems interested in security.

  20. Re:Subject Discussed Years Ago: FIRE THEM! by TheRealHocusLocus · · Score: 1

    Sure, you can scan the net and find some SCADA controllers small water pumps in East Podunk Oklahoma. But they don't control big city plants.

    Well here I am in West Podunk Oklahoma and our water pump is controlled by an Emitrol Sytstems relay board connected to an IMSAI 8080 with RS-232 to our PDP-8. I talk to the pump and post to Slashdot with an LA36 DECWriter. It does lower case but it sure looks funny.

    Don't go touching my pump now.
    This is my pump.
    There are many like it, but this one is mine.

    --
    <blink>down the rabbit hole</blink>
  21. Re:Private networks? by Darinbob · · Score: 1

    Or one unsecured PC server (most likely running Linux) that can communicate to the devices and then it is irrelevant what protocol is being used on the end point devices. The same is true for SCADA systems, or any other system where you don't want to flip switches or read dials manually, or drive out to the remote sites to be flipping the switches.

  22. Re:So... by currently_awake · · Score: 1

    Embedded systems (only) get replaced when they break.

  23. Why is this so complicated? by Karmashock · · Score: 1

    Write protect the appliances. It is impossible to remotely modify the code then installing malware should be very difficult. The next trick would be making it impossible to pass executable code to the system's ram.

    Even if you couldn't accomplish the second part... the first part is easy and it would mean recovering from any breach with a reboot.

    There are ways to secure these systems. But ultimately they're going to have to have limited access from remote users. Security updates and modifications to the software should be done locally. That means a hacker needs to gain physical access to the appliance to compromise it. Then you can keep most attacks out with a good lock.

    Some will say this defeats the purpose of these systems. That teh whole thing was supposed to be remotely administered from some central computer command center. How much is that dream worth? Is it really worth all this trouble to not send a technician by every so often to make changes?

    Hard code and write lock the appliances. Then sleep like a baby.

    --
    I've decided to stop wasting my time responding to AC trolls/sockpuppets... so if you want a response from me... login.
  24. Power Grid Vulnerable? by codeusirae · · Score: 1

    "The NY Times has an interesting story about a pair of researchers who 'discovered that they could freeze, or crash, the software that monitors a [power] substation, thereby blinding control center operators from the power grid.' .. It's scary that our electrical grid is so vulnerable and there doesn't seem to be much urgency to get it fixed"

    Then don't connect your electrical grid directly to the Internet !!

  25. Re:So... by K.+S.+Kyosuke · · Score: 1

    You must be Captain Obvious! It's a pleasure to meet you, sir!

    --
    Ezekiel 23:20
  26. Re:So... by K.+S.+Kyosuke · · Score: 1

    And they don't get replaced when it comes to light they've been broken all along? Sounds almost like the "broken windows" thingy.

    --
    Ezekiel 23:20
  27. Internet? by Porchroof · · Score: 1

    Whoever is giving access to vital national resources on the Internet should be arrested and shot.

    --
    Fata viam invenient.
  28. Re:Subject Discussed Years Ago: FIRE THEM! by thegarbz · · Score: 1

    This problem is often brought about by a LACK of IT involvement. In many operational systems the control system is maintained by a small group with more knowledge of the plant and the vendor package than IT infrastructure. You may be targeting the wrong people.

    In any case you're still right. DNP3 is about the most secure of the telemetry protocols, and actually has some basic form of encryption. An attacker shouldn't even be able to get as far as to see or communicate with it.

  29. Problem is at the wrong level. by thegarbz · · Score: 1

    The researches have shown that the system can be compromised from within the network. This should come as no surprise. In many regards DNP3 is far better than any alternative, many of which do not even offer basic authentication let alone encryption. The critical part is the researchers were effectively sitting at the keyboard of their targeted machine. They shouldn't be able to get remotely that far. They should be separated by isolated networks, firewalls, etc.

  30. Re:So... by kilodelta · · Score: 1

    I bet 99.99% of this is the SCADA systems. Those are huge open sores in the power distribution network because of the SCADA vulnerabilities. And the geniuses at the power companies thought it was a-ok to hook SCADA into an ethernet network.

    If you want to be entertained go read the NERC documents.