Slashdot Mirror

A quartz crystal has excellent short-term accuracy, but lousy long-term accuracy. And you don't grasp that his statement makes no sense?

because the power companies handle all of the necessary corrections. No. Power companies make no "corrections". They attempt to keep the grid frequency _stable_ If the grid was below desired frequency in the morning, because of people suck unexpected more power, they do nothing in the evening to compensate for that. Why would they?

Actually, they do correct average frequency. Not necessarily every day, but it is done. At least in 1st world countries.

Why do they do this? Because the grid frequency is supposed to be 50 or 60Hz, and it is run by engineers who take pride in their work.

"And the funny thing is, the old technology does it better because the frequency was controlled by a large physical inertia."

It wasn't really just inertia. The generators also act as synchronous motors. Each ends up loaded more by the grid more when they're getting a bit ahead of the "consensus" frequency and less when they get behind. So once they get synchronized they stay that way. (Barring the occasional screw-up - which usually leads to a regional blackout.)

But if they're heavily loaded they slow down, and if lightly loaded they speed up. They have no inherent absolute speed referenc. So the power companies have to keep them "on time" by comparing them to a good time reference and giving a little extra push (with more steam or whatever) when they're getting behind, less when they're getting ahead - or by lowering the voltage (a brownout) or cutting off parts of the grid (rotating blackouts) when the load is getting too big for them to keep up to speed. If they don't, the generators get slowed down a tad and the clocks slow down. (That's what happened in Europe.)

Manual frequency corrections are becoming less and less common in the US, as described in this white paper. In fact, it is proposed to eliminate them.

A different NERC document tries to explain how balancing authorities work. It is quite complicated but a lot of very smart people have worked on the problem for the past 120 years.

"And the funny thing is, the old technology does it better because the frequency was controlled by a large physical inertia."

It wasn't really just inertia. The generators also act as synchronous motors. Each ends up loaded more by the grid more when they're getting a bit ahead of the "consensus" frequency and less when they get behind. So once they get synchronized they stay that way. (Barring the occasional screw-up - which usually leads to a regional blackout.)

But if they're heavily loaded they slow down, and if lightly loaded they speed up. They have no inherent absolute speed referenc. So the power companies have to keep them "on time" by comparing them to a good time reference and giving a little extra push (with more steam or whatever) when they're getting behind, less when they're getting ahead - or by lowering the voltage (a brownout) or cutting off parts of the grid (rotating blackouts) when the load is getting too big for them to keep up to speed. If they don't, the generators get slowed down a tad and the clocks slow down. (That's what happened in Europe.)

Manual frequency corrections are becoming less and less common in the US, as described in this white paper. In fact, it is proposed to eliminate them.

A different NERC document tries to explain how balancing authorities work. It is quite complicated but a lot of very smart people have worked on the problem for the past 120 years.

Depends on what procedures they adopted. If it was something like the PCI standard they likely could have followed everything, well except the part about not retaining sensitive information, and still gotten hacked. The PCI standard is the bare minimum that should be followed but is something written for MBA types so it has checkboxes that give you a warm fuzzy feeling. It does offer some protection but there are better standards but these are harder and require actual thought. Also if they were reasonably intelligent they would have implemented some well known system benchmarks but those can be inconvenient for people who want the keys to the kingdom. Given what has happened I would guess they implemented the parts of PCI that didn't deal with personal information and called it a day.

Personally, even if they were using PCI, I would love to see them get browbeat because there are better standards, such as the US government's NIST Special Publication 800 and/or 1800 series, the NERC CIP standard, the Cybersecurity Procurement Language for Energy Delivery Systems document. If those weren't enough there are other well respected ones out there as well to choose from. If a business, especially a large one, isn't required to be covered by one I would suggest looking at all of them and make rational choices out of each of them. If a business is required to follow one fully implement that but then still pull from the others to go beyond and then get regulators to scrutinize competitors who are lacking.

I see someone has no idea of what they are talking about in this regard. Here is the current standard that grid operators have to comply with. Also here is what is currently being asked of suppliers by the grid operators when getting a new system. Add in that the systems be benchmarked against these or these is also becoming written into the contracts now. I would assume that operators in the oil and gas industry either have similar things or are at least smart enough to re-purpose the above as the effort to do so would be minimal. A lot of the security efforts for securing the grid are not to protect it from the general internet, they are already separated and if not the company fucked up really bad and if NERC finds out the company will be paying some huge fines so let NERC know. Instead the security is to protect the control system from stupid users who find a USB rubber ducky in the parking lot, connects their corporate laptop to the control network, someone doing malicious things out at some remote substation that then gets into the main control system, or malicious insider. The people going after the grid are professionals and more often than not state actors not little Timmy from down the street who just found out about Low Orbit Ion Cannon or Armitage.

I see someone has no idea of what they are talking about in this regard. Here is the current standard that grid operators have to comply with. Also here is what is currently being asked of suppliers by the grid operators when getting a new system. Add in that the systems be benchmarked against these or these is also becoming written into the contracts now. I would assume that operators in the oil and gas industry either have similar things or are at least smart enough to re-purpose the above as the effort to do so would be minimal. A lot of the security efforts for securing the grid are not to protect it from the general internet, they are already separated and if not the company fucked up really bad and if NERC finds out the company will be paying some huge fines so let NERC know. Instead the security is to protect the control system from stupid users who find a USB rubber ducky in the parking lot, connects their corporate laptop to the control network, someone doing malicious things out at some remote substation that then gets into the main control system, or malicious insider. The people going after the grid are professionals and more often than not state actors not little Timmy from down the street who just found out about Low Orbit Ion Cannon or Armitage.

Probably except for the part about not storing personal information but then they aren't card processors. The PCI standard while it is a standard is really the bare minimum that companies should be held to for them to not be found guilty of criminally negligence for breaches. The actual standard is here and having had to deal MBAs asking about our compliance makes it seems like it is something written for the MBA types to check off a bunch of stuff. There are much better standards and if you aren't an MBA you can figure out how to make them applicable to your business. Personally I like the NERC CIP standard with liberal utilization of the CIS benchmarks as a good starting point for securing a system. If you want others there is always the US government's set of security benchmarks, the DoE document Cybersecurity Procurement Language for Energy Delivery Systems, or a bunch of stuff at the SANS site that you could use as a guide.

In the US if a power company loses computer control of their portion of the grid they still get the joy of rolling trucks out to substations and other locations to maintain control. An interesting thing about the Russian hack of the Ukrainian grid is that the Russians also DoSed the call center to prevent the outages from being reported sooner. Like with any number of cyber attacks there were multiple ways that this should have been stopped but wasn't. One can read all about findings either here or here for good analysis of what happened. Besides if people think a cyber attack against the power grid is the greatest threat they should consider those bastard squirrels instead. If one really wanted to do some damage discharging a high powered rifle (think .30-06 deer rifle) into some of those large transformers at substations would be easier and cause a longer outage than a cyber attack as there just aren't many spares around.

That isn't to say don't worry about cyber attacks and don't mitigate things but there are a lot of other threats that are as damaging or more so that should also be prepared for.

Depends on the industry. The companies that handle the bulk electric grid in the the US and Canada do have rules and regulations covering their security. While not quite as strong companies handling payment card information also have some rules but they don't come backed with the force of law.

See my subject & this link: No denying it /https://it.slashdot.org/comments.pl?sid=9995967&cid=53488785b [slashdot.org] & it's FAR from a complete list (even though it shows 100's of router security + inefficiency issues).

Your argument is so old and tired I get a /. 404 error, seriously I do. That said anyone who is using the factory provided firmware on a consumer router/firewall is dumb. OpenWRT or DDWRT are much better choices that offer better security and better options. Or if you prefer go and drop pfSense on some "powerful" but inexpensive hardware. As you will have a device like these between your computer and the internet I don't see how an argument about cost is an issue as you have your modem connected to the internet (DSL or Cable) and then either a router or firewall that your other gear sits behind. Depending on what hardware you have and layout your setup behind the router or firewall will vary greatly. * LMAO - again, that's you "networking menials" (that can't program their OWN solutions because you're limited) to a tee

Not a millennial (I assume that it what you meant) by a long shot I do actually program and have through my employer contributed to a number of open source projects. You may have heard of a few of them.

WRONG! I don't understand "layered-security"/"defense-in-depth"? I wrote guides on it that even GOT ME PAID https://www.google.com/search?... [google.com]

Guess what I have contributed to guides on securing systems and am paid by my employer to do so when new versions and updates are sought. The difference is that what I have contributed to are respected and well known.

Also it looks like you are a bit to copy/paste happy as I see you are getting frustrated and double posting (see above and below). You really should look into getting treatment for your ails as something does appear to be wrong.

You'd probably be surprised just HOW vulnerable most of the world's critical infrastructure really is.

Concerning power grids, no I wouldn't and people in the US and Canada would actually be surprised how well protected the bulk electrical system is here when compared to what is reported. Even small operators like to follow the security requirements that the large ones have to even if they don't as it does allow them to say that they are following the industry best practices which is a good CYA from lawsuits. Other countries are a different story and vary greatly but even those who hadn't cared much before are coming around after the Dec. 23, 2015 hack of the Ukranian grid caused a lot of European companies to collectively shit themselves.

I'll just leave a few things here for you. In the US and Canada those are either the regulations for cyber security of our power grid or specific requirements being written into contracts for new control systems for our power grid. All of them have to follow NERC CIP with the the other 2 being optional but widely used as a CYA. The Europeans do not have such requirements and it varies from country to country but those that do have regulations they are often very far behind even previous version of NERC CIP. That is not to say that those make you secure but they do offer a good start and following any one of those documents would provide more security than the preferred PCI DSS standard that everyone outside of power grid world thinks is great and the be all end all.

To be fair not all SCADA systems are as unprotected as you would imply but they are not the fortress for security one would hope. In North America there is the NERC CIP standards that need to be followed for grid operators which are a good start and should be approachable for most /. readers. The nice thing is that NERC has teeth and fines can be huge (I believe up to $1,000,000 per violation per day of non compliance) The NERC CIP standards go a whole lot farther than the other major standard that is mentioned often in these discussion which is PCI DSS which seems to be written more for managers who like check boxes. Another consideration is the Cybersecurity Procurement Language for Energy Delivery Systems which is being picked up by a number of organizations as a set of requirements. Then there is always the reasonable and prudent CIS Benchmarks for the OSes and software you are running. I do think that a lot of SWIFT operators think that something like PCI DSS is good enough but it isn't.

Sometimes there are industry regulations that have the backing of laws that demand reporting at specific stages. That is the set of requirements I am most familiar with and violations of NERC CIP can be absolutely devastating to a company reaching to $1,000,000 a day for violations.

Sometimes, its use is even legally mandated.

Shut your pie hole NERC and the DOE would never recommend or mandate such terrorist supporting activities.

Yes that was sarcasm and it is sad that without this someone would misinterpret it.

Sounds like I have been doing shit wrong and could have gotten things done quicker and slacked off. I do start with the lists of best practices and regulations. Then I go and check their layout, settings, firewall rules, configuration, physical security, etc. seeing how they are running things. After that I go and do a proper vulnerability scan and system scan (outside looking in and inside looking out) to see if what they say their system is setup as is what is actually is. If the customer allows it I do some pen testing on links coming in, physical penetration testing with a little bit of social engineering, or pen testing from machine to machine in their environment. Finally after all that I spend a whole pile of time going over the collected results and create a nice report where I organize the threats and risks into actual threat levels and provide mitigation or remediation steps. Typically I spend 2 weeks on site gathering data, and then about another month going over it. I have never been a big fan of checkbox security as it leads to lots of stupid crap but there is something to be said for going through them because I have found a lot of low hanging fruit that was simply overlooked by others.

Defence against 'computer network attacks', that would be like trying to stop their Microsoft Windows computers being hacked. No one in their right minds would put Microsoft Windows anywhere near a war domain. Have they that short a memory:

Software glitches leave Navy Smart Ship dead in the water

Technical Analysis of the August 14, 2003, Blackout:

Slammer worm crashed Ohio nuke plant network

You do realize that there are systems that are not connected to the internet as a whole that exist in secure buildings and while they rely on external data that data is brought in on direct connections that do not go over the public internet. Modern society depends on such systems and some operators of such systems are better at resisting the temptation to just connect everything to the internet directly or indirectly. If following a proper defense in depth strategy these isolated systems still have lots of security on top of them even though they are not connected to the public internet. If you are interested in what the going state of the art in security for these types of systems is you can read the Cybersecurity Procurement Language for Energy Delivery Systems document and go read the NERC CIP v5 standard. These set the minimum level of security that exist on the systems.

Well said. The North American power grid was built out as needed, where needed... in every instance adding just enough spare capacity to accommodate Summer or Winter peaks without alarming long-term investors. Few redundant interconnects. There was no Central Planning Committee deciding how much redundancy may be required, and especially no paranoid engineering on what are essentially un-protectable fragile spans of infrastructure. As with most other modern systems its very existence relies on human restraint.

Which is why only the dreariest of personalities are attracted to the "terrorist alarm industry" where people stay up nights brainstorming all the various things terrorists could do... so terrorists don't have to. They share their findings to an excitable tabloid press and hold conferences, tongues lolling and eyes rolling back as they receive a congratulatory 'pat' on the head for proclaiming the latest "thing" that terrorists could do. In the place of the Cold War excess we now have a behemoth DHS arm of the government who considers the US as its enemy. Every penny spent on it has been wasted.

The real problem --- if in fact there is one --- is that so many are engaged in this paranoid (but fun for them!) pastime of pointing out vulnerability to potential social malfeasance and so few have been engaged in advancing technology in ways that may alleviate all kinds of threat. This means the harnessing and producing of more energy, not less.

Sorry! To all of you in the US who are pushing for micro-grids of wind and solar as a 'plus', it is not. It is a drain, a bad idea, and dangerously stupid. You are being isolationist and foolish, advocating the most expensive and ultimately disastrous options a time when half of all Americans have no savings whatsoever. As if the greatest industrial power the world has ever known should scale back to some quasi-medieval level of energy consumption. As if grid would be made 'better' by introducing countless points of failure (foreign made) devices. Yeah, let's take power generation outside shall we. During the first continent wide hard Winter freeze a hundred million might die from this Darwinian experiment. Meanwhile your ridiculous dreams will bankrupt us all. Every penny spent on it has been wasted. What stark clinical madness! Your own children will not forgive you this frankly 'hippie' level of denial, which has persisted for decades.

The only way out of this mess is to create wealth the old fashioned way by the creation of something that did not exist before. A relatively few massive energy sources that are completely self-contained, defensible, protected from the elements, stock enough fuel for weeks or months or years, and help to decrease the corporate and personal cost of living. Some have heard me say it all before: put a national priority on grid scale DC-AC tech, build overlapping HVDC loops across the country to feed the legacy grid, and above all, feed those HVDC loops with nuclear energy --- yes, fission --- in ways that are proven and new ways we already know can be done.

FRANKLY, everything else, including the mass distribution of fragile natural gas pipeline networks, are shit solutions.

(the following is a repost but relevant to this discussion)

Take a moment to review NERC EOP-005-2: System Restoration from Blackstart Resources. If you live in North America, plans described in this document are your only real line of defense from the chaos and harm that may arise from grid-down disaster. Here is a peek at some software tools used by the industry and Black Start specific enhancements in prog

Take a moment to review NERC EOP-005-2: System Restoration from Blackstart Resources. If you live in North America, plans described in this document are your only real line of defense from the chaos and harm that may arise from grid-down disaster. Here is a peek at some software tools used by the industry and Black Start specific enhancements in progress [2013].

Note that NERC's Compliance and Enforcement process is voluntary. This means no one's going to jail for failure to implement these measures... and there are many in the industry who prefer it that way. We have witnessed the growth of the Department of Homeland Security way past its original mandate. Indeed there is a slow motion power grab in progress.

If you distrust large corporations and the consortiums they form then you're already suspicious. But few can argue that the grid is not resilient or well designed. In most cases frequency and voltage give operators all the feedback they need. But it has not ever been shut off completely, and the electrical equivalent of post-9/11 'ground stop' is neither practical nor possible to test black start capability... NERC does do regular computer simulations of country-wide restarts.

So if you are fortunate to live near one of the ~7,304 operational power plants in the United States (for example) and know some people who work there, you might pose these questions:

Has your plant participated in EOP-005 drills?
Has there ever been a country or region-wide drill where procedures are acted out in real time?
Do you feel the time presently devoted to this scenario is adequate, and plans are in place?
Do you have confidence that the grid could be restarted successfully?
Are there any 'old school' approaches to this problem you feel are not addressed or trained adequately?
To what extent are these black start procedures reliant on computers and functional computer networks?
What kinds of grid-wide inter-plant communications are in place for coordination when the grid is down?
Would any coordination efforts rely on carrier networks (telephone, cell, Internet) being up?

The very first BBC episode of Connections The Trigger Effect explores how we have become reliant on modern technology without needing to understand its intricacies, and uses the Northeast Blackout on November 9, 1965 and peoples' reactions to illustrate this.

If Black Start should fail or become delayed indefinitely, National Geographic: American Blackout is a documentary that dramatically explores effects of an extended grid outage. It is a tame outage -- no Winter freeze or volcanic ash --- with cyberattack as its rather specious scenario. At present the operational controls of power plants are diverse and there is a great deal of manual control, and a coordinated attack could only target the grid monitoring systems and communications between plants.

Take a moment to review NERC EOP-005-2: System Restoration from Blackstart Resources. If you live in North America, plans described in this document are your only real line of defense from the chaos and harm that may arise from grid-down disaster. Here is a peek at some software tools used by the industry and Black Start specific enhancements in progress [2013].

Note that NERC's Compliance and Enforcement process is voluntary. This means no one's going to jail for failure to implement these measures... and there are many in the industry who prefer it that way. We have witnessed the growth of the Department of Homeland Security way past its original mandate. Indeed there is a slow motion power grab in progress.

If you distrust large corporations and the consortiums they form then you're already suspicious. But few can argue that the grid is not resilient or well designed. In most cases frequency and voltage give operators all the feedback they need. But it has not ever been shut off completely, and the electrical equivalent of post-9/11 'ground stop' is neither practical nor possible to test black start capability... NERC does do regular computer simulations of country-wide restarts.

So if you are fortunate to live near one of the ~7,304 operational power plants in the United States (for example) and know some people who work there, you might pose these questions:

Has your plant participated in EOP-005 drills?
Has there ever been a country or region-wide drill where procedures are acted out in real time?
Do you feel the time presently devoted to this scenario is adequate, and plans are in place?
Do you have confidence that the grid could be restarted successfully?
Are there any 'old school' approaches to this problem you feel are not addressed or trained adequately?
To what extent are these black start procedures reliant on computers and functional computer networks?
What kinds of grid-wide inter-plant communications are in place for coordination when the grid is down?
Would any coordination efforts rely on carrier networks (telephone, cell, Internet) being up?

The very first BBC episode of Connections The Trigger Effect explores how we have become reliant on modern technology without needing to understand its intricacies, and uses the Northeast Blackout on November 9, 1965 and peoples' reactions to illustrate this.

If Black Start should fail or become delayed indefinitely, National Geographic: American Blackout is a documentary that dramatically explores effects of an extended grid outage. It is a tame outage -- no Winter freeze or volcanic ash --- with cyberattack as its rather specious scenario. At present the operational controls of power plants are diverse and there is a great deal of manual control, and a coordinated attack could only target the grid monitoring systems and communications between plants.

Well a couple of good places to start for standards in this area would first be the NERC CIP standard and once you have got that down then proceed to the Cybersec Procurement Language for Energy Delivery Systems (warning PDF) for a set of industry best practices that are highly encouraged to be in vendor contracts. While they are written for energy management systems the ideas and regulations should mostly be applicable to all other systems that need computer security as well.

and the person can put a hold on the bill so it can't come to the floor for a vote and they can do it anonymously

Wait what? Can someone explain this to an outsider? Snide comments aside this sounds like the exact opposite of a democracy. I thought only the President had, what it sounds like, something akin to veto powers over bills.

There are two different bills that the GP referenced, the Grid Act and the SHIELD Act.

The GRID act gives special emergency powers to The Federal Energy Regulatory Commission (FERC) to order utilities to do something. This was widely rejected by the industry because some of the powers could force the utility to keep their plants online, even if their machines were being damaged. That's not reasonable. If a grid problem gets to the point where it is damaging generators and other grid infrastructure, we should shut it down. Intentionally damaging a bunch of generators isn't going to keep the grid online if things get to that point.

The SHIELD act was about electromagnetic interference. FERC asked NERC last month to look into this some more. I would rather a government agency with some knowledge and experience on the matter write the rules, rather than a bunch of politicians who are pushing a bill that a lobbyist wrote.

> Further since an EMP is extremely unlikely to happen

What?!

https://en.wikipedia.org/wiki/...

A powerful EMP affecting the entire power grid is inevitable. There has been a lot of discussion about this.

Which is probably one of the reasons that the Federal Energy Regulatory Commission (The federal agency "FERC") asked the North American Electric Reliability Corporation (FERC's rulemaking organization "NERC") to investigate this. Less than a month ago. And their documents look like a common-sense plan.

The quote from Executive Director of the EMP Task Force Dr Peter Pry
"Well, the short answer to [why we aren't defending against EMPs] is called the North American Electric Reliability Corporation. They used to be a trade association or a lobby for the 3,000 electric utilities that exist in this country. ... There is no part of the U.S. government that has the legal powers to order them to protect the grid."

is just ridiculous in that context. FERC is the government agency responsible and they have asked their rulemaking body to make or revise some rules on the subject. And NERC is not a lobbying group. They make rules. The reporting requirements for some of their rules are onerous for the utilities (although they are generally common-sense and reasonable). I have a hard time believing they are in anybody's pocket.

The causes of an EMP are nuclear blast or solar flare, I think in case of the former you would have far larger problems than the grid to worry about.

That's why I find this quite sinister. It looks like they are just blatantly misleading the public to get more funding.

Yep, this guy is full of crap. The telling statement is:
"Well, the short answer to [why we aren't defending against EMPs] is called the North American Electric Reliability Corporation. They used to be a trade association or a lobby for the 3,000 electric utilities that exist in this country. ... There is no part of the U.S. government that has the legal powers to order them to protect the grid."

That's very misleading. The Federal Energy Regulatory Commission (FERC) is a government agency that has powers to make rules regarding the grid. They decided that this is a highly technical industry, so they basically created the North American Electric Reliability Corporation (NERC) to investigate potential issues, draft rules, and get the industry on board with them. FERC tells NERC which kind of rule they want, NERC drafts it. Then FERC decides if the draft rules should be made law. NERC doesn't have any legal powers to protect the grid, they are just a rulemaking organization. So the statement "no part of the U.S. government that has legal powers to order them to protect the grid" is very misleading since NERC doesn't have the power to protect the grid anyway. That's FERC's job.

If NERC is in the industry's pocket, they aren't in it very deep. They have made rules regarding cybersecurity and IT systems that have cost utilities hundreds of thousands (small utilities) to millions (large utilities). Just look at some of their recent filings (proposed rules.) Especially this one - The North American Electric Reliability Corporation’s Report on the Potential Impacts of the Environmental Protection Agency’s Proposed Clean Power Plan—Chapter 7 Reliability Assurance Mechanism. If NERC was in the industry's pocket, this would be some drivel about how the EPA's clean power plan was rubbish. It isn't. It basically just says "hey this EPA plan might affect grid reliability, we better develop a metric to measure grid reliability". It's very reasonable and obviously written by an engineer, not a lobbyist.

The causes of an EMP are nuclear blast or solar flare, I think in case of the former you would have far larger problems than the grid to worry about.

That's why I find this quite sinister. It looks like they are just blatantly misleading the public to get more funding.

Yep, this guy is full of crap. The telling statement is:
"Well, the short answer to [why we aren't defending against EMPs] is called the North American Electric Reliability Corporation. They used to be a trade association or a lobby for the 3,000 electric utilities that exist in this country. ... There is no part of the U.S. government that has the legal powers to order them to protect the grid."

That's very misleading. The Federal Energy Regulatory Commission (FERC) is a government agency that has powers to make rules regarding the grid. They decided that this is a highly technical industry, so they basically created the North American Electric Reliability Corporation (NERC) to investigate potential issues, draft rules, and get the industry on board with them. FERC tells NERC which kind of rule they want, NERC drafts it. Then FERC decides if the draft rules should be made law. NERC doesn't have any legal powers to protect the grid, they are just a rulemaking organization. So the statement "no part of the U.S. government that has legal powers to order them to protect the grid" is very misleading since NERC doesn't have the power to protect the grid anyway. That's FERC's job.

If NERC is in the industry's pocket, they aren't in it very deep. They have made rules regarding cybersecurity and IT systems that have cost utilities hundreds of thousands (small utilities) to millions (large utilities). Just look at some of their recent filings (proposed rules.) Especially this one - The North American Electric Reliability Corporation’s Report on the Potential Impacts of the Environmental Protection Agency’s Proposed Clean Power Plan—Chapter 7 Reliability Assurance Mechanism. If NERC was in the industry's pocket, this would be some drivel about how the EPA's clean power plan was rubbish. It isn't. It basically just says "hey this EPA plan might affect grid reliability, we better develop a metric to measure grid reliability". It's very reasonable and obviously written by an engineer, not a lobbyist.

The whole thing is basically one big "please interpret me however you see fit" paper

So like any number of regulations. The one I am most familiar with is NERC CIP which from what I can tell from practice means whatever the regulator thinks it means. I design for the most strict interpretation as that just prevents any future problems but that gets expensive and customers don't always want to pay.

Because NASA isn't in charge of the energy sector? They monitor and advise. DOE via FERC is in charge of the electrical sector. The ES-ISAC, run by the FERC-appointed ERO, NERC, and the regional Reliability Coordinators (PeakRC in the western US, formerly the WECC RC).

More to the point, there are NERC standards being developed which deal with geomagnetic disturbances. A TPL and EOP standard: http://www.nerc.com/pa/Stand/P...

The bigger issue is cost. We can prepare for anything, but at what cost? Are you ready for your electricity rates to double to cover a 12% chance in the next 10 years? It's a tough balanacing act.

Why would rates double as a result of putting into place a plan (and probably a few layers of communications systems on top of already existing infrastructure) to mitigate the problem before it starts? Oh right, because we would have to pay for a team at NASA, a team at FERC, a team at each of the regional ISO, etc. to all do the same thing? Ugh. Put NASA in charge, they got us to the moon damnit. If rocket scientists cant fix it, no one can.

Basically, the problem can be almost entirely blamed on FirstEnergy of Ohio. They had, in a matter of hours: - A software bug in the monitoring tool. - No backup monitoring, so when the first one wasn't started properly there was no way of knowing there was a problem. - A plant shutdown due to poor maintenance. - Multiple power lines failures due to not cutting back trees as they were supposed to. - Alarm systems breaking, that were simply ignored. - Utterly failing to notify nearby states that there was a problem so they could prevent it from spreading.

You'll notice that almost all of these problems would not have happened had they not cut corners wherever they thought they could get away with it. And if the US electric grid is in trouble, I'd have every reason to expect that it was other electric companies doing the same sort of thing.

Can we get Morgan Freeman on the case?

I can tell you that the industry has really taken this event to heart and learned from it. The linked articles are based on some awfully shoddy conclusions- the scientific article is about interconnected networks in a theoretical sense, and not one of the references has anything to do with the electrical grid. The other link is from "somebody" making conclusions about the power grid based on the scientific article. The grid today is not the same grid we had in 2003. For the last 10 years, NERC has been throwing down standards and requirements for electrical production and distribution based on the lessons learned in 2003. NERC's website may make them seem like "recommendations", but for many parts of the country, an power station or transmission company must follow their standards if they wish to do business.

A failure of the type experienced in 2003 is unlikely to happen. Even if a company such as FirstEnergy makes colossal screwups, rules are in place which make the other parts of the grid more robust to that kind of problem. The chance of a large-scale blackout is reduced in the last 10 years (as opposed to the articles arguments that it is the same, or greater than ever before).

Think about it. Unless you live on the end of a low-population road, your electricity is probably more reliable than any other service you have. The average electric customer in the US loses service for about 8 hours a year. That is 99.9% reliability. The average Japanese electric customer has 5 minutes of outage per year. That 99.999% reliability sounds great, but those extra 9's cost them dearly. The average TEPCO customer pays about 26-32 cents per KWH. My cost in Connecticut is about 8 cents per KWH. I don't want to pay 3-4 times as much for electricity just to have five 9 reliability. Do you?

Which is why you need to heed warnings about deadlines well in advance - these SCADA issues wouldn't have been a problem if planning had started two years ago rather than now.

We did. But it takes time. It isn't our fault that Microsoft waited from 2001 (XP released) until 2009 (Windows 7 released) to make a new operating system worthy of businesses using it. Would you install Windows Vista on anything? Companies saw what a turd Vista was and avoided it for Windows 7.

Windows 7 was released in October 2009. That is only 2-1/2 years ago. My company waited 6-8 months to see if Windows 7 sucked or not. They also investigated switching to Linux, but with the massive code investment and serious code tie-ins with the inner workings of Windows, they decided against it.

So that leaves us with around 2 years of time until now. We had to account for a lot of changes within Windows 7 and work around them. There are new NERC* requirements that we had to understand, figure out best practices, and comply with. We had to get hardware which is industrial-grade computers (IPC) with the right drivers, which is outside of our control. We had to test the software extensively over months- it is controlling multimillion dollar machines which generate millions of dollars a day in profit so we can not make a mistake that causes downtime.

We are now rolling out our control systems with Windows 7 and have been for a couple of months. Did we take too much time to do so? Maybe. But we are a conservative company and don't like to rush out buggy software, and some of the delays were outside our control.

*These NERC requirements say that certain power stations must not run outdated and unsupported software. In the future, all power stations may need to comply with this also. So it is a good time to be selling control equipment.

I will preface this by saying I have no idea of the comparative cash flows in different countries, or between different parts of the utility/electric industry. That said...

In the U.S., if you are part of the power grid (critical infrastructure, also known as the Bulk Electric System, or BES) and are found in violation, NERC has the power to fine you one million dollars per violation, per day. This fine starts at the outset of the violation (not when it was actually discovered) and can continue until it is rectified. Example trade magazine discussion, second paragraph under NERC Basics.

One of the big problems with "smart grid" as a term is it's so nebulous. There are plenty of people who are really, really clever and very, very experienced who will argue passionately that we already have a smart grid and we should be more properly talking about a smarter grid. But there are as many definitions of smart grid as there are consultants looking to make a buck.

Look at this definition:

smart grid - The integration and application of real-time monitoring, advanced sensing, communications, analytics, and control, enabling the dynamic flow of both energy and information to accommodate existing and new forms of supply, delivery, and use in a secure, reliable, and efficient electric power system, from generation source to end-user.

Frankly, I don't like this definition because it's way too verbose. If you want to get to the essence, it's two way communication and control.

But here's the thing to keep in mind. Lots of people have cars. Not a lot of people have a pumped storage facility, or even the geography to set one up.

Energy demand is indeed highly predictable, but there's always an element of the unknown. The issue we're finding these days is energy generation can be highly predictable but public sentiment wants it to be clean and green without realizing that you sacrifice the reliability and/or the price efficiency of coal. Everything in electricity is about arbitrage, at least at the wholesale level.

Forget about saving money. If you already have a car, which in modern day U.S. many people see as a necessity (please, no-one respond with your anecdotal "not me, I take the bus/ride my bicycle everywhere" - that's not my point), and are now told you can effectively use it as a giant whole-house UPS, that's going to be worth something to a lot of people.

And the *best* part of trolling slashdot is watching *nobody* get the right answer.

I didn't have it either, of course, but that's not important right now. Courtesy of a gent on NANOG with better google-fu than me:

http://www.nerc.com/page.php?cid=6|386

and

http://www.nerc.com/files/NERC_TEC_Field_Trial_Webinar_061411.pdf

And the *best* part of trolling slashdot is watching *nobody* get the right answer.

I didn't have it either, of course, but that's not important right now. Courtesy of a gent on NANOG with better google-fu than me:

http://www.nerc.com/page.php?cid=6|386

and

http://www.nerc.com/files/NERC_TEC_Field_Trial_Webinar_061411.pdf

NERC, in charge of all the power regulations in the US under FERC, requires A/V as well in CIP-007 R4. In fact, it is required for anything that is "cyber" (which means anything with an IP address). Got a networked printer, switch, router, firewall which cannot have A/V? Get ready to file a bunch of paperwork (known as a TFE), yearly, and prove that the vendor says you cannot get A/V for it. Better to install a dumb unmanaged switch or non-networked printer (share it via a workstation) so you can avoid paperwork.

McAfee VirusScan Enterprise for Linux works on RHEL5.5. However, McAfee recommends not running it on RHEL5.6 (although our testing has found no problems and we're not using NFS in our NERC areas, but we'd be officially unsupported by McAfee). It will not work on RHEL6 or any of the newer Fedora 13+ releases.

Don't go off the beaten trail and expect support either. Oracle Enterprise Linux is based significantly on RHEL, but yet McAfee won't support OEL.

I'm not sure what all the requirements are some folks may have, but I use ClamAV just so I can say I have some A/V on my desktop and laptop and so I can scan USB devices that others may ask me to check.

NERC, in charge of all the power regulations in the US under FERC, requires A/V as well in CIP-007 R4. In fact, it is required for anything that is "cyber" (which means anything with an IP address). Got a networked printer, switch, router, firewall which cannot have A/V? Get ready to file a bunch of paperwork (known as a TFE), yearly, and prove that the vendor says you cannot get A/V for it. Better to install a dumb unmanaged switch or non-networked printer (share it via a workstation) so you can avoid paperwork.

McAfee VirusScan Enterprise for Linux works on RHEL5.5. However, McAfee recommends not running it on RHEL5.6 (although our testing has found no problems and we're not using NFS in our NERC areas, but we'd be officially unsupported by McAfee). It will not work on RHEL6 or any of the newer Fedora 13+ releases.

Don't go off the beaten trail and expect support either. Oracle Enterprise Linux is based significantly on RHEL, but yet McAfee won't support OEL.

I'm not sure what all the requirements are some folks may have, but I use ClamAV just so I can say I have some A/V on my desktop and laptop and so I can scan USB devices that others may ask me to check.

Enter FERC/NERC which then mandates and fines utilities for not doing the right thing. The bad thing is this puts a burden on utilities already doing the right thing as they now have to deal with NERC audits. In the end, it's they way to go, but it would have been better of the power industry policed itself (but responsibility without authority is pointless, which is where FERC authority to fine millions per day gives NERC the ability to carry out this responsibility).

First of all, you'd better find out of you have to follow NERC/CIP regulations. If you are in the US, you most likely do.

Even if you are not, they are a good set of standards to follow: http://www.nerc.com/page.php?cid=2|20

Second, I think the keys are following procedure, 100% of the time, no exceptions.

One procedure we have added is that all DVDs and USB ports are disabled. We use tamper-evident labels. All network ports not connected to operational computers are shut down. There is literally no way to get software into our network without passing through two sets of firewalls. The first set of firewall scans and only allow access to a secured host where we drop off files. That host is a hardened Linux machine running secure ftp in a chroot setup that also has real-time AV (to additionally check). From there we download the software into our test network and write up the exact procedures to follow to install. We then run our tests to see exactly what changed and make sure it doesn't add any new ports or services (and/or verify this is the expected behavior and document). Then we revert our test machines back to matching production state and follow the step-by-step procedure and no others steps/changes allowed. Then we test again that the same results occur. Then we let it sit and bake and observe that there is no extra connections going out from the box/network or extra processes that expected running (a ton of "good" software insists on phoning home and/or checking for CA cert revokes). Finally, if it passes all of this, then another team follows are exact step-by-step process to install this on our production SCADA primary system (after first switching to our backup SCADA system), then we do our verifications to make sure all is functioning, then we switch to running to our primary SCADA system for the bulk of the day, and if all is well then we follow the same procedure on our backup SCADA system.

We never ran into problems with Stuxnet, but just as the Arizon utility that did run into it, they caught it due to odd behaviors that their SIEMS equipment detected. We'd catch the same thing. The only way we'd miss something is if it somehow ran so quick that our SIEMS equipment never had anything to detect and it "slept" for a long time, longer that we let things bake in a test environment (typically a week or more). But then, if something was really that dormant, I don't know how it would call itself up. Plus, we have Tripwire watching all crucial system files and registry settings, and A/V protecting Tripwire. I'm don't think there is a way to foil all of this.

On the regulatory side, for networks the NERC Reliability Standards for the Bulk Electric Systems of North America address similar concerns (including cyber security) in electrical grids. For highly integrated systems MILS kernels are an engineering solution e.g. to keep actuators and monitoring subsystems apart.

Remember the blackout of 2003?

Yes. I believe I lost a half-day of work in that. Maybe a whole day. It was more than 5 years ago, and for most (not all) of the people affected it was an inconvenience, not a crisis. It was not a national disaster on the scale of Hurricane Katrina or the 9/11 attacks, which is what the fear mongers are trying to compare solar flares with.

What is the biggest power failure that happened since then?

These are not imaginary threats. They are very real.

There is the solar storm of 1859 which caused fires that burned down multiple telegraph offices.

Remember the blackout of 2003? The link is to a report straight from NERC, the power grid regulatory commission responsible for the area involved in the blackout.

Then let's not forget about stuxnet worm.

It is painfully obvious that these are not just crazy fears. As someone who has intimate knowledge of IT systems within a major U.S. power company conglomerate, and is very close to someone who designs/tests/commissions power plant generator hardware, I can assure you that these threats are very real.

I hope the NERC is nothing like the FAA.

I drilled down through their links to a page on compliance. One line struck me:

Whenever a possible violation is discovered, a thorough review is conducted...

That sounds pretty much like the FAA's practice of auditing manufacturers and airlines only following an accident. At other times, these parties are self certifying and the FAA pretty much stays the hell off their property.

Yes, this certainly sounds like a classis under-frequency event caused by the unexpected loss of a generator station. When you have unexpected loss of a station or a transmission line, the frequency on the grid can drop precipitously. Key to frequency is that every (conventional) power plant provides some level of frequency response built-in, while operating under Automatic Governor Control (AGC).

Other power plants under AGC are supposed to put the breaks on an uncontrolled voltage drop. Sounds like the system wasn't able to recover in this case -- either because there weren't enough other plants online to absorb the frequency deviation, or the backup plants that should have been ready to respond weren't quite so ready after all. Here's a link to the National Electric Reliability Counsel (NERC) whitepaper describing such an event and how to prevent it: http://www.nerc.com/docs/standards/sar/Frequency_Response_White_Paper.pdf (NERC is the United States quasi-governmental agency responsible for electric reliability rules.)

They were cutting costs, and since there was no oversight from NERC/FERC, they got away with it, just as they did in the years before they were deregulated. Since 2003, NERC has developed an extensive system of regulatory controls and FERC has been given the ability to levy fines to keep compliance.

So I got "lack of oversight" mixed in with "deregulation". You'll pardon, I hope, my mingling of these two related notions under the same umbrella.

As for being wrong, someone already pointed out to you that hydro-quebec wasn't cut apart for their so called failures, but because of the technical nature of their transmission lines, and you'll also note that their failures were due to extraordinary catastrophic events (geomagnetic and ice storms) and not human error, and that they have since taken steps to remedy the weaknesses that allowed these acts of god to mess with our flow of electrons. Which brings me back to my point: We need government oversight (regulation, if you will) to ensure a safe supply, so that only mighty forces of nature, such as the sun's unpredictable eruptions can cause failures, and not mere hackers.

Close, but you got all of the reasons wrong.

FirstEnergy still had a requirement to remove vegetation under its wires (while "dangerously deregulated") under state deregulation just as it did as a vertically integrated company. The fact that their maintenence crews failed to do so was FirstEnergy's flaw, not deregulation. They were cutting costs, and since there was no oversight from NERC/FERC, they got away with it, just as they did in the years before they were deregulated. Since 2003, NERC has developed an extensive system of regulatory controls and FERC has been given the ability to levy fines to keep compliance.

And besides, the root cause of the blackout was a deadlock in the mainframe at FirstEnergy, where their staff failed to properly recognize that the system was reporting old data as if it were fresh. FirstEnergy had over an hour and a half to take action to correct for the loss of the transmission lines, but instead failed to observe the overloads which eventually resulting in the separation of the load around Lake Erie and the eventual blackout along the PA/NJ border between GPU, PS, and NYISO. The government's report was very watered down on this area.

This might help you understand the root causes, instead of blaming some phantom "deregulation" as the root of all evil.

Oh, and Quebec was isolated from the rest of the Eastern Interconnection (connected only via HVDC ties) in 1990 because of its demonstrated repeated inability to stop cascading blackouts, long long before deregulation hit the scene. Quebec physically could not be affected by the 2003 blackout on the HVAC system.

Give the electric companies 2 choices: Fix your own damn shit with your profits or we fix it and lease it back to you or nationalize you.

Sure there are people that are going to bitch because they're used to their handout. But handouts aren't going to help anyone. Make everyone work.

It's not perfect but it's a hell of a lot better than handing it over to a bunch of people who managed to already lose $700b.

[0].M-F you live in work housing or you work 4 - 10s or 7 on 7 off.

I hate to ruin your rant with what we call "facts", but the grid in the United States is not owned by private companies that you can just boss around from your ivory tower of uninformed tripe. It is an amalgamation of state-run and multi-state entities called ISOs (Independent System Operators) that both contract and coordinate with the transmission agencies in concert with privately-owned and state-owned generation assets to produce consistent and reliable power. A grid, in the strictest sense of the word, is a series of transmission lines, owned by multiple companies, that are interlinked and under the complete autonomy of the ISO. Nothing happens without the permission and direction of the ISO or FERC (and NERC as its enforcement arm). The grid is aging, but since the ultimate authority to direct replacement lies with both federal, state, and multi-state agencies, who precisely in your little world bears the fiscal burden?

May I suggest for your education:
http://www.ferc.gov/
http://www.nerc.com/

And for ISOs:
http://www.ercot.com/
http://www.caiso.com/
http://www.nyiso.com/public/index.jsp
http://www.pjm.com/index.jsp
http://www.midwestiso.org/home

Find the one that serves your area, and berate them with your uninformed bile since you obviously understand all of this better than anyone else.

Or do you?

To be fair, there have been steps taken since the 2003 blackout to make the power system more reliable. With the Energy Policy Act of 2005, membership in the North American Reliability Council (NERC) has gone from completely voluntary to federally mandated. Failure to adhere to industry standards can carry a fine of up to $1 million per day. The CIP- standards all deal with cyber-security, and the EOP- standards specify what happens in an emergency situation - for example, a big node goes down, and initiates a cascading failure. Automated systems are required to be in place that will cut the power in such situations, leaving some people in the dark, but protecting the grid as a whole. Is the system perfect? No, probably not. A good social engineer could probably still weasel his way into a system. But steps have been taken to minimize damage in such a situation.

And there are fast gradients. Some of them are small, like an entire office building starting or shutting down their lights. Some, however, are not so small - like - let's say - an entire neighboorhood starting their electric boilers at the same time). When this happens, a brownout ensures - the electric plant is overwhelmed, and its output voltage drops.

Also, while I'm here, I'd like to point out that the NERC CIP standards, which are being mandated for power companies in the very near future, are intended to address many of these issues that have been brought up. The industry hasn't been sleeping, and with FERC taking over the standard and having audit and penalty authority, all NERC members are beginning to take security seriously. Chances are, if you're reading this in the continental USA or Canada, your power is being supplied by a NERC member.

NERC CIP standards: http://www.nerc.com/~filez/cip.html

spending $50M for underground and for sake of argument 0 for maintenance of the same period.

First you are working under the assumption that if you put anything in the ground, it is preserved perfectly forever. You will *never* put something in the ground and do 0 maintenance on it for even 5 years. Ask anyone who works in a NAP, or does fiber work how many times a month they have to do maintenance because of anything. Backhoes, trees, and vermin as stated in a previous post, have an odd way of breaking things that is left under dirt.

There is no market for power distribution. If you are dissatisfied by the reliability of your electrical grid, you cannot switch to a competitor's grid. The owners of the grid will charge you the cost of running the grid plus as much as they can get away with over that.

Just for research check out and dig through the following:
http://www.ferc.gov/ Federal Electricity Regulatory Commission
http://www.nerc.com/ North American Electric Reliability Council
http://www.pjm.com/ PJM Regional Transmission Organization
http://www.midwestiso.org/ Midwest Independant Transmission System Operator, Inc.

The most amazing thing about the electrical grid is that it works at all. And indeed most of the time it works well when compared to, say, Iraq. But although it works in routine cases, it does not work in even moderately exceptional cases, such as peak demand for air conditioning. And it certainly does not work to address problems like the California power crisis of several years ago.

It is only amazing that something that is watched over by the government is able to function as efficiently as the Electric Grid, other than that standards and policies and procedures out the rear are in place to ensure that the lights stay on. Also, as someone else already said to you, the situation in California was staged specifically to line the pockets of officers of a Public Utility.

Looking forward two to three decades, the electrical grid is probably the single most important piece of infrastucture to improve

The grid is being upgraded on a regular basis. It is known that the US is power hungry (in the electic sense here) and Utilities are working almost daily to get funding, and zoning, and laws to build more distribution sites. Bureaucracy is a slow, tedious process.

But the more I look at it, the more it all boils down to one poorly-trimmed tree.