Deregulation and Niagara Mohawk - Is There a Story?

It's just a few hours after the Northeast U.S. power outage, and facts are trickling in; as of right now, it looks like an accidental overload knocked out a large part of the Niagara Mohawk power grid. A few years ago, California went through rolling blackouts that were largely due to a poorly-executed deregulation of that state's power industry. The question that's probably occurring to many of us is, did late-'90s deregulation play a role in today's power event? I don't know the answer, so I'm turning it over to you -- moderators, please check links and up-mod the most informative, pro or con. Here is some information to get you started: "We support deregulation 100 percent..." (N-M spokesman, 1997; notes N-M wanted to sell generators and "concentrate on the transmission and distribution of energy" -- did it?); N-M made some bad investments and is scheduled to request a rate hike (did it?); and N-M's own website says: "Deregulation [has] changed the laws and regulations governing the electricity industry to promote competition..." (how so?).

Comments

Nothing to do with deregulation

[mjmalone]

I don't see how this has to do with deregulation. It has more to do with poor design of the power infrastructure. From what I have heard, the way the power grid works is there are switching stations which link various networks together much like a router on a lan. When one switching station goes down, for whatever reason, there are fail safe systems which move affected areas over to other switches.

What can happen is, if all stations are working at or near capacity and a part of the network goes down for whatever reason (fire, or too much power being drawn for example) then when power is routed from the other switching stations they become overburdened as well and there is a ripple effect of outages across the grid.

When this occurs, power companies have to be careful when bringing power back online as they may become overburdened again as soon as they become operational. The U.S. power grid has become extremely complicated and vulnerable as it has scaled. Fail safe systems often fail in their fail safe components.

Regarding the rolling blackouts in California, they had more to do with Enron witholding power than with deregulation. I have not researched deregulation sufficiently so I can't really argue for or against it, but blaming everything on it is not helpful.

Re:Nothing to do with deregulation

[TedCheshireAcad]

Oh God no, no power to run the computers.

You mean I actually have to interact with people?????

Re:Nothing to do with deregulation

[Wesley Everest]

I don't know if the overload was due to deregulation, but one of the purposes of regulation is to ensure that the power company can satisfy demand, even relatively unlikely peak demand. It's possible that deregulation led to them running leaner with less margin of error for a big spike in demand.

Add to that an unexpected increase in air-conditioner usage and there you go -- overload and outages. That's one possibility. I suppose we'll find out the facts soon enough, though.

Re:Nothing to do with deregulation

[straybullets]

It has more to do with poor design of the power infrastructure

well, bad design is linked to deregulation, since good design takes time, and deregulation wants money fast. It goes the same for taking good care of the existing installation : it costs money, and deregulation is about profit more than service.

the true fact is that deregulation is a joke, it was well seen with the english rail system. And the joke is on us !

Re:Nothing to do with deregulation

[PFAK]

I'm going to have to disagree with you.

Deregulation usually ends up having the power company as a corporation, and not a Crow Corporation or such. When a power company (or any other company for that matter), has to become profitable they will cut costs, and when they are cutting costs the level of service usually falls.

The most likely reason that this has happened is that the power companies did not want to spend as much money on the grids to maintain them, and make sure that they were in complete working order, and add more grids and upgrade their equipment -- all because of deregulation, and saving money on in the "short term".

Re:Nothing to do with deregulation

[ascii3f]

Wrong. It has everything to do with deregulation.
If a power company builds a new power line at their expense, they must allow other companies access and sell power using that line.
Companies aren't building power infrastructure beyond the minimum required because they get screwed. You will see more of this happening.

PS Full Disclosure. I am biased as I have worked in the power supply business for several decades.

Re:Nothing to do with deregulation

[Frymaster]

i live in alberta. a few years ago, the provinicial government - which has an ideologicl committment to fiscal ultraconservatism - deregulated the power industry.

the results have generally been regarded as disasterous - most notably a rise in power bills for both domestic and industrial consumers that topped out at well over double. the power rate increase resulted in less disposable consumer income and increased cost of doing business in the province and was regarded as an election-killer by the current administration.

so they spent their way out of it to the the tune of $2.3 billion. that was direct subsidies to rate payers. of course the whole subsidy was a charade since those same rate payers were going to pay for their "subsidies" in income tax increases or reduced social spending in other areas. clearly a case of cutting you a cheque with your own money.

so who got rich? the power companies. same service, same power, more money.

bottom line: electricity is a necessity. like water, or the police service. it is a completely inelastic commodity and privatizing it is only encouraging the new power overlord (since there is, really, only one major power provider... a monopoly) to charge the maximum the market will bear and damn the consequences.

source here: here

Re:Nothing to do with deregulation

[straybullets]

The same for the british rail : it worked perfectly. deregulation came in and it went down in flames, late trains, dirty wagons, and dead peoples in accidents : you need to take care of the infrastructure, you need to plan for the future, and keep the whole engine running smooth.

You can't do that if you have to keep an eye on your competitors and keep lowering the prices some more, and fire those expansive workers.

forget what you've been told : open your eyes and think.

Re:Nothing to do with deregulation

[djblair]

A former DTE employee, I am typing this on my laptop with no power here in Detroit, MI. I agree, this is certainly not a result of deregulation. Perhaps I can offer some insight on some of the specifics.

The reason so many plants are now offline is because of a safety system put in place to protect their generating equipment. An overload can severely damage generators. The device which disconnects the plant from the grid is a shoebox-sized relay. The great northeastern blackout of 1965 was actually caused by a defective relay.

However, it is highly unlikely that a relay was the cause of this outage. If not for faulty equipment, what caused it to happen? Since the problem seems to have originated in Niagra Falls, New York, I suspect that a major line which provedes part of the northeastern US with power from generating plants in Canada went down. This event would have triggered the above scenario, causing plants in both the US and Canada to shut down.

It is interesting to note that, as with land-based phone systems, little has changed in the way power is distributed to customers in the last 30 years (certainly advances in fiber optics have advanced phone systems, but the last-mile copper systems have remained unchanged in over 50 years). Hopefully now, systems will be put in place to prevent outages of this magnitude from happening again. A system of automated switches with real-time network links could be used to disconnect parts of the grid instantly before the problem could spread. Maybe we will see some of this technology in the future, now that there is a definate need to persue it.

Re:Nothing to do with deregulation

[Anonymous Coward]

Yeah, CNN has been so hard on the Republicans these days, which is so unfair because so many Republicans have been doing so many things so very well. The way they blow any little scandal all out of context and proportion and just harp on it for weeks on end, their steady refusal to walk the party line, their whole anti-war attitude, their open mockery of the president, it's just ridiculous. And I hate how they have that "Impeach Bush" ticker running across the bottom of the screen all the time. Damn liberal media. That's why I only watch Fox.

Re:Nothing to do with deregulation

[macdaddy357]

The only risk publicly traded corporations worry about is not making projections for this quarter. If short term buck grabbing causes a long term problem, it will probably be someone else's problem. Besides, the boys in the boardroom can always blame the lowest level drones, and assure shareholders they will be dealt with.

Re:Nothing to do with deregulation

[rsborg]

That doesn't make too much sense to me. Assuming the blackout is a result of cost cutting, it seems like an unreasonable risk, because now that the power's out, people CAN'T use the power, and thus the utilities can't bill until they lights come back on.

That would make sense, if you don't compare what they saved (in running lean) vs. what they lose (in one blackout). In short, I bet "unreasonable risk" you pose, would be, financially reasonable if it only happened occasionally, and that's probably what they calculated. The same concept as "acceptable level of casualties"... and although not as amoral, still causes hell for everyone not making that extra buck that the power utility is.

Re:Nothing to do with deregulation

[Chris Burke]

Deregulation would only help this sort of crisis, because it would be in the individual stake-holder's best interest to shield themselves from such an event.

Well, that's one possible outcome, but I wouldn't say only. It is also in the stake-holder's best interest to cut as many corners as possible, reducing costs and maximizing profits so they can cash in and get out before the inevitable disaster hits.

Re:Nothing to do with deregulation

[utmecheng]

No way. The cost of dealing with a wide-spread blackout like this one is gigantic. Think about how many man-hours were billed in today's fiasco? They had to shutdown all generators and then turn them back on. That combined with the damage done to their system and the repairs they will have to make means big losses. Not to mention parent's point about loss of money tonight. When you have rolling long term blackouts due to localized stresses then you can point the deregulation finger, but with major grid-wide blackouts its a huge financial burdon.

Re:Nothing to do with deregulation

[Zeinfeld]

The same for the british rail : it worked perfectly. deregulation came in and it went down in flames, late trains, dirty wagons, and dead peoples in accidents :

British Rail did not work 'perfectly' by any standard - with the possible exception of the privatized service.

The problem is not regulation, deregulation, privatization, nationalization or any of the surface reasons thrown about. The real problem is people who substitute ideology for thinking about a problem.

The free market is not the solution to every problem. Get over it.

The state is not the solution to every problem either. Get over it.

There are occasions when you have to use one strategy and occasions when you have to use another. Understanding that there are potential problems with a proposed change is essential if you are going to avoid them.

Instead what we get is politicians who use ideology as a substitute for thought. The solution to every domestic energy issue must be to drill oil wells in Alaska. The problem to every foreign policy problem must be to invade a country in the gulf with large oil reserves. The problem to every economic problem must be to give stupendous tax cuts where at least 80% but hopefully as much as is possible goes to the richest of the rich, and in particular rich Texas oil-men. One thing is certain, W. is not going to say a word about the NYC power cut until he can work out how it can be used to justify some policy to benefit Texas oil men.

The free market is one thing, if you could establish a free market in energy that would be a great solution. The problem is that it is not possible to do that, the market is illiquid, supply and demand are constrained in certain ways. But to the ideologue these problems simply cannot exist, they don't exist in the theory so they cannot exist. Its like a robot in a bad 1960s Sci-Fi serial. So the ideologue plows ahead with a broken scheme and creates an unmitigated mess.

That is exactly what happened with privatization of BR. There is no reason the UK rail network cannot be private, it was built entirely with private capital. But the Tory privatization plan based on the politics of sticking your head in the sand was never going to improve matters.

Re:Nothing to do with deregulation

[Darby]

I suppose we'll find out the facts soon enough, though.

Ladies and Gentlemen, we have an optomist among us.

Re:Nothing to do with deregulation

[stanwirth]

I agree with you, macdaddy. The power failures in Auckland NZ in 1998 were caused by deregulation. Lasted over a week. The cause was failure to properly maintain underground mains -- lines maintenance had been farmed out to a variety of cut-rate contractors who just didn't do a bloody thing. They had to call back in the old AEPB (Auckland Electrical Power Board) engineers who'd been made redundant in the deregulation craze of the mid-90's. Brought back in as contractors at top dollar, might I add.

The solution was to separate the lines companies from the generators and retailers -- i.e. re-regulation was introduced

This hasn't fixed things, really, because the lines companies now have no incentive to invest in maintenance, and the maintenance the do do (when there's a blackout -- we get 'em all the time where I am) is way piss-poor and the contractors do it in such a way as to maximise their own revenue.

For example, a few weeks ago, the power went out in a big rain/windstorm like it always does. So, Alstom (the same contracting outfit that maintains the public telephones for NZ telecom, which is the latest spectacular privatised public services failure story here) comes out to solve the immediate problem, which is the fact that some wet bamboo had blown over the three main lines servicing the whole valley, shorting us out. So they remove the one piece of bamboo causing the short. I watched them. This bit of bamboo had come off a whole stand of bamboo right next to the main lines, and there were over a DOZEN other bits of bamboo in the stand that had obviously also been blown over the same three lines -- burnt off at the same spot. I asked the Alstom guys if they could request that the stand of bamboo could be reported as the cause of repeated problems -- Answer? oh, no, that has to come from "head office."

Now, there was another contractor (different company) halfway up the road cutting down some foliage on behalf of the lines company -- foliage which had nothing to do with the problem . Funny, that. So I went and asked the tree cutter guy, there with all his gear and ladders, if he could please either cut down the stand of bamboo that was the real source of the problem, or notify "head office" (the lines company) that there was just a tad more foliage trimming that would need to be done to solve the source of the problem . No, he couldn't do that. He'd been sent out to cut down some tree branches, and by god that's what he was going to do.

Now why should either of these contracting companies do anything to solve the cause of the problem? After all, leaving the source of the problem in place assured them of ongoing work -- at top dollar . While this ate into any budget the lines company might have for other maintenance (like maybe uh, moving the 40-year-old overhead lines to a safer and more modern underground system so we don't get the spectacular exploding-transformer effect every time there's a lighting storm maybe? DUH!)

Privatisation is the cause of our blackouts, that's for sure! And it's caused the rates to go up. Worse service, higher cost, poorer maintenance.

And what's the deal? They privatised and deregulated the airline, the electric companies and the phone company -- but in each case, as a sop to the poor NZ citizen, who may have been losing an asset but was certainly getting poorer and more expensive service, they earmarked blocks of stock that could only be issued to New Zealanders.

So that some New Zealanders could have the privilege of purchasing an asset FROM THEMSELVES. They only fell for it because of this farking myth about competition leading to better service and lower cost. Yeh right.

Re:Nothing to do with deregulation

[dspeyer]

Or not get out.

Seriously, I don't think the power companies have been significantly harmed here. They won't lose customers, seeing as the outage was purely geographic. All bad will is directed against the power companies equally.

Let's face it: the only thing power companies compete on under deregulation is price. They have the same product, the same reliability, etc. This means the only viable business model is to cut every corner you can.

But money is the one true god, and questioning deregulation is unamerican, so we don't see a problem here, right?

Re:Nothing to do with deregulation

[stanwirth]

And involving the government would have solved any of this how?

First of all, rather than having two different contractors, both with a profit motive for not solving the problem, you would have had basically lazy government employees that would have had it in their best interest to solve the source of the problem -- so they wouldn't get future callouts on cold rainy days. Who the heck wants to go out on a rainy day for n+1 emergency calls-- when they could schedule preventive maintenance requiring them to go outdoors during the months when it doesn't rain nearly as much.

When the government was directly involved (i.e. owned the power companies, and the telephone companies, and the national airline), there was *one* bureaucracy -- that was still ultimately accountable to the public. And the quality of service was very high.

Now there are half a dozen "state owned enterprises" (SOEs) which means the public still foots the bill, but now (being "privatised") we have absolutely no oversight, and they are accountable to their boards to make a profit, rather than directly to the people to provide a service. In the case of the lines companies and phone companies, the quality of service is directly related to the integrity of the lines themselves -- and without public feedback, there is no incentive to improve service, because the lines companies still have a monopoly. No such thing as competition there. Yet the SOEs were set up on the pretext of "providing competition" where no competition can possibly take place. It's an argument that would only appeal to a neocon ideologue.

Furthermore, because it's ex-government department employees in these SOEs that operate the SOEs but now without public oversight -- well, it's the same fat, lazy, corrupt, theiving do-nothing slobs you get in government departments, but now they can do nothing for more money and nobody but other fat, lazy, corrupt, theiving slobs --other ex-government bureaucrats--are looking over their shoulders. The public has no right to even find out what's going on---because now it is a private company.

They're more organisationally inefficient and cut more corners on service and funnel more plum contracts to their buddies now than they did when it was a government monopoly/bureaucracy-- because at least when it was a government monopoly/bureaucracy, we had some right to oversight. New Zealand has seen how the privatisation of public services gives you the worst of both the private and public sectors.

Unfortunately.

(You've got a red-tape bound beuacuracy. Governments are famous for them.)

And when the NZ government-run electricity, telephone and airlines were privatised into SOEs, the tape only got longer, redder and meaner--and more expensive.

Re:Nothing to do with deregulation

[jmorris42]

> The excuse for the 1965 power outage was effectively "we didn't know."
> Obviously they know now, so "tbey didn't care" is a plausible theory.

A temporary failure of a complex system like the American power grid every few decades doesn't sound like a "I don't care" attitude to me. Sounds like imperfect systems built by imperfect humans. The engineers will study this incident and improve the system. And we will discover yet another failure mode after another couple of decades of rapid demand growth. NIMBY attitudes towards building power plants are most likely the largest contributing factor though, since had the industry been able to build new plants to keep up with demand the system wouldn't have been running so close to capacity and that isn't a problem for engineers.

Of course as a Dean supporter, brains and rational thought isn't likely to be your strong suit. Raw emotion, mostly a blind hatred of Shrub, are his draws.

Re:Nothing to do with deregulation

[spurious cowherd]

Actually they do know.

Federal Power Commission investigators found a single faulty relay at the Sir Adam Beck Station no. 2 in Ontario, Canada, which caused a key transmission line to disconnect ("open").
This small failure triggered a sequence of escalating line overloads that quickly raced down the main trunk lines of the grid, separating major generation sources from load centers and weakening the entire system with each subsequent separation.

As town after town went dark throughout the northeast, power plants in the New York City area automatically shut themselves off to prevent the surging grid from overloading their turbines.

Re:Nothing to do with deregulation

[arivanov]

It is a result of american dislike of mathematics and long term scientific projects.

You cannot replace mathematics with elementary computer control systems based on simple feedback. That is the reality. And the american power blackouts are one of the common illustrations given in mathematical modeling classes in most of Europe of why simple feedback systems fail.

After the big blackout of the sixties every single European country has put this task to their universities and/or specilized institutes. In all cases it was given to mathematicians, not engineers (or ended up with the mathematicians after the engineers failed).

I happen know some the people who did the modelling in three countries. It took anything between 7 and 11 years to come up with viable models as well as analysis of viable failure scenarios. The scenarious have been rolled out by the 80-90-es so we are talking 20+ development and deployment cycle.

As a result you simply cannot take out the grid like this in any European country unless the country has grown complacent and has stopped updating the models to account for change in power usage patterns.

history repeats itself

[REden]

This certainly sounds like the 1996 Great Northest Blackout.

http://blackout.gmu.edu/events/tl1965.html

Robert

Fraud a significant contributing factor

[Anonymous Coward]

"California went through rolling blackouts that were largely due to a poorly-executed deregulation of that state's power industry"

Actually, there was a significant amount of fraud involved. Check it out here: http://www.cbsnews.com/stories/2003/03/26/national /main546097.shtml

DAMN!

[DAldredge]

The people that RUN THE FSCKING GRID do not know what went wrong and /. is posting articles asking if X caused it???

Are you insane?

Re:DAMN!

[EmbeddedJanitor]

people that RUN THE FSCKING GRID do not know what went wrong

I worked in the power control/data aquisition field for a while and can assure you that in a complex grid it is very difficult to pin-point failures.

Consider that there are normally many redundant lines and generation points. If a generation point goes offline, then the load through the lines changes. If a line's capcacity is exceeded it trips. This increses the load through other lines and you can get run-away instability. All this shit goes down in a second or so, so figuring out where things went wrong is not easy. Figuring the *trigger* might be easy (eg. generation point failed), but at what point does the redundancy fail (ie does the system itself fails)?

To keep on top of this, most grids run constant 'what if' analysis of their network. ie. if line x or generator y trips what will happen? if load increases at point x what will happen? The analysis helps to ensure that sufficient redundancy is switched in to cover certain failures to a certain risk level.

Unfortuantely with cost cutting etc, building of new lines and upgrading often gets delayed. Thus, the opportunities for redundancy are decreased and the risk levels are increased.

BTW: It is also a hell of a task to restart a grid.

Re:DAMN!

[HiThere]

Be aware that there can be multiple layers of "causes". At one level the cause may be "the insulation on some wires failed and a short developed across the coils of a transformer".

OK. But why?

OK. But after the last New York blackout, there were many promisses that new switches were going to be emplaced that would automatically shutdown connections to large areas that were failing. This doesn't seem to have happened. Did the switches fail, or were they never installed.

OK. But I heard that there was this computer model of the entire electrical system that was being built, which would be able to examine it's sensors an from a log of failures would be able to instantly report on what the cause had been, and where it had occured. Did the model fail? Is there some reason it isn't being used?

Ok. But...

Some of these could be answered immediately (by those with knowledge), some require on site investigation. Some are obvious without any statement, e.g., clearly the super power grid model isn't being used, or they could immediately pinpoint the problem. But why? (Probably it takes a super-computer to run it, and they can't afford to have it running most of the time...if it actually works, that is.)

New Zealand

[SimonInOz]

In March, 1998, Auckland - New Zealand's major city (though not the capital, that's Wellington, in case you need to know) - had a FIVE week blackout.
This was after the system was privatised. They cut back on maintanance and instead of three main feeds, they had one. It blew up.
Five weeks with no power. In a major(-ish - hey, I live in Sydney) city. Incredible.
If any city NOT privatised has suffered such an indignity I have not heard about it.
So I blame privatisation - the accountants tend to outrank and overrule the engineers (heard that one before? Remember Challenger?)

Re:New Zealand

[automatix]

After doing some research on this for an undergrad paper, it turns out that it was a combination of bad luck and accounting/management.

The major 110kV CBD feeder lines had their lifetimes "reassessed" and it was decided that there were still plenty of years left in them. So they took their time replacing them (it was underway when the crisis started), but it turns out their lifetimes were more like the original specifications (funny that).

One major 110kV line failed while one was down for maintenance, which lead to the failure of another two 110kV lines a few days later due to overloading. It didn't help when some monkey roadworker dug thru one of the smaller 40kV feeders that were helping prop up the cbd either.

Then it got fun - rolling morning/afternoon blackouts, companies moving to offices out in the suburbs, temporary overhead lines erected running 20km to one of the other distribution yards, generators everywhere...

Deregulation hadn't been completed at that stage - the new lines/distribution company in Auckland which came in to being a year or so after the crisis is taking their job very seriously and has done a lot to improve uptime and redundancy.

Rob :)

How the power grid works...

[Zachary Kessin]

In case people are wondering how the power grid works, here is an article on howstuffworks.com on how
The power grid works

Other problems with the power grid?

[karpenl]

Maybe regulation/deregulation is not as much the issue as redundancy in the power grid. I would think that it would make sense for there to be enough reduancy and backup systems in a power grid as large as the one described so that black outs such as this one do not occour.

On the other hand, the need for redundancy, or possibly for areas to draw power from other sources is expensive, and does not fit the model of a profitable buisness. Regulation could help by fueling money into redundancy and requireing a certian ammount of backup systems in place so that major black outs occour. Also, as far as I understand, the power grids is large cities have not grown to keep up with demand in said cities making blackouts or atleast brown outs more plausable.

Then again, this is only news because black outs of the magnitude happen so rarely. In all likelyhood this was a freak accident on the level that will not happen again for another 30 years or so. Hopefuly the people in charge of both the power grid in most areas as well as most major metropolitan areas have backup plans for when events such as this one occour. One can only hope.

The problem is it WASN'T deregulated

[digrieze]

I just saw the first political spin on this mess. Bill Richardson, the Former Energy Secretary, was on CNN saying we have a "third world power grid". What he didn't say and the CNN sycophant wouldn't bring up is that while he was in office the Clinton administration turned down every request to build new or upgrade existing power stations. The theory of the grid is that when one part of the grid needs power it can be shunted from areas with excess capacity. Just as in California (who also refused to build new capacity) THERE IS NO EXCESS CAPACITY! When one part is at capacity, they all are.

Quite frankly, we're a living in a tech world now. We need the power. Until we stop politically cowtowing to "eco-nuts", "consumer advocates". and other neo-luddites this is going to keep happening.

Sorry!!!

[Chester K]

Sorry everyone, this one was my fault... I accidentally plugged my toaster oven in the same outlet as my microwave. :(

Thunderstorm US side of the border 19:35

[Ummite]

Last theory from Canadian governement is a thunderstorm near the border of canada-us, US-side, that cause the blackout.

Deregulation a contributor to that fraud...

[poptones]

And so it goes. Widespread deregulation of public infrastructure is going to be remembered as a phenomenal mistake. People with way too fucking much money and power are driving this phenom, and only a fool believes they're doing it out of some touchy feely public compassion.

They used to point at the airline industry - remember? "Oh, look how great the airline industry did after it was deregulated!" Yeah, well, so now the taxpayers get to bail them out to the tune of tens of billions of dollars. Might as well have subsidized them from the beginning...

They've behave better because they'd have to.

[raehl]

California put some rather peculiar rules in place regarding how much should be paid to move power at certain times. What ended up happening is that companies were able to take advantage of these fees by moving power around unnecessarily and at peak times, forcing the state and other consumers of the power to pay more. It was the rough power equivalent of having your cab driver take you from O'hare to the Loop via Milwaukee.

So yes, in this case, eliminate the rules that dictated that the price of power was based on the competition for transmission lines at a particular time (something the people controlling the transmission lines could easily inflate by moving power around unnecessarily) and the companies would not have been able to misbehave. The regulations gave the power companies the ability to set the prices for what they were selling.

So much for private entreprise.

[Pig Hogger]

40 years ago, Quebec nationalized all power-procuction and put it under Hydro-Quebec's umbrella. The State-owned corporation has never since failed to yield enormous profits (all going to the Quebec government - that's so much we won't have to pay in taxes), yet is providing the cheapest electricity in the world.

Private-entreprise zealots quickly lose steam whenever you point Hydro-Quebec at them as a shining example of profitable State ownership.

Calm down... don't forget Occum's Razor

[EmagGeek]

Look, people. There isn't anything or anyone to blame for this.

The Niagra Mohawk power grid serves the area in question. The way a power grid works is that there is a mesh of generation stations that are all interconnected by high-voltage transmission lines, 480kV on up. Each generation station has a primary service area and one or more (usually more) entry/exit stations where energy can either enter or exit the primary service area, depending on what they're telling the control system to do.

A network of generation stations makes up a grid, and at the boundary of a grid, there are similar entry/exit stations.

All generators, whether they be nuclear, hydro, wind, or whatever, have TONS of safety interlocks that engage at various points during abnormal conditions to prevent catastrophic failure. One of these interlock behaviors is to shut down and remove the generator from the grid in the event of an overload.

The likely sequence of events in this situation is that there was a failure at one of the generators in the N-M grid that resulted in the shutdown of that generator. What happens when a generator shuts down is that all of the entry/exit points flip to "entry" mode to allow neighboring generators to take up the slack. Most generator companies have agreements with their neighbors to buy however much electricity they need at whatever the current price is, without acknowledgement, when one of these shutdown events happens.

Anyway, once the initial generator shut down and the entry/exit stations flipped to entry mode, the neighboring generators were unable to take up the slack, so they in turn shut down as well. Then, a domino effect set in until it reached the boundary of the N-M grid, or when someone at the operator station woke up and hit the red button that prevents the transfer stations from automatically flipping to "entry" mode.

Keep in mind that it didn't necessarily have to be an overload that caused it - a generator can shut down for a number of reasons.

This all could have been a control system failure, an operator error, or some other unfortunate combination of events that happened to lead to a catastrophic grid failure.

Moment of silence

[abe ferlman]

Let's all have a moment of silence for all the linux and *bsd boxen whose legendary uptimes were mercilessly snuffed out by this service interruption. Some clung to their UPS's bravely until every ampere of juice had been drained.

They will be missed, and we will build even longer uptimes to replace them.

Re:Moment of silence

[rsax]

I know that your comment was meant to be humourous but on a more serious note, perhaps this event could serve as a wake up call to all the people who needlessly consume more than they have to (Yeah right, I can hope can't I?). I know so many geeks who run non-critical computers 24/7 which are chugging along just for the fact that they want to see those high uptime numbers. We all know *BSD/Linux is stable. You don't need to suck up power (pollute more by making fossil fuel power plants work extra hard) to prove something that's already obvious.

It went something like this...

[raehl]

One of the costs in getting power is that power's distribution - transmission lines cost money, and moving power over transmission lines thus costs money. California, as part of their "deregulation", put in place a system where the price of wholesale power was affected by not only the amount of power, but how the power got from point A to point B. The real pitfall with the rule was that it ALSO took into account supposed congestion on transmission lines - prices were set higher for wholesale power delivered when there was apparently a lot of people wanting to transmit power over given lines.

The obvious pitfall being that power line congestion could be artificially created - Enron (and others) took to moving power around more than necessary, creating more congestion, and thus artificially inflating the prices local power providers had to pay to get power over transmission lines.

Even worse, not only did the "deregulation" regulations allow Enron et. al. to artificially inflate the price of wholesale power, they ALSO prevented local power providers (the guys who actually delivered the power to your house) from raising the prices to make up for it, forcing them to sell power at a loss.

This could only go on so long before the local power companies started to run out of money, at which point they just said "Screw it", and instead of delivering power at a loss that they couldn't pay for anyway, they just stopped delivering power at all.

I can readily attest to this

[Hamster Lover]

I moved to Alberta from BC, to the left of Alberta on a map for those that don't know Canada, where the power is generally supplied by two companies, BC Hydro or West Koutenay Hydro (recently changed their name to something I can't remember). BC has abundant supplies of energy in the form of hydro electric power and depending on where you live you are supplied by one of these two companies.

My electrical cost in BC was more than half the KWh rate it is in Alberta, somewhere around 4.5 cents/KWh. On top of the KWh rate, I pay a consent fee and a storage rider and a whole host of bullshit fees that I did not see in BC because of REGULATION. I paid usage in KWh and that was it. I could even look on the meter and calculate my KWh usage and get a rough idea of what my bill was going to be (if you remember this from High School). You sure as hell can't do that here, who knows what the "storage rider" will be this month.

I have never understood the deregulation mentality; electricity is a necessity and business, especially high technology sectors, require and are attracted to cheap, reliable power. Deregulation has done none of that here in Alberta, costs are up and generation is down to maximize profit. I know several companies that locate themselves in BC due to the high demand they place on electricity, power that cannot be supplied by other provinces at such an attractive rate.

Now they are talking about partial deregulation of the BC market. Once again businesses small and large will get the shaft and the electrical producing companies will reap the rewards. Talk about robbing Peter to pay Paul.

Niagara Mohawk's Problems

[Anonymous Coward]

As at this time it has not been determined exactly why the Northeast Blackout of 2003 has occurred, there has been much speculation that it is due to Niagara Mohawk's grid failure.

Being an "insider", I would just like to say that the thought that is running through my mind about this is that several years ago, when the original Niagara Mohawk wanted to sell the company, suddenly all the engineering employees were told that they were no longer to perform preventative maintenance work on tranmission and distribution lines. For quite a period of time, they literally sat around with nothing to do. Then the company was sold to a British company, and this "hands off" attitude has continued.

So, I am very curious to know whether NiMo's lack of maintenance has something to do with today's problem. Another aspect of this problem is the fact that many long-time technical workers at Niagara Mohawk have either retired or been forced out, with their jobs not being filled. The crew sizes are down considerably. The amount of work never decreases - it mostly likely increases - but there are less KNOWLEDGEABLE people on board to handle such technical matters.

I truly hope that a full investigation into this matter is done, and if NiMo has dropped the ball, they be held accountable.

Re:Do I hear a chorus of 'Blame Canada'???

[Anonymous Coward]

I was waiting for it too - as soon as it came on that a Canadian part of the grid was down, I was thinking, "They'll be blaming Canada any minute now." Sure enough, right on cue, it starts flying back and forth from CNN to MSNBC to FOX - Canada Canada Canada Foreigners it's all the damn furriners.

Turned out not to be true, but honest to god the USA needs to get a grip. Not everything bad that happens is the fault of other nations. It's getting impossible to even talk to Americans these days - the concept of the USA being less than heavenly perfection personified, coming down from above to light the way for backwards and brown peoples who should shut up and do as they are told is rampant - you can see the rage rising behind their eyes when you even suggest that the USA is not to be envied in all things.

The USA is becoming strange and unpleasant. If it were a high school student it would be a wealthy jock, well-dressed, undeniably smart and handsome but with an ugly, arrogant soul. "They only hate me because they're jealous."

I know America - I like America. All the same America as a whole needs to rediscover a bit of humility.

Lets Play the Happy Terrorist Asshole Game!

[Greyfox]

What did the terrorists learn today? They learned that with their next major attack, two or three well placed attacks could plunge the entire nation into darkness, exponentially increasing the chaos. Anyone even want to think of what would have happened if they'd taken out that power station on 9/11?

On the other side of the coin, we just learned that two or three well placed attacks could plunge the entire nation into darkness and we can start planning now to make sure that doesn't happen. Do you think we will?

I'd start by mandating that towns either take their traffic signal systems off the main power grid or insure adequate backup power for them. The last thing we need in the middle of a blackout is traffic jams preventing emergency vehicles from getting where they need to go.

I'd also make sure hospitals and air ports have adequate backup capacity. Apparently a lot of them don't.

Then I'd have the Al-Capone Teamwork dinner with the CEOs of the various power companies, during which the NiMo CEO would get asked why one power station going down can take out a quarter of the nation's power. You know how that scene goes. Teamwork!

That'd be a good start I guess. Gives us something to do for the next 5 years or so.

Cause: Overloaded grid and bad logic

[Jimithing DMB]

The original post was very informative. EmagGeek was right on track when he mentioned that one generator got knocked offline for some reason and because of that the power grid compensated by rerouting electricity from other generators.

My dad was vice president of electric supply for NIPSCO for a number of years after having worked his way up the chain of command starting off working at a power plant as an electrical engineer. As VP of electric supply his job included ensuring NIPSCO was generating enough power to cover the needs of all the power customers (including several steel mills), working with regulators to ensure the rates were reasonable so that money could be spent to increase capacity when needed, and working with environmentalists to ensure that emissions were well below accepted government levels.

NIPSCO was a company very interested in serving its customers. As a heavily regulated utility the only reasonable business decision is to service your customers the best that you possibly can. My dad took that to heart. He was strongly opposed to deregulation. Why? Because the simple fact of the matter is that my dad was somewhat of an exception. Most executives tend to look strictly at the bottom line and lose sight of the forest for all the trees. He knew that deregulation would inevitibly lead to cost cutting in areas where costs should not be cut simply because without regulation the power company is at the mercy of its shareholders and shareholders are very often in it strictly for the money.

So, tonight I had a discussion with him about this mess. First of all, the background. Apparently a generator went off grid this afternoon forcing other generators to take up the slack. That can happen for a number of reasons. Equipment does fail, humans do make mistakes, etc. What's supposed to happen is that the rest of the generators and the grid should have enough capacity to take up the slack. Should there not be enough capacity then someone needs to lose power. This should happen at the customer side. That is, a portion of the customers should be blacked out to reduce the load on the grid and allow normal operations to continue. I believe that is what you meant by "putting a release on the chain." You are correct, that's what should have happened. The fact that it didn't indicates that there was some major problem with the logic of the grid. It would have been far better to cut the power to thousands of customers than millions.

Bad logic was part of the cause. The other problem was a seriously overloaded power grid. The power grid was designed to handle the situation where a power company normally had sufficient capacity but due to generator failures was unable to meet demand. Notice that I said failures (plural). If a few generators are knocked off the grid the company ought to have enough energy to supply all of its customers. Furthermore, it ought to have backup generators that can be started and on the grid within an hour. Those backup generators are just that: backups. They cost a hell of a lot of money to operate but they aren't as expensive to build as a main generator. If a few more generators get knocked off grid it's reasonable to expect that a power company will be unable to handle this situation without buying power from another company. That is what the grid is for.

Unfortunately, because of government stupidity (deregulation) and corporate greed the grid is now being used in a way it was never intended to be. It is often loaded to near full capacity drawing power over very long distances. The idea of deregulation was that loosely regulated for-profit companies would compete to generate electricity which the local power companies could purchase instead of generating their own. Because the power companies no longer had to be responsible for providing capacity in excess of what is needed the rates could be

Re:Cause: Overloaded grid and bad logic

[EmagGeek]

Thanks for the backup on this one.

While I don't necessarily agree that regulation is the answer, it's a simple matter of ethics.

De-regulation wasn't really de-regulation. It was RE-regulation. The rules simply changed, and there became many more of them, one of which was that no new generating plants could be built. Why the hell they decided this was beyond me. Most of these generators were built "way back when" before the age of computers and ubiquitous use of air conditioning. PECO Energy became the most expensive electricity in the nation after PA "de-regulated" the electricity industry. I pay almost $0.16 per kWh, which is ridiculous by any standard. That money is used to pay for electricity that is practically given away to neighboring producers like PP&L and ConEd.

Anyway...

You'd be AMAZED at what percentage of all generated power is dissipated in either a computer or an air conditioner/chiller/etc. 100 million computers at 200 watts each is 20 BILLION watts. 20 GIGAwatts. That's the capacity of more than 20 average-sized nuclear reactors. Limerick here in PA has two reactors each capable of about 1.134 (I was really hoping it was 1.21, really I was!) gigawatts.

Here's a Link to a list of all U.S. Nuclear facilities and their statistics and capacities.

And here is a link to a list of all the reactor statuses showing they're loaded to the teeth - almost all of them at 100%.

The U.S. Department of Energy maintains lots of useful information about the power grids in the United States and how they are running. There are also publicly available status reports on each generation facility.

One graph on the DoE site showed that generation capacity hasn't increased at all since about 1992 (when Clinton took office, what a surprise... bastard killed the military AND our power infrastructure... but that's another thread)...

It's not surprising that this happened since we've been increasing generation rapidly due to the deployment of computers and other tech gadgets, but not increasing capacity to make up for it. It also doesn't help that there's no incentive other than cost for people to use Alternative Energy like solar or wind. Well, that's not totally true, there are actually Lots and Lots of Incentives in some states for end-user renewable energy, but it's still really expensive.

insufficient margin

[Wansu]

This is not the first massive northeastern blackout. There were wipespread blackouts like this in the early 1960s. The engineers learned that all sections of the grid must have significant over-capacity designed in so that the entire system could recover from large transients and short duration system oscillation without tripping protection devices. They beefed up the system so it could ride through these events.

The safety margin is gone. Demand has grown but capacity has not. If lightning runs in on a substation, it can trigger a chain of events leading to a couple generator switchyards opening their air breakers. From there, the overload snowballs.

Does deregulation play a part? Yes. Power brokering activities create additional burden on the system. There is less incentive to increase capacity. There is also diffusion of responsibility.

The electric power industry was not broken prior to deregulation and didn't need fixing. It's infrastructure and regulated monopolies suck less than gov't run or private run ventures.

This is apt to get worse.

This had to do with the design of the grid!

[Newer Guy]

What causes cascades like this actually has very little to do with megawatts
and everything to do with frequency. See, every generator on the power grid is
syncronized to a common source. Indeed, before a power plant comes back on
line it must first syncronize its generators. The generators normally sit
there running at a boring 3600 RPM (60hz*60 seconds). All plants have a
monitor that kicks them off line if their frequency varies by more than +/- a
hz or so. As an aside, the power grid is not always EXACTLY 60 hz. The
frequency of the entire grid is allowed to float a bit, though drifts are
corrected so the frequency averaged over a certain time is a nominal 60 Hz.
The cascade happens when a either big plant or a big load suddenly goes off
line. In the case of a big plant the other plants try to take up the load, but
in the process their frequency drops as the generators get loaded more (much
like shifting a car's manual transmission to a higher gear before it hits the
right engine RPM). Once a generator drops below 59 hz, it also trips off
making it even harder for the ones left to keep up, and generators begin to
fall off the grid like dominoes.
The opposite happens when a load suddenly goes away, but in that case the
generators' frequency abruptly jumps upward, which also results in it tripping
off the grid. Either way the result is a cascade like happened today.

Once the dominoes (generators) begin to fall off, the grid becomes unglued.
There's an old saying in the power industry:
59.5 Hz = trouble. 59 Hz = BIG trouble!

I believe the new power management software mentioned in the news reports that
should have prevented this works by intelligently shedding loads distant from where the anomaly occurrs (for example, shedding load in NYC for an anomaly in Canada). This would give the generators time to react to the change. Obviously it didn't work.

I know EXACTLY what they'll do about this -read on

[Denver_80203]

"Today's failure is a dramatic reminder of the importance of the uninterrupted flow of power to the health, safety, and well being of our citizens and the defense of our country. "This failure should be immediately and carefully investigated in order to prevent a recurrence. "You are therefore directed to launch a thorough study of the cause of this failure. I am putting at your disposal full resources of the federal government and directing the Federal Bureau of Investigation, the Department of Defense and other agencies to support you in any way possible. You are to call upon the top experts in our nation in conducting the investigation. "A report is expected at the earliest possible moment as to the causes of the failure and the steps you recommend to be taken to prevent a recurrence." Signed, Lyndon B. Johnson

De-regulation and Competition

[sterno]

De-regulation works well when there is a competitive marketplace and it fails utterly when there isn't. Witness the airlines for examples of both how it succeeds and fails. If you are travelling between major hubs in the US, you have multiple airlines to choose from and the price you pay is pretty low. If you are travelling between off-hub points, then you pay a premium because it's likely only one or two airlines serve that route.

The electrical industry, much like the phone and cable industry is too dependent on the connection to the house to be truly competitive. Ultimately whoever controls the wires into the home runs the show and has a competitive (and frequently regulatory) advantage over anybody who would need to run new wires.

There seems to be this belief that privatizing and de-regulating are magical cure alls for many problems. They aren't. If a market is naturally prone to creating uncompetitive monpolies, then neither government nor private industry will make it more efficient over the long run. Thus you are better off with government where at least the motivations are to please the citizenry rather than please the shareholders.

A "fair and balanced" analysis

[useosx]

Greg Palast takes a look at why the lights went out.