Slashdot Mirror
Satellite Views Of The Blackout
An anonymous reader writes "These Before and After satellite views of the blackout, from the NOAA, show the geographic extent and intensity of the outage. Toronto, Ottawa, and Detroit seem the worst hit. Currently, a cnn article mentions that a reverse of power flow around Lake Erie may have caused an overload that triggered the programmed shutdown of the power grid. Would be interesting to know how the system and software works, but then again, that information could be dangerous in the wrong hands."
[source: North American Electric Reliability Council press briefing, press releases Aug. 15; files]
"WE DESIGNED THE SYSTEM FOR THIS NOT TO HAPPEN." At a phone briefing this morning, which had more than 700 reporters on the line, NERC CEO Michael Gent said he was "embarrased" by the widespread black-out yesterday, because NERC was created after the massive 1965 East Coast black-out to prevent such an occurrance from repeating itself. But it is clear that the system NERC's rules were created for no longer exists.
A reporter asked Gent how the system NERC designed has changed under deregulation. "When the industry deregulated," he said, "the owners of transmission and generation were separated. You no longer have transmission to match the generation." Power producers "built for convenience to their system load, without concern for the transmission," he said. This has led to the overloading of power lines and reduced reliability.
{EIR} asked about the impact of the "economy transfers" of bulk power over the system since deregulation, where companies "wheel" cheaper power for hundreds of miles, which the system was never designed for. "They have added congestion," to the system, Gent replied. "We thought we were on top of these added transfers," but they will be looked at during NERC's investigation of the black-out.
In 1965, an outage on a 230-kilovolt transmission line in Canada led to a series of failures that in minutes resulted in power swings that produced a cascaded outage, blacking out 30 million people down the East Coast for up to 13 hours. NERC was formed in response.
In July 1977, when a transmission tower north of New York City was struck by lightning, the system collapsed. While 9 million people in New York City were left in the dark for up to 26 hours, the problem was isolated, and no other systems were affected. The reliability rules NERC had put into effect, worked. "We never have anticipated a cascading outage, since 1965," Gent stated today.
But, that was then, and this is post-dereg. [mgf]
[as above]
NERC INVESTIGATING TEAM WILL WORK TOWARD REPORT NEXT WEEK. Michael Gent announced that he was forming a task force of "forensic experts" to investigate the causes and circumstances of the black-out. He said at the press briefing that while it would take months to have a final report, "we should know next week what happened." [mgf]
[as above]
NOT TERRORISM, BUT COLLAPSE OF INFRASTRUCTURE. During the NERC briefing, Gent explained that while they haven't pinned down the initiating event for the black-out, they do know that the problem was in the transmission loop around Lake Erie and Lake Ontario. This Loop connects up-state New York west around Erie to Ohio, north to Detroit, and through Canada back east to New York. This Great Lakes area has been a well-known problem for years, Gent said, and cables under Lake Erie were planned to relieve the congestion, but never built.
On the question of terrorism, Gent said that there was no evidence of any physical attack on any infrastructure. As far as cyber terrorism, he said that they have logs of all communications and computer activities at all critical facilities. If there were a "cyber intrusion," he said, the person would leave tracks. They wouldn't necessarily who did it, but they would know it took place.
Leading to the black-out, the destabilization of the transmission system took only 9 seconds. About 50 million people were affected, as some Canadian and 9 U.S. nuclear plants went off line, as well as more than 80 fossil fuel plants.
Gent said that power was being restored faster than he'd anticipated. According to NERC, at 24 hours after the incident, about 20,000MW of capacity was still down, out of more than 61,000MW lost during the outage, which was about 10% of all the capacity east of the Mississippi River. [mgf]
[source: EIR files]
AMPLE WARNING WAS GIVEN THAT THE GRID WAS AT RISK. For more than a decade, NERC has been sounding alarm a
Read the reports in various newspapers, you will have come across many articles saying how antiquated the power transmission system has become.
Power companies have specifically stated that putting in new power grids is very problematic because people don't want this anywhere near their property.
This view is exactly like those bastards at Cape Cod. They scream themselves hoarse that they are enviornmentalists and then fscking say no to wind mills 6 miles off the shore.
Same thing with this power grid. Companies that want to lay new power grids cannot go foward and lay lines because the residents will not waste anytime taking them to court. "We don't want it in our backyard".
Well, somebody has to pickup the cost.
Also, Canada has an excess of power generation capacity. If the US had better lines, it could have taken up the excess power Canada generates.
[ "The strain on transmission capacity is particularly acute in New York State, which is known in the industry for having far too few high-voltage power lines",
"community resistance to new lines has been high and continues to prevent new lines from being built, particularly in high-density areas like the northeast. While the federal government can step in and insist on construction of natural gas pipelines, it has no such power related to electrical transmission lines. "People want more power, but they don't want those lines"".
"Most of New York City's and Long Island's power at peak times must be generated in the city and on the Island, because it is physically impossible to transmit that much power into the area along the existing lines." ]
Maybe its just me, but those pictures are rather inaccurate.
The after picture doesn't have a single light on anywhere near Ottawa. However, there is a large urban area right across the river from Ottawa, cnosisting of Aylmer, Hull and Gatineau, all Quebec cities. We all had power here.
Have you had a chance to compare this picture with a concurrent cloud map? IIRC, optical imaging has problems seeing through clouds...
Thankfully. Any decent electrical engineer knows something about hvac grids and tripping them. They are supposed to work exactly how they worked. I suspect that the grid frequency went down below a certain threshold and the grid tripped. This is a very essential feature because all the machinery on the grid and off the grid is designed for 60hz power freq. If it goes any lower, their efficiences wud go down hill and each piece will produce tremendous amounts of heat, which can possibly lead to mass fires/transformer explosions across the grid.
Why this happens is simple - the generators are asked to provide more power can they can ever generate - and they slow down(just like a motor loaded with a mechanical load)- also some generators that are supposed to come on-line but they didnt. Thankfully the grid equipment works by sensing - you know it - the grid, which is good because asshats cant interfere with it without getting soot on their hands. I am glad that cows like the ones that frequent slashdot dont write software for the grids. Its done by more deligient and more perfectionist electrical engineers (I am one - any doubts??).
As far as terrorists are concerned,I wouldnt worry too much about they getting this info (they have it already), because all it takes to know info about a grid is a decent electrical engineer, of which there is no dearth - american or non-amercan.
The after picture was 7 hours after...About 11PM
I used to travel to Columbus for work a lot, and frankly am not surprised to see it dark by 11PM!
Not the greatest night spot on earth IMHO...
It may have stopped short of Rhode Island, but apprently it may have actually started in my home town. Check this.
--Kevin
Most universities have couses on power systems. As a mechanical engineering student, I took several courses on nuclear power plant design and operation. These classes included several tours to working power plants and training sites. This information is not really hard to get.
I can't speak on power plants in general, but I can comment a bit on nuclear plants. Most plants running in the US are quite old, thanks to public perception preventing any new plants from being built. So, most of them run pretty old systems. Most I've seen run on unix variants, mostly HP-UX and AIX. The software used is really just a backup, the plants can operate pretty much without the computer systems. The hardware is pretty much big old mainframes and mini-mainframe type stuff. IBM, Sun, HP, etc.
The primary function of the computer systems it to simplify some operations and to more easily report on conditions. For example you can view the power output of both reactors on one screen at the control center rather than having to walk over to the analog dials to check it out. They also monitor safety systems and can report on the state of different valves and things in the plant, rather than requiring you to go look at all the lights for individual valves.
Most plants are starting to modernize and new software is being developed to allow complete control of the plant. Currently most of the software used is for monitoring only, but it's starting to be deployed for control as well. So, rather than having to walk over and switch a lever to close a vavle, or turn a dial to up reactor power, you can just click. But this isn't really widespread yet.
There is some windows software out there for this stuff, but it's not widely used, at least in the US. Some of the newer advanced control systems are focusing on windows, so it looks like in the future there might be more windows in the plants.
Which happens at most doppler sites with or without buildings coming down. This is the doppler at Brookhaven (Upton NY). If there's nothing else (emerging cloud tops, big storms) to look at, radar is usually aimed pretty low and this looks like ground clutter - moisture is a typical culprit. This one was about 6 AM local, the artifact is centered over the doppler location, not the WTC, you can see one like it on most unremarkable weather days. Here's a FAQ image from AccuWx -m
http://www.accuweather.com/iwxpage/paws/ex2.ht
"Win treats sysadmins better than users. Mac treats users better than sysadmins. Linux treats everyone like sysadmins."
http://www.pbs.org/wgbh/pages/frontline/shows/cybe rwar/vulnerable/grid.html
I've got a composite of the difference that the blackout made areas that were darker during the blackout are in red. Areas that were bright at both times are white.
Free Software: Like love, it grows best when given away.
I wasn't going to get into this bit, but so many recently have gone on and on about power deregulation. Last year, power deregulation was blamed for CA's issues; now it's deregulation that has caused this. This was a simple case of a failure of the automatic systems followed by a failure of the human systems.
Let's start with CA. Deregulation, I think not! A system that puts a price ceiling in place is not deregulation. CA's issue was simply a combination of:
- A recent reluctance to add any "real" generating power. I'm not talking a co-gen here, a co-gen there. I'm talking power plants that generate around 1000Mw. In the past years, CA's added somewhere near 300Mw total.
- An increase in load.
- And the number one reason, a price ceiling that prevented the delivery companies from buying power at market rate.
Now we hear a group of people crying deregulation in this particular instance. What part of deregulation caused this? Nothing. The simple answer is that there was a failure. The automatic systems didn't handle it. The humans could have; they didn't. The same thing happened in 1965. And with similar results. The North East went dark. So where did the power stay on? Pennsylvania
Why did it stay on here? Well, as history tells us, the person in charge of one of PA's power companies saw the load increase significantly and got on the phone with the neighboring manager. The phone call went something like this:
"We have a problem, you must shed NY now!"
"I don't know if I want to do that."
"You will shed NY by the count of 10 or I will shed you and you will go down with them! 1 2 3 4 5 [load drops] thank you."
This is how the power system is supposed to work. When load goes to high, you get shed it and there is a small blackout. Based on what I'm hearing there was a 1000Mw differential in the direction power was flowing around the great lakes. At the moment that happened, someone should have jumped to look for places to isolate. Apparently the PA connects did. We stayed up once again.
I just talked about PA's proper implementation of the procedures and the benefits we've obtained from them. I know that's not necessary due to deregulation, but I'd also like to point out that we do have a true model deregulation. We deregulated power a few years back. I chose my power generation company a few weeks back. I had the choice of a few different providers (some offer green-only, some produce the power cheaper). All in all, we have affordable, reliable power.
Basically we didn't mess with the system, we let the market handle it when it came to price and we followed the procedures put in place when it came to disasters.
... there are textbooks and IEEE tutorials. It's too complicated to explain in a simple message.
The devices that did the shutdown were probably the protective relays that are individual devices. During the Y2K effort I saw a log file from one of them at a presentation and it made me think of a single board processor running MS-DOS, but it might have been only a logging processor and not the hardware that does the work.
The control centers run variants of Unix, with some running NT. The control centers monitor the field devices, allow some of them to be controlled, and run some very sophisticated prediction algorithms to see what alarms would result from hundreds of contingencies. The algorithms essentially simulate the failures and figure out what else would fail as the power redistributes itself to compensate.
The investigation will need to figure out why those predictions didn't alert operators to change the configuration of the grid to stop the cascade that happened.