Slashdot Mirror

← Back to Stories (view on slashdot.org)

US/Canada Power Outage Task Force Event Timeline

Posted by michael on from the alone-in-the-dark dept.

bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."

18 of 303 comments (clear)

  1. text version (aka karma whoring) by glassesmonkey · · Score: 5, Informative

    12:05:44 - 1:31:34 PM - Four Generator trips

    2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio

    3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio

    3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio

    4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan

    4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania

    4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York

    4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio

    4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection

    4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.

    4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out.

    1. Re:text version (aka karma whoring) by K_J_Raine · · Score: 5, Funny

      Ah, so the Canadians weren't responsible for the blackout... as the nice mayor of New York originally contended.
      Think he must have watched a bit too much South Park!

      --
      There is only one satisfying way to boot a computer. -- J. H. Goldfuss
    2. Re:text version (aka karma whoring) by connsmythe96 · · Score: 5, Funny

      You left out:
      Friday September 12, @03:58PM: Server goes down due to unusually high traffic.

      --
      if(!cool) exit(-1);
    3. Re:text version (aka karma whoring) by PD · · Score: 5, Funny

      It's not just us. The English used to have a saying: "Poor Canada: so far from God, so close to the United States."

      --
      If tits were wings it'd be flying around.
    4. Re:text version (aka karma whoring) by yeti-graf · · Score: 5, Funny

      The tragedy of Canada is that they had the opportunity to have British culture, French cuisine, and American technology. Instead they ended up with American culture, British cuisine, and French technology.

  2. The blame game by trolman · · Score: 5, Interesting
    The really bizarre part that has not been explained is that the frequency deviated. I saw frequency deviation alarms on my equipment (ups and pdus) on the data center floor in NY, NJ, and CT and this is the first time in my 20 years that this has happened. A glaring ommission is is that PJM stayed up as an entity. Kudos to PJM.

    The blame will be put onderegulation and lack of government oversight.

    1. Re:The blame game by morcheeba · · Score: 5, Informative

      If they're using AC generators (which I suspect provides most of the power - they're used in coal, nuclear, and hydro plants, but not solar, and not newer high-voltage DC transmission lines), then the frequency output is related to the spin of the shaft. Two things control the speed of the shaft - power in (water pressure) and power out (load demand from the grid), and a control system tries to keep the frequency a constant 60 Hz.

      The trick is that the control system can only react so fast - suddenly disconnect an entire town, and the load drops, causing the power in to spin the generator too fast. If the control system overcorrects, then you'll get too low of a frequency. If a far-away generator drops out and you've got to supply more current to your local region, then the demand has gone up, slowing the frequency.

      If you've been around generators, you can hear this exact phenonemoa - if the load changes suddenly, the motor will hunker down a little and then catch back up to normal speed. Usually a flywheel can damp out extremely short transients, but it would be prohibitively big if it were sized to handle transients as large as the control system (throttle) will allow.

      --
      HIV Crosses Species Barrier... into Muppets
    2. Re:The blame game by Orne · · Score: 5, Informative

      It's not that bizarre if you think about it... the bulk power frequency is actually one big juggling act between all of the generators that are synchronized on the system...

      The Eastern Interconnection (everything in North America east of the Rockies and north of Texas) is tied together at many stations, such that there are many parallel paths to deliver energy to a customer load, providing an excellent level of stability. Simply put, the frequency is the prime measure of the balance between energy production and consumption. Energy generation is not a smooth process, it spikes as fuel is delivered and burned. If enough generators are synchronized with one another, they can automatically cover for each other's dips, and thus the frequency stays balanced.

      Now, when the system split, imagine you had all of the generation on the west side, and all the load on the east side. For those of us in PA, we saw a huge loss of load, and the frequency shoots up. For those on the wrong side of the blackout, you suddenly lost your generation source, and your frequency drops.

      Transmission equipment is easily damaged at low frequencies, so many are equipped with underfrequency relays that open breakers to protect themselves. What happened is that lines tripped and load sheds, forming smaller and smaller zones, until there were only small pockets of load and generators remaining (see the notes on western NY). Without the rest of the interconnection to syncronize with, your local generator was trying to maintain the frequency by itself as best it could, and was probably all over the map due to uneven fuel burn. Then, a few minutes later, you might have auto-reclosing of breakers (try-backs). If a line trips, some are programmed to auto-reclose, which, in an event like this, can suddenly add thousands of MW of load to an already stressed system, pulling the frequency down even more until everything is black.

    3. Re: The Blame Game by shoemakc · · Score: 5, Funny


      "It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines."

      Wait a minute, could that be the problem? I hear in Canada they use cosine waves. ;-)

      -Chris

      --
      --an unbreakable toy is useful for breaking other toys--
  3. the timeline... by fuckfuck101 · · Score: 5, Funny


    12.04 - power on
    14.11 - power off

    --
    Comment: Yes I realise the username 'fuckfuck101' makes me sound intelligent, no you cannot buy it from me.
  4. MSBlaster.exe by devphaeton · · Score: 5, Interesting

    Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K) getting nailed by the Blaster worm? The timing is right, and there are a number of indications thereof:
    See:

    this or
    this or
    this.

    --


    do() || do_not(); // try();
    1. Re:MSBlaster.exe by insecuritiez · · Score: 5, Insightful

      A lot of people suspect that. I personally think it's the best explanation. However, even if Blaster caused the outage and every "expert" at the plant knew it, it would NEVER be published that way. That would open up a whole new can of worms in the public eye. A security and publicity nightmare. No, if Blaster caused anything that issue will be quietly swept under the rug. Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

  5. Bloomberg by oZZoZZ · · Score: 5, Interesting

    Mayor of NYC decided to publicly blame Canada for this before any facts surfaced.. while Lastman, the mayor of Toronto said something along the lines of "Do you expect the US to take blame for anything?", after *some* facts surfaced
    Neither responses were politically acceptable, however the media coverage of the blame game seemed to evaporate as soon as it was clear that it wasn't Canada's fault.
    I found that more than a little interesting.

    1. Re:Bloomberg by ergo98 · · Score: 5, Insightful

      To be fair, before Bloomberg made his idiotic statement (which included the words "absolutely certain") various levels of the Canadian government had blamed it on a New York power station, and then a Pennsylvania nuclear power plant fire (information they got from the US Department of Defense, as a sidenote, but nonetheless it was irresponsible to repeat it verbatim so early on). I think a big difference though is that when I hear that it's a New York power plane or a nuclear station in Penn, most Canadians don't think "Those damn Americans!", or any "us versus them" nonsense. Instead it's a power plant going down, and why the hell is this grid so fragile? I suspect for Bloomberg and crew, though, it's nothing of the sort: Not only is it not "Ontario", it's "Canada" (which gives you an idea of the perception right there), but there's a definite slant of "Well that explains it right there!". This is par for the course for the various levels of government in New York state, though: Hillary Clinton has made countless nonsensical statements about how Canada is to blame for every fault in her little fantasy world.

  6. Late Night "Wacko" Talk Radio by GSpot · · Score: 5, Interesting

    A couple of days after the blackout, I was staying up late and unfortunatley listed to some late night radio. Two different kooks were speculating on the cause of the blackout. One was positive that the Federal government was testing out some "advanced" weapon and the other freak was convinced that the culprit was the power companies looking for an edge to justify raising rates to build more infrastructure.

    just my .02$

  7. What they *are* confident of... by Not_Wiggins · · Score: 5, Funny

    "We're can't say for certain what happened and when, but we can say with almost 90% certainty that when the power went out, people went without power. We think it might be related to some electrical do-hickey thingy that someone was supposed to be watching carefully in case it broke, but we're not confident enough to make that bold a statement... nor to claim that anything actually 'broke'.

    "But what we *can* say is that we feel strongly that we feel something different should have happened. An appropriate amount of blame will be laid... oh yes... and we'll make the bad people pay."

    --
    Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
  8. Alternate Source by Hal+The+Computer · · Score: 5, Informative


    http://www.nrcan-rncan.gc.ca/media/documents/Black out_Summary.pdf

    I think I will be fair and equitable and allow Slashdot to take out a Canadian website as well. Please be kind to Natural Resources Canada.

    --

    int main(void){int x=01232;while(malloc(x));return x;}
  9. The deal with the frequency by Spamalamadingdong · · Score: 5, Informative
    Others have kind of poked at this, but they haven't really explained it for the neophyte. I've had some education in electrical power engineering, so I'll try to fill that gap.

    There are two things you need to keep in mind here. The first is that phase in AC systems performs much the same function as voltage in DC systems; just as power flows from higher voltage to lower voltage across a DC connection, power flows from leading phase to lagging phase along an AC connection. (This has to do with reactance; all power lines are inductive.) Counterintuitively, voltage helps move power but it mostly balances VARs (volt-amperes reactive); if you have a local low-voltage situation, you can connect a capacitor to add some VARs and the voltage will come up. This is part of why big inductive loads cause line voltage to dip.

    The second thing is that frequency variation is just a phase change over time. If the local frequency falls for a bit, it means that the local phase is moving behind the rest of the grid. This is what you would expect if some large load was added (or a generator lost) and more power had to come from elsewhere on the grid; the delta-phase across the interconnecting lines has to shift to allow more power to flow. What little energy buffering there is is mostly the rotational energy of generators and motors, so phase changes don't quite happen instantaneously.

    If you had a serious local power shortage leading to shutdown, under-frequency is exactly what you would expect. Generators trip off-line, and the phase of the local grid backs off to pull more power from outside. It would take a full second at 59 Hz to shift one cycle, so this can go on for a fair fraction of a second. If the phase change over a transmission line increases past 90 degrees it will have to trip off-line, and once the local grid is an island you can have just about any frequency that the system will try to operate at. It's my understanding that most generators trip off-line at more than a fractional Hz off 60, if for no other reason than that they aren't designed or certified to operate on a grid that's obviously malfunctioning and such a condition means trouble. Mechanical resonances at off-operating rotational speeds are another reason to shut down.

    Last, I suspect your conclusion is correct.

    --
    Scientists restrict study to entire physical universe; creationist