Slashdot Mirror


US/Canada Power Outage Task Force Event Timeline

bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."

34 of 303 comments (clear)

  1. text version (aka karma whoring) by glassesmonkey · · Score: 5, Informative

    12:05:44 - 1:31:34 PM - Four Generator trips

    2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio

    3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio

    3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio

    4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan

    4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania

    4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York

    4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio

    4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection

    4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.

    4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out.

    1. Re:text version (aka karma whoring) by K_J_Raine · · Score: 5, Funny

      Ah, so the Canadians weren't responsible for the blackout... as the nice mayor of New York originally contended.
      Think he must have watched a bit too much South Park!

      --
      There is only one satisfying way to boot a computer. -- J. H. Goldfuss
    2. Re:text version (aka karma whoring) by connsmythe96 · · Score: 5, Funny

      You left out:
      Friday September 12, @03:58PM: Server goes down due to unusually high traffic.

      --
      if(!cool) exit(-1);
    3. Re:text version (aka karma whoring) by PD · · Score: 5, Funny

      It's not just us. The English used to have a saying: "Poor Canada: so far from God, so close to the United States."

    4. Re:text version (aka karma whoring) by yeti-graf · · Score: 5, Funny

      The tragedy of Canada is that they had the opportunity to have British culture, French cuisine, and American technology. Instead they ended up with American culture, British cuisine, and French technology.

  2. It's not the end of the world by Brahmastra · · Score: 3, Funny

    A taskforce to survive power outages.... Lets show the world that we are stronger than ice-cream.

  3. I CAN'T READ IT. MY SCREEN IS ALL BLACK. by Anonymous Coward · · Score: 4, Funny

    n/t

  4. The blame game by trolman · · Score: 5, Interesting
    The really bizarre part that has not been explained is that the frequency deviated. I saw frequency deviation alarms on my equipment (ups and pdus) on the data center floor in NY, NJ, and CT and this is the first time in my 20 years that this has happened. A glaring ommission is is that PJM stayed up as an entity. Kudos to PJM.

    The blame will be put onderegulation and lack of government oversight.

    1. Re:The blame game by morcheeba · · Score: 5, Informative

      If they're using AC generators (which I suspect provides most of the power - they're used in coal, nuclear, and hydro plants, but not solar, and not newer high-voltage DC transmission lines), then the frequency output is related to the spin of the shaft. Two things control the speed of the shaft - power in (water pressure) and power out (load demand from the grid), and a control system tries to keep the frequency a constant 60 Hz.

      The trick is that the control system can only react so fast - suddenly disconnect an entire town, and the load drops, causing the power in to spin the generator too fast. If the control system overcorrects, then you'll get too low of a frequency. If a far-away generator drops out and you've got to supply more current to your local region, then the demand has gone up, slowing the frequency.

      If you've been around generators, you can hear this exact phenonemoa - if the load changes suddenly, the motor will hunker down a little and then catch back up to normal speed. Usually a flywheel can damp out extremely short transients, but it would be prohibitively big if it were sized to handle transients as large as the control system (throttle) will allow.

    2. Re:The blame game by 4of12 · · Score: 4, Funny

      what exactly causes frequency deviation? I'm not terribly familiar with it

      Glad you asked.

      It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines.

      This frequency is related to the 60 Hz rate of revolution maintained by most generators.

      Occassionally, something will happen to upset this steady rate of revolution of the generators.

      In particular, Homer, a donut that falls into the space between the stator and armature will cause this problem.

      Lubrication from jelly-filling in the donut can help stave off the inevitable disaster of frequency deviation, but it is simply an old-wive's tale to believe that pouring coffee into the space will ungunk the works in time.

      --
      "Provided by the management for your protection."
    3. Re:The blame game by Orne · · Score: 5, Informative

      It's not that bizarre if you think about it... the bulk power frequency is actually one big juggling act between all of the generators that are synchronized on the system...

      The Eastern Interconnection (everything in North America east of the Rockies and north of Texas) is tied together at many stations, such that there are many parallel paths to deliver energy to a customer load, providing an excellent level of stability. Simply put, the frequency is the prime measure of the balance between energy production and consumption. Energy generation is not a smooth process, it spikes as fuel is delivered and burned. If enough generators are synchronized with one another, they can automatically cover for each other's dips, and thus the frequency stays balanced.

      Now, when the system split, imagine you had all of the generation on the west side, and all the load on the east side. For those of us in PA, we saw a huge loss of load, and the frequency shoots up. For those on the wrong side of the blackout, you suddenly lost your generation source, and your frequency drops.

      Transmission equipment is easily damaged at low frequencies, so many are equipped with underfrequency relays that open breakers to protect themselves. What happened is that lines tripped and load sheds, forming smaller and smaller zones, until there were only small pockets of load and generators remaining (see the notes on western NY). Without the rest of the interconnection to syncronize with, your local generator was trying to maintain the frequency by itself as best it could, and was probably all over the map due to uneven fuel burn. Then, a few minutes later, you might have auto-reclosing of breakers (try-backs). If a line trips, some are programmed to auto-reclose, which, in an event like this, can suddenly add thousands of MW of load to an already stressed system, pulling the frequency down even more until everything is black.

    4. Re: The Blame Game by shoemakc · · Score: 5, Funny


      "It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines."

      Wait a minute, could that be the problem? I hear in Canada they use cosine waves. ;-)

      -Chris

      --
      --an unbreakable toy is useful for breaking other toys--
  5. Future Prevention by LamerX · · Score: 3, Interesting

    Hopefully this is something that they can actually use to learn from. Seems to me that they should put safeguards in between each one of these events. Because if just ONE of these events could have been stopped, it would have stopped the whole east coast from losing power.

    1. Re:Future Prevention by dirc · · Score: 3, Informative
      Energy companies are finding it cheaper to buy electricity on the open market instead of generating their own.

      If a supplier finds it cheaper to buy from someone else rather than produce his own, it indicates either: (a) the supplier is a less efficient producer. Someone else is able to do it more cheaply., or (b) the cost of building new capacity is so high that the cash flow from the new capacity does not justify the cost to build it.

      If the problem is (a), then as a consumer, you want the supplier to outsource the work to someone else, because you get a cheaper product.

      If the problem is (b) and the demand for the product is increasing in the long run, then as a consumer you will likely start to face spot shortages and price spikes caused by those shortages, since the outside supplier will eventually have less and less excess capacity to sell.

      California has had experience with (b) in two markets, the electric power market and the gasoline market. In the former, the power outages were caused by high demand and insufficient generation capacity (only part of which was due to attempts to manipulate the spot price of energy). The major culprit was the length of time it took to get State regulatory approval to build new power plants. When Gov. Davis finally realized there was a problem, he relaxed the approval process and dozens of new plants were built.

      The gasoline market suffers from regulatory restraint as well. Since 1985, not a single new oil refinery has been built in CA. Ten have closed. (See this for info. Now, refineries are running at over 90% capacity, rather than the 65% capacity they ran at in the mid-80s. Every time a refinery has a problem, the supply is out-stripped by demand and prices spike. The refiners, needless to say, love this. The are making huge profits because the government has imposed an almost complete barrier to new competition. (Presumably there is some price of gas at which it would be profitable to run the regulatory gauntlet.) The situation will get worse, since it is expected that in the new 5 years about 10% of CA refining capacity will be closed down.

      As the two examples suggest, regulation of the market between suppliers and consumers is not the issue. The issue is government regulation by of the suppliers which restricts capacity. The government has some legitimate reasons for those regulations (such as protecting the environment), but their citizens must accept the consequences of growing demand and static supply. Those consequences will include outages, shortages and higher prices.

  6. The last entry: by Faust7 · · Score: 4, Funny

    5:08 PM - U.S./Canada Power Outage Task Force web server goes out.

  7. the timeline... by fuckfuck101 · · Score: 5, Funny


    12.04 - power on
    14.11 - power off

    --
    Comment: Yes I realise the username 'fuckfuck101' makes me sound intelligent, no you cannot buy it from me.
  8. MSBlaster.exe by devphaeton · · Score: 5, Interesting

    Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K) getting nailed by the Blaster worm? The timing is right, and there are a number of indications thereof:
    See:

    this or
    this or
    this.

    --


    do() || do_not(); // try();
    1. Re:MSBlaster.exe by insecuritiez · · Score: 5, Insightful

      A lot of people suspect that. I personally think it's the best explanation. However, even if Blaster caused the outage and every "expert" at the plant knew it, it would NEVER be published that way. That would open up a whole new can of worms in the public eye. A security and publicity nightmare. No, if Blaster caused anything that issue will be quietly swept under the rug. Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

    2. Re:MSBlaster.exe by gregmac · · Score: 4, Interesting
      Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K)

      I've still never understood this. I think most systems are actually based on NT, but maybe they are migrating to 2k now. Either way, the fact that these automation systems are based on a system like windows is very strange to me. OPC (the protocol used to communicate between sensors and databases) is based on DDE (or OLE), which seems so incredibly strange to me.

      I've been developing a linux-based SCADA system. I took a look at quite a few systems, and I just didn't feel comfortable running any of them for a number of reasons. Stability and security being two major issues. Another was cost - these are being deployed in small installations, mostly for remote monitoring, which wouldn't typically have a SCADA system due to the cost. Between a mixture of existing open source software, some nice hardware, and in-house development (mostly me), the system has cost us about $20k to develop, which is less than it would cost to licence most software per site.

      Anyways, that was a bit OT, but the point is, very early on we decided that deploying on windows would be a pain. These are all remote installations, with no one on site that can service them. If something goes down, I want to be able to remotely fix it as much as possible. I just don't feel comfortable deploying a remote windows system and relying on it to stay running, not to mention the fact that people's health could be affected (water treatment). To me, windows is not the proper platform to be using for this situation.

      --
      Speak before you think
  9. Bloomberg by oZZoZZ · · Score: 5, Interesting

    Mayor of NYC decided to publicly blame Canada for this before any facts surfaced.. while Lastman, the mayor of Toronto said something along the lines of "Do you expect the US to take blame for anything?", after *some* facts surfaced
    Neither responses were politically acceptable, however the media coverage of the blame game seemed to evaporate as soon as it was clear that it wasn't Canada's fault.
    I found that more than a little interesting.

    1. Re:Bloomberg by ergo98 · · Score: 5, Insightful

      To be fair, before Bloomberg made his idiotic statement (which included the words "absolutely certain") various levels of the Canadian government had blamed it on a New York power station, and then a Pennsylvania nuclear power plant fire (information they got from the US Department of Defense, as a sidenote, but nonetheless it was irresponsible to repeat it verbatim so early on). I think a big difference though is that when I hear that it's a New York power plane or a nuclear station in Penn, most Canadians don't think "Those damn Americans!", or any "us versus them" nonsense. Instead it's a power plant going down, and why the hell is this grid so fragile? I suspect for Bloomberg and crew, though, it's nothing of the sort: Not only is it not "Ontario", it's "Canada" (which gives you an idea of the perception right there), but there's a definite slant of "Well that explains it right there!". This is par for the course for the various levels of government in New York state, though: Hillary Clinton has made countless nonsensical statements about how Canada is to blame for every fault in her little fantasy world.

  10. Late Night "Wacko" Talk Radio by GSpot · · Score: 5, Interesting

    A couple of days after the blackout, I was staying up late and unfortunatley listed to some late night radio. Two different kooks were speculating on the cause of the blackout. One was positive that the Federal government was testing out some "advanced" weapon and the other freak was convinced that the culprit was the power companies looking for an edge to justify raising rates to build more infrastructure.

    just my .02$

  11. What they *are* confident of... by Not_Wiggins · · Score: 5, Funny

    "We're can't say for certain what happened and when, but we can say with almost 90% certainty that when the power went out, people went without power. We think it might be related to some electrical do-hickey thingy that someone was supposed to be watching carefully in case it broke, but we're not confident enough to make that bold a statement... nor to claim that anything actually 'broke'.

    "But what we *can* say is that we feel strongly that we feel something different should have happened. An appropriate amount of blame will be laid... oh yes... and we'll make the bad people pay."

    --
    Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
  12. BitTorrent link by mskfisher · · Score: 4, Informative
    I've started up a BitTorrent mirror of the PDF here:
    http://www.mskf.org/BlackoutSummary.torrent
    --
    0x0D 0x0A
  13. Disturbances Before Outage Itself by shoemakc · · Score: 4, Interesting


    Did anyone else notice a strobing effect in their fluorescent lighting in those 20-30 seconds before the full power outage? My understanding is that any sort of arc lamp (fluorescent, metal halide) will extinguish if the voltage sags beyond a certain point, so I doubt it could have been a voltage sag before the full blackout.

    It almost seemed as if the power frequency itself had gone unstable...say from a nominal 60Hz to like 5Hz. Then again, with the modern electronic ballasts used today, who knows how they respond to a voltage sag. Maybe they strobe. Any one have any thoughts on this?

    -Chris

    --
    --an unbreakable toy is useful for breaking other toys--
  14. Alternate Source by Hal+The+Computer · · Score: 5, Informative


    http://www.nrcan-rncan.gc.ca/media/documents/Black out_Summary.pdf

    I think I will be fair and equitable and allow Slashdot to take out a Canadian website as well. Please be kind to Natural Resources Canada.

    --

    int main(void){int x=01232;while(malloc(x));return x;}
  15. Re:Imagine if... by NerveGas · · Score: 3, Insightful


    I'm all for it.

    I have a propane camping stove, plenty of propane, and plenty of canned food around. The servers I'm responsible for will stay running, the data center has its own generators, but our office will be dead.

    That means I'd get to sit at home and play Monopoly all day. If I get bored of that, there are about a hundred things I can do for fun that don't require a bit of electricity.

    Shoot, it would probably even be beneficial to people in my neighborhood. I'll bet that the park down the street from me would be teeming with people outside, enjoying wholesome activities and human interaction.

    Instead of neighbors walking around the block looking for code violations to report to the city, they'd probably be actually interacting with each other, maybe even solving their problems without running to a baby-sitting city government!

    I'm sure there would be consequences. Analysts would talk about how many hundreds of trillions of dollars were lost, but in the end, we'd all go back to work, take care of the stuff that didn't get done, and we'd have had a good time while it lasted.

    Now, if the power outtage also included me being somewhere like the island of Manhatten or on a subway when it hit, that might be a bit less enjoyable, but unbeknownst to New Yorkers and Californians, the rest of the country works a whole lot differently than they do.

    steve

    --
    Oh, you're not stuck, you're just unable to let go of the onion rings.
  16. Ohms and amps by Anonymous Coward · · Score: 3, Insightful

    Most writeups of this event blather about "power flow" and "electricity sloshing". I wish someone would explain this in terms of resistance, impedance, and current.

    "Suddenly the impedance in Michagan dropped. With Ontario as a constant-current source, the current through Niagara increased ..."

    That'd make more sense, no?

  17. The deal with the frequency by Spamalamadingdong · · Score: 5, Informative
    Others have kind of poked at this, but they haven't really explained it for the neophyte. I've had some education in electrical power engineering, so I'll try to fill that gap.

    There are two things you need to keep in mind here. The first is that phase in AC systems performs much the same function as voltage in DC systems; just as power flows from higher voltage to lower voltage across a DC connection, power flows from leading phase to lagging phase along an AC connection. (This has to do with reactance; all power lines are inductive.) Counterintuitively, voltage helps move power but it mostly balances VARs (volt-amperes reactive); if you have a local low-voltage situation, you can connect a capacitor to add some VARs and the voltage will come up. This is part of why big inductive loads cause line voltage to dip.

    The second thing is that frequency variation is just a phase change over time. If the local frequency falls for a bit, it means that the local phase is moving behind the rest of the grid. This is what you would expect if some large load was added (or a generator lost) and more power had to come from elsewhere on the grid; the delta-phase across the interconnecting lines has to shift to allow more power to flow. What little energy buffering there is is mostly the rotational energy of generators and motors, so phase changes don't quite happen instantaneously.

    If you had a serious local power shortage leading to shutdown, under-frequency is exactly what you would expect. Generators trip off-line, and the phase of the local grid backs off to pull more power from outside. It would take a full second at 59 Hz to shift one cycle, so this can go on for a fair fraction of a second. If the phase change over a transmission line increases past 90 degrees it will have to trip off-line, and once the local grid is an island you can have just about any frequency that the system will try to operate at. It's my understanding that most generators trip off-line at more than a fractional Hz off 60, if for no other reason than that they aren't designed or certified to operate on a grid that's obviously malfunctioning and such a condition means trouble. Mechanical resonances at off-operating rotational speeds are another reason to shut down.

    Last, I suspect your conclusion is correct.

    1. Re:The deal with the frequency by EuropeanSwallow · · Score: 3, Informative
      Hello

      Just to correct you:
      1. Not all power lines are inductive. Underground cables (like NY underground high voltage power network) are mainly capacitive.
      2. Frequency variation has nothing to do with phase shifts. As it says, it has to do with the fact that, in a transient state of the network after a load change, the generators have to balance the new load (electrical torque the generator "sees") with a change on its mechanical torque (acting on the prime mover). This doesn't happen in all generators, only the ones equipped with a speed governor, which are said to "contribute to frequency regulation". This adjustment is not instantaneous, and causes oscilation in the frequency value, until all the generators stabilize in the new operating point. Should the load change be too big, it might trip some of the generators, either by absolute deviation from the 60Hz (generally around 1Hz), or by rate of frequency change.
  18. So long for a timeline? by kosibar · · Score: 4, Funny

    It took this long to get an official timeline? Hmm. Makes me think that maybe there's some of this going on.

  19. Re:Canada topic/icon? by quacking+duck · · Score: 3, Funny

    A USA flag preceded by a "!" symbol.

    As in "NOT USA" ;-)

  20. the twelve oclock effect by Jack+Schitt · · Score: 3, Funny

    i wonder what would happen if you called Time in New York right after the power came back on... "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP" Click

    --
    This message brought to you by Jack Schitt's Previously Shat Shit
  21. Power by Orne · · Score: 4, Insightful

    No. The Voltage component of Power is important when managing the health of the bulk power grid. For those that don't remember, Power is Volts Amps, and because current is directional, power is directional also. This is why people often use "power flow"...

    The impedence in line in a function of the amount of energy flowing through it; as current increases, capacitive losses increase, causing the voltages at the ends to drop. This is sometimes called surge impedence loading. Impedence across a power line is constantly changing, and it is easier to wrap both variables into Power.

    Next, the use of transformers makes amps by themselves meaningless. Power is near constant across a transformer, so High Amps Low Voltage can become Low Amps High Voltage. By talking about everything in the form of Power, then you can easily measure the transfer of energy between the various voltage levels of your system, which eases explaining the system.

    Finally, Power is an easily understood market concept. If I run a generator at a low voltage (13kV), and produce 10 Amps, I'm generating 130 kW. I pipe that through a large number of transformers & lines, and deliver it to a load running at 23kV, and maybe tomorrow I sell it to someone at 9kV. By keeping everyone running in Power notation, we can all agree that money is exchanged for work, and the proper energy is delivered and paid for.