Slashdot Mirror
US/Canada Power Outage Task Force Event Timeline
bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."
12:05:44 - 1:31:34 PM - Four Generator trips
2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio
3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio
3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio
4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan
4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania
4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York
4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio
4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection
4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.
4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out.
A taskforce to survive power outages.... Lets show the world that we are stronger than ice-cream.
n/t
The blame will be put onderegulation and lack of government oversight.
Hopefully this is something that they can actually use to learn from. Seems to me that they should put safeguards in between each one of these events. Because if just ONE of these events could have been stopped, it would have stopped the whole east coast from losing power.
5:08 PM - U.S./Canada Power Outage Task Force web server goes out.
The coolest voice ever.
12.04 - power on
14.11 - power off
Comment: Yes I realise the username 'fuckfuck101' makes me sound intelligent, no you cannot buy it from me.
Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K) getting nailed by the Blaster worm? The timing is right, and there are a number of indications thereof:
See:
this or
this or
this.
do() || do_not();
Mayor of NYC decided to publicly blame Canada for this before any facts surfaced.. while Lastman, the mayor of Toronto said something along the lines of "Do you expect the US to take blame for anything?", after *some* facts surfaced
Neither responses were politically acceptable, however the media coverage of the blame game seemed to evaporate as soon as it was clear that it wasn't Canada's fault.
I found that more than a little interesting.
A couple of days after the blackout, I was staying up late and unfortunatley listed to some late night radio. Two different kooks were speculating on the cause of the blackout. One was positive that the Federal government was testing out some "advanced" weapon and the other freak was convinced that the culprit was the power companies looking for an edge to justify raising rates to build more infrastructure.
.02$
just my
"We're can't say for certain what happened and when, but we can say with almost 90% certainty that when the power went out, people went without power. We think it might be related to some electrical do-hickey thingy that someone was supposed to be watching carefully in case it broke, but we're not confident enough to make that bold a statement... nor to claim that anything actually 'broke'.
"But what we *can* say is that we feel strongly that we feel something different should have happened. An appropriate amount of blame will be laid... oh yes... and we'll make the bad people pay."
Diplomacy is the art of saying, "Nice doggie!" until you can find a rock.
"it does paint a pretty good picture of what happened and when it happened."
4:10:50 - 4:11:57 PM
Technician 1: "Hey I wonder what this big red button does?" {Click}
Technician 2: "NOOOOOO!!!!"
Chaos reigns within.
Reflect, repent, and reboot.
Order shall return.
0x0D 0x0A
12:05:44 - 1:31:34 PM - energy.gov publishes timeline
2:02:00 - 4:50 PM - Business as usual. Interested parties are viewing the published timeline.
04:58:00PM - Existence of timeline is revealed to slashdot.org
05:00:03PM - energy.gov? what's that?
I wonder if it's bad when we slashdot effect gov't sites...
---- Move SIG...For great justice!
Oh yeah, these guys run a nUUclear powerplant that um.. is like a college landlord of mine. Cheap and dirty as possible.
Did anyone else notice a strobing effect in their fluorescent lighting in those 20-30 seconds before the full power outage? My understanding is that any sort of arc lamp (fluorescent, metal halide) will extinguish if the voltage sags beyond a certain point, so I doubt it could have been a voltage sag before the full blackout.
It almost seemed as if the power frequency itself had gone unstable...say from a nominal 60Hz to like 5Hz. Then again, with the modern electronic ballasts used today, who knows how they respond to a voltage sag. Maybe they strobe. Any one have any thoughts on this?
-Chris
--an unbreakable toy is useful for breaking other toys--
"A place I worked" went dark a few years ago in a similar event. A large generator's main breaker tripped, system went unstable and underfrequency, and UFR's (which are set in multiple stages) cleared the lines. Took 8 hours to recover the 2000 MW that were lost (light load).
It's all part of standard protection equipment doing its job. You see 59 Hz? Open the breakers! 58 Hz? Open more breakers! 0 Hertz? Uh oh, here come the guys in suits...
http://www.nrcan-rncan.gc.ca/media/documents/Blac
I think I will be fair and equitable and allow Slashdot to take out a Canadian website as well. Please be kind to Natural Resources Canada.
I'm all for it.
I have a propane camping stove, plenty of propane, and plenty of canned food around. The servers I'm responsible for will stay running, the data center has its own generators, but our office will be dead.
That means I'd get to sit at home and play Monopoly all day. If I get bored of that, there are about a hundred things I can do for fun that don't require a bit of electricity.
Shoot, it would probably even be beneficial to people in my neighborhood. I'll bet that the park down the street from me would be teeming with people outside, enjoying wholesome activities and human interaction.
Instead of neighbors walking around the block looking for code violations to report to the city, they'd probably be actually interacting with each other, maybe even solving their problems without running to a baby-sitting city government!
I'm sure there would be consequences. Analysts would talk about how many hundreds of trillions of dollars were lost, but in the end, we'd all go back to work, take care of the stuff that didn't get done, and we'd have had a good time while it lasted.
Now, if the power outtage also included me being somewhere like the island of Manhatten or on a subway when it hit, that might be a bit less enjoyable, but unbeknownst to New Yorkers and Californians, the rest of the country works a whole lot differently than they do.
steve
Oh, you're not stuck, you're just unable to let go of the onion rings.
Most writeups of this event blather about "power flow" and "electricity sloshing". I wish someone would explain this in terms of resistance, impedance, and current.
..."
"Suddenly the impedance in Michagan dropped. With Ontario as a constant-current source, the current through Niagara increased
That'd make more sense, no?
East coast switches to linux powered servers
All east coast power generating stations, from nuclear reactors to hydroelectric dams, have suddenly decided to switch to Linux. When asked for the reason behind this decision, the response was:
"Well, erm, it was because of bl*cough* erm, I mean just a financial decision from the boys upstairs, I don't know anything about it"
The USA topic/icon. Canada's a protectorate allowed to play at having its own foreign and defense policies, not a soverign country of its own.
There are two things you need to keep in mind here. The first is that phase in AC systems performs much the same function as voltage in DC systems; just as power flows from higher voltage to lower voltage across a DC connection, power flows from leading phase to lagging phase along an AC connection. (This has to do with reactance; all power lines are inductive.) Counterintuitively, voltage helps move power but it mostly balances VARs (volt-amperes reactive); if you have a local low-voltage situation, you can connect a capacitor to add some VARs and the voltage will come up. This is part of why big inductive loads cause line voltage to dip.
The second thing is that frequency variation is just a phase change over time. If the local frequency falls for a bit, it means that the local phase is moving behind the rest of the grid. This is what you would expect if some large load was added (or a generator lost) and more power had to come from elsewhere on the grid; the delta-phase across the interconnecting lines has to shift to allow more power to flow. What little energy buffering there is is mostly the rotational energy of generators and motors, so phase changes don't quite happen instantaneously.
If you had a serious local power shortage leading to shutdown, under-frequency is exactly what you would expect. Generators trip off-line, and the phase of the local grid backs off to pull more power from outside. It would take a full second at 59 Hz to shift one cycle, so this can go on for a fair fraction of a second. If the phase change over a transmission line increases past 90 degrees it will have to trip off-line, and once the local grid is an island you can have just about any frequency that the system will try to operate at. It's my understanding that most generators trip off-line at more than a fractional Hz off 60, if for no other reason than that they aren't designed or certified to operate on a grid that's obviously malfunctioning and such a condition means trouble. Mechanical resonances at off-operating rotational speeds are another reason to shut down.
Last, I suspect your conclusion is correct.
Scientists restrict study to entire physical universe; creationist
It took this long to get an official timeline? Hmm. Makes me think that maybe there's some of this going on.
Your forgetting that the Priminister of Canada was the first one along with one of his ministers to directly blame the US, that being New York(first) and Pennsylvania(second), NY blaming Ontario(third) for the blackout.
Tit for Tat.
Om, nomnomnom...
We had a flourescent tube that had been "burned out" for about 6 months suddenly come on about 2 minutes before we lost power here in Toronto.
While it is worrisome that any system at a nuke plant could be disabled by worm traffic (appalling breach of good network operating practices), the criticality of a backup radiation monitoring system is very, very low.
Scientists restrict study to entire physical universe; creationist
A USA flag preceded by a "!" symbol.
;-)
As in "NOT USA"
i wonder what would happen if you called Time in New York right after the power came back on... "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP" Click
This message brought to you by Jack Schitt's Previously Shat Shit
No. The Voltage component of Power is important when managing the health of the bulk power grid. For those that don't remember, Power is Volts Amps, and because current is directional, power is directional also. This is why people often use "power flow"...
The impedence in line in a function of the amount of energy flowing through it; as current increases, capacitive losses increase, causing the voltages at the ends to drop. This is sometimes called surge impedence loading. Impedence across a power line is constantly changing, and it is easier to wrap both variables into Power.
Next, the use of transformers makes amps by themselves meaningless. Power is near constant across a transformer, so High Amps Low Voltage can become Low Amps High Voltage. By talking about everything in the form of Power, then you can easily measure the transfer of energy between the various voltage levels of your system, which eases explaining the system.
Finally, Power is an easily understood market concept. If I run a generator at a low voltage (13kV), and produce 10 Amps, I'm generating 130 kW. I pipe that through a large number of transformers & lines, and deliver it to a load running at 23kV, and maybe tomorrow I sell it to someone at 9kV. By keeping everyone running in Power notation, we can all agree that money is exchanged for work, and the proper energy is delivered and paid for.
Did you notice that the effect stopped at the Quebec border? They also have a seperate and distinct power grid (in addition to a seperate and distinct culture) as a remnant of the repairs after the 98(?) Ice storm. They were quite pleased that the effect could not propogate through the DC buffers at the NY and Ontario borders
Even though I am sure he had nothing to do with it.
A friend of mine works at NASA's Glenn Research Center near Cleveland (about where this whole mess started). GRC is where NASA has a very large, supersonic wind tunnel (large as in can accommodate full size mock-ups of fighter planes). When they run it during peak hours (rarely), it costs upwards of a million dollars per hour and causes localized blackouts in Cleveland. So I will have to make some sort of comment along the lines of "You just HAD to push to big, green button, didn't you?"
I also know that it wasn't the tunnel I work with in Chicago; we can only cause localized brownouts from the transient currents when we start the motor (rated for 1.5MW, starting power-draw is several times that). On an unrelated note, the sound of the capacitors charging is very similar to the noise you would expect from an evil villain's death ray.
could it have been an electro magnetic pulse, like in that movie where they break into the vault in the vegas hotel. Only it went a little haywire and gave them more than a 20 second window. That was my first thought. Probably a little imaginative, but it's still always nice to hear someone explain why this isn't the case.
yes, yes, I know with that attitude we'd have to hear people explaining about why it's not aliens, and not due to sun spots and any other thing some wacko can think up. But at least my idea was in a recent movie, so that gives explaining it away some entertainment value.
Or are they not explaining this away for the same reason they haven't explained away the crash over pennsylvania NOT being caused by an ordered shot. I.E., we'd rather not lie to you so we'll just not answer that question.
Anyone missing some money following that blackout?
because I have been enjoined by this Holy Office to abandon the false opinion which maintains that the Sun is the centre
Is it my imagination or did we just give the bad guys a roadmap on how to do this on purpose? I understand the public 'needs to know' but on the other hand i need to be able to sleep at night! and while im sure this report will fade into memory as most things in our country seem to do, i'll bet someone else will remember it just fine.
Motors initially draw an excess of current as they start up (they would also do this if the mechanical load were suddenly increased). If the voltage drops so low as to not allow the motor actually get going, it would be "stuck" in the high current situation for quite some time. This is why you see the lights dim briefly in many places when motors like A/C compressors start up.
In 2-phase circuits, the transformer is center tapped on the secondary, and the primary is on just one phase of the three phase source. You get 120v between either leg and the center (ground/neutral), and 240v between the opposing legs. A phase difference would certainly lower the voltage, but that's not going to happen on a 2-phase circuit unless you have other reactive loads there.
But, it may have actually been a 208v circuit between 2 legs of a 3-phase circuit. Most modern A/C systems are rated to operate even on 208 volts, so this kind of thing is not uncommon. It's cheaper to supply 208v instead of 240v to comemrcial buildings which already have a need for 3-phase motors (larger motors and A/C systems) than it is to supply genuine 240v (because this would require making the 3 transformers center tapped, and running 7 wires (6 point star and neutral) rather than 4 (3 point star and neutral). If a leg changed phase relative to the others, it could certainly upset the voltage balance, and the A/C could get undervoltage or overvoltage.
A frequency change can also cause problems, especially with 3-phase motors. A sudden phase shift (a brief frequency change) can cause a motor to go out of sync and draw more current to get back in sync.
now we need to go OSS in diesel cars