US/Canada Power Outage Task Force Event Timeline

bofus writes "The U.S./Canada Power Outage Task Force issued the Aug. 14, 2003 Sequence of Events at noon today. While no conclusions are drawn at this point, it does paint a pretty good picture of what happened and when it happened."

Comments

text version (aka karma whoring)

[glassesmonkey]

12:05:44 - 1:31:34 PM - Four Generator trips

2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio

3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio

3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio

4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan

4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania

4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York

4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio

4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection

4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.

4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out.

Re:text version (aka karma whoring)

[K_J_Raine]

Ah, so the Canadians weren't responsible for the blackout... as the nice mayor of New York originally contended.
Think he must have watched a bit too much South Park!

Re:text version (aka karma whoring)

[connsmythe96]

You left out:
Friday September 12, @03:58PM: Server goes down due to unusually high traffic.

Re:text version (aka karma whoring)

[cavemanf16]

9/12/2003 5:05:04 PM EST - 9/12/2003 9:03:04 PM server hosting timeline seperates from rest of internet.

Re:text version (aka karma whoring)

[Dr.Zong]

Yes. Mr. Bloomberg was very quick to point fingers, but it was fairly certain within 12 hours - exactly what happened where first, this is just a nail in the coffin.

It's nice to know that people in political power down there seem to think we're the devil or something. Makes me feel special :-)

Re:text version (aka karma whoring)

[clckwrkMalChick]

4:11:57 PM - 9:00:00AM - Entire city of Manhattan gets drunk.

Re:text version (aka karma whoring)

[Anonymous Coward]

8:58 PM - 4:13 AM - U.S./Canada conception rate soars to all time high.

Re:text version (aka karma whoring)

[Anonymous Coward]

You left out:

12:05:44 - 1:30:02 PM - BSOD

Re:text version (aka karma whoring)

[PD]

It's not just us. The English used to have a saying: "Poor Canada: so far from God, so close to the United States."

Re:text version (aka karma whoring)

[Frymaster]

just be careful, my friend! there's already been a war for oil this term...

let's hope there isn't one for electricity too!

Re:text version (aka karma whoring)

[Anonymous Coward]

9/12/2003 5:10:05 PM EST all the posts about the server getting /.ed except the first one are modded redundant.

Re:text version (aka karma whoring)

[Anonymous Coward]

In all fairness, Canadian officials were pointing fingers before all the facts were known, and doing so publically. I remember statements from Canadian officials regarding a US Nuclear Plant being to blame.

So, yea, both sides were acting stupidly.

Disclaimer: I live North of that long undefended border. And damn, what a difference a few degrees on the map can mean..

Re:text version (aka karma whoring)

[Anonymous Coward]

We'll still blame them anyway. =P

Re:text version (aka karma whoring)

[Anonymous Coward]

I wonder, if it had occurred in Canada, if we'd now be discussing the need to secure our powergrid from our friendly but technologically backwards Nothern neighbors and their retrofitted igloos. Probably... Gotta luv politicians....

Re:text version (aka karma whoring)

[Anonymous Coward]

I think his words were something like "We 'know' that the problem originated in Canada there is no question about that".

The Mayor probably doesn't even know what fucking day of the week it is.

Re:text version (aka karma whoring)

[Anonymous Coward]

At least Canada took Quebec by force. The US actually paid for Louisianna

Re:text version (aka karma whoring)

[yeti-graf]

The tragedy of Canada is that they had the opportunity to have British culture, French cuisine, and American technology. Instead they ended up with American culture, British cuisine, and French technology.

Re:text version (aka karma whoring)

[Jeremiah Cornelius]

The tragedy of Canada is that they had the opportunity to have British culture, French cuisine, and American technology. Instead they ended up with American culture, British cuisine, and French technology.

Ahhh...

I always heard they could have had French culture, British law and American economy. I guess the key being Government instead of Technology. French technology includes Dassault - who make the core avionics software in every modern military aircraft , including U.S. They also have a decent home-grown telecommunications satellite network - which the U.S. depended on sub-leasing to meet the command-and-control bandwidth requirements during Guerre d'Gulf Deux.

Now- as for French Government...

Re:text version (aka karma whoring)

[dadragon]

You probably would, but we wouldn't mind. That would mean that California and Washington would stop leeching power from BC.

Re:text version (aka karma whoring)

[GoofyBoy]

I would much rather have French women, British humour and American economy.

Re:text version (aka karma whoring)

[swschrad]

sure looks like a lot of power companies didn't pay their own bills and got turned off, doesn't it.

beware the little guy in the old truck with a key to the switches' padlocks

Re:text version (aka karma whoring)

[david614]

I believe the saying was actually by the Spanish referring to Mexico: "Poor Mexico: so far from God, so close to the United States."

Re:text version (aka karma whoring)

[letxa2000]

That saying refers to Mexico, not Canada. "Tan lejos de Dios, tan cerca a los Estados Unidos."

Of course, it could apply to Canada anyway, but the original expression was in Spanish and applied to Mexico.

Re:text version (aka karma whoring)

[Anonymous Coward]

At least the Prime Minister was right on that the power outage caused by the US side, something Blameburg FAILED to describe. I am not going to travel to New York until this moronic governor start apologizing.

Re:text version (aka karma whoring)

[Anonymous Coward]

Of course the chance of being right was 50%

Re:text version (aka karma whoring)

[Anonymous Coward]

English humour? Never saw that. Canadian humour on the other hand is great. Something like 20-30% of U.S. popular "funny actors" are canadians, like Mike Myers, (late) Jonh Candy, Matthew Perry , Dan Aykroyd , Jim Carrey , Michael J. Fox , Leslie Nielsen ... or the most funny of all, William Shatner and Jean Chretien!

http://www.canadiancelebs.com/act.html

Re:text version (aka karma whoring)

[GoofyBoy]

Monty Python, BlackAdder, Douglas Adams just to name a few good ones.

Re:text version (aka karma whoring)

[Xad]

There's no electricity outages in France...

Re:text version (aka karma whoring)

[THE ROCK]

Maybe if they bathed a little more, they might put a little more strain on their power grid.

It's not the end of the world

[Brahmastra]

A taskforce to survive power outages.... Lets show the world that we are stronger than ice-cream.

Re:It's not the end of the world

[edrugtrader]

its not funny to bite other comedians... namely lewis black.

get a life.

Re:It's not the end of the world

[metalligoth]

Actually, I was stuck in downtown Detroit when the outage hit, with no gas in my tank. I was just about to fill up and then *BAM!* gridlock hit. Me having to spend the night in downtown Detroit isn't the end of the world, no. However, people out there need respirators to live. Hospitals need power to save lives, and with no power for the gas pumps, a mutil-day outage is a huge deal, unless you enjoy having 3rd world medical care.

Oh, and let's not forget all the raw sewage they had to pump into Lake Erie due to the lack of power so it didn't start backing up into people's homes. It's good for the ecosystem, really! Oh, wait, it ended up backing up into peoples homes anyways.

I don't think people were crying over spilt milk or melted ice cream. They were pissed because they couldn't buy food, go to the bank, and then on top of it all they had to clean other people's excrement out of their basements.

Re:It's not the end of the world

[dosius]

Well, hospitals at least will have backup power (generators), if they're reputable, wouldn't they?

-uso.

Re:It's not the end of the world

[metalligoth]

Yes, but once again, if a power outage lasts for a while and there is no way to pump fuel due to a the lack of power for the pumps, the generators will fail eventually.

Re:It's not the end of the world

[KillerLoop]

Use some of generated power to drive the fuel pumps?

I CAN'T READ IT. MY SCREEN IS ALL BLACK.

[Anonymous Coward]

n/t

Re:I CAN'T READ IT. MY SCREEN IS ALL BLACK.

[Anonymous Coward]

you insensitive clod... I don't even have a screen!

Re:I CAN'T READ IT. MY SCREEN IS ALL BLACK.

[canfirman]

Try plugging it in.

Oh, wait...

The blame game

[trolman]

The really bizarre part that has not been explained is that the frequency deviated. I saw frequency deviation alarms on my equipment (ups and pdus) on the data center floor in NY, NJ, and CT and this is the first time in my 20 years that this has happened. A glaring ommission is is that PJM stayed up as an entity. Kudos to PJM.

The blame will be put onderegulation and lack of government oversight.

Re:The blame game

[twofidyKidd]

Well, what exactly causes frequency deviation? I'm not terribly familiar with it, and I can only imagine that there are a few things associated with it?

Re:The blame game

[GigsVT]

I'm not exactly sure what could cause it on a large scale. If a generator goes off sync, it's basically going to be a dead short wherever the voltages don't sync with the line voltage.

The only thing I can think of is if a generation station had all it's transmission lines tripped off, and was basically an island in the grid. Then if they somehow lost speed control momentarily....

I don't know, it seems like a very unlikely situation.

Maybe his frequency monitoring equipment is very sensitive to deviations, and when the plants were resyncing to rejoin the grid, they drifted their frequency for a while to get into phase.

Re:The blame game

[morcheeba]

If they're using AC generators (which I suspect provides most of the power - they're used in coal, nuclear, and hydro plants, but not solar, and not newer high-voltage DC transmission lines), then the frequency output is related to the spin of the shaft. Two things control the speed of the shaft - power in (water pressure) and power out (load demand from the grid), and a control system tries to keep the frequency a constant 60 Hz.

The trick is that the control system can only react so fast - suddenly disconnect an entire town, and the load drops, causing the power in to spin the generator too fast. If the control system overcorrects, then you'll get too low of a frequency. If a far-away generator drops out and you've got to supply more current to your local region, then the demand has gone up, slowing the frequency.

If you've been around generators, you can hear this exact phenonemoa - if the load changes suddenly, the motor will hunker down a little and then catch back up to normal speed. Usually a flywheel can damp out extremely short transients, but it would be prohibitively big if it were sized to handle transients as large as the control system (throttle) will allow.

Re:The blame game

[Anonymous Coward]

The frequency changed as generators tried to cope with overloads. In general, the only way to increase a generator's power output is to make it rotate faster- i.e. increase the frequency. Normally this happens on a very slow scale (generally variations of +/- 200ppm excursions over the course of day, with an average of 10pmm or so during the same period to keep clocks accurate). Obviously this is somewhat coordinated among different generation plants, since they need to be more or less in sync.

Anyway, severe load changes are likely to cause severe frequency changes, severe being +/- 10000 ppm or more. Naturally, this isn't helpful, and weird low-frequency feedback effects can happen as generators in different areas try to react to each other, and to the radically changing loads.

Re:The blame game

[4of12]

what exactly causes frequency deviation? I'm not terribly familiar with it

Glad you asked.

It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines.

This frequency is related to the 60 Hz rate of revolution maintained by most generators.

Occassionally, something will happen to upset this steady rate of revolution of the generators.

In particular, Homer, a donut that falls into the space between the stator and armature will cause this problem.

Lubrication from jelly-filling in the donut can help stave off the inevitable disaster of frequency deviation, but it is simply an old-wive's tale to believe that pouring coffee into the space will ungunk the works in time.

Re:The blame game

[alienw]

That was most likely a sign of power grid problems. Fluctuations in frequency are caused when generators get out of sync with each other and the grid. That's pretty much what caused those lines to trip and the generators to disconnect. When you have en masse transmission line tripping, you will see power spikes and frequency fluctuation.

The important thing in the investigation is the root cause of it and the reason why one problem caused a domino effect.

Re:The blame game

[Anonymous Coward]

thx a lot man for that post.

Re:The blame game

[sacherjj]

My guess is that the dropping of load in such a short timeframe allowed the generators to spin up faster than the regulating systems reduced power to the generator. This would drive the frequency on the line up. Then, very quickly, the generator safeguards realize that the load of power they are generating no longer has anywhere to go and does and emergency shutdown to keep from entirely destroying the generator.

Re:The blame game

[PD]

You can try this for yourself. Get a little bitty electric motor. DC will work fine. An electric is basically the same as a generator, and if you turn the motor, it will generate an electric current.

Spin the thing by hand and see how easy it is to turn. Pretty easy. Now, short out the connections to the motor and you will find that it's considerably harder to turn it. There's some resistance to the turning there.

When you shorted out the leads on the motor (which is operating as a generator), you've increased the load on the generator from nothing to something very large. And, as that load increases, the generator gets harder to turn.

Since the frequency of AC power coming out of a generator is set by the speed at which the generator rotates, it's simple to see that if the load makes the generator harder to turn, it will slow down because of that load. That makes the frequency of the AC coming out go down.

Re:The blame game

[Orne]

It's not that bizarre if you think about it... the bulk power frequency is actually one big juggling act between all of the generators that are synchronized on the system...

The Eastern Interconnection (everything in North America east of the Rockies and north of Texas) is tied together at many stations, such that there are many parallel paths to deliver energy to a customer load, providing an excellent level of stability. Simply put, the frequency is the prime measure of the balance between energy production and consumption. Energy generation is not a smooth process, it spikes as fuel is delivered and burned. If enough generators are synchronized with one another, they can automatically cover for each other's dips, and thus the frequency stays balanced.

Now, when the system split, imagine you had all of the generation on the west side, and all the load on the east side. For those of us in PA, we saw a huge loss of load, and the frequency shoots up. For those on the wrong side of the blackout, you suddenly lost your generation source, and your frequency drops.

Transmission equipment is easily damaged at low frequencies, so many are equipped with underfrequency relays that open breakers to protect themselves. What happened is that lines tripped and load sheds, forming smaller and smaller zones, until there were only small pockets of load and generators remaining (see the notes on western NY). Without the rest of the interconnection to syncronize with, your local generator was trying to maintain the frequency by itself as best it could, and was probably all over the map due to uneven fuel burn. Then, a few minutes later, you might have auto-reclosing of breakers (try-backs). If a line trips, some are programmed to auto-reclose, which, in an event like this, can suddenly add thousands of MW of load to an already stressed system, pulling the frequency down even more until everything is black.

Re:The blame game

[Vihai]

No, it doesn't work this way.

Synchronous generators doesn't need much frequency regulation. They're more like a series of gears, all connected together.

If you don't apply power, the generator rotates freely at 50/60 Hz and appears as a (nearly) inductive load.

If you provide mechanical power, the generator just sucks power and provides power to the grid in form of in-phase current.

If you apply a load, the generator (now a motor) still spins at 50/60 Hz, but sucks energy from the grid.

The only thing that changes is the phase of the current, no (little) voltage changes and no frequency changes.

The frequency regulation problem is something that is accomplished in large scale, a signle generator could not modify the frequency of the whole grid.

Re:The blame game

[Andrewkov]

One word: Aliens.

Re:The blame game

[Dr Caleb]

Tit for Tat.

Sometimes our elected officials act like school children. "I'm telling MOMMYYYYY"!!! I'm glad they aren't representative of those of us here on slash. . . oh . . .wait.

Re:The blame game

[Anonymous Coward]

He didn't forget -- Canadians all ready new he was an idiot and we've just learned to turn him out.

Idiocy is relative however, and Chretien is is sounding enlightened compared to ... well, I suppose it is rude for us to remind you. It's not like you REALLY elected him or anything.

Re: The Blame Game

[shoemakc]

"It's a highly technical term that refers to the frequency of the AC sine wave of electric power that is delivered over your lines."

Wait a minute, could that be the problem? I hear in Canada they use cosine waves. ;-)

-Chris

Re:The blame game

[Pieroxy]

Except that they were right where the US were wrong. Both reactions are childish, but only one is justified...

Re: The Blame Game

[crmartin]

You mean to say it's all just a phase?

Re:The blame game

[Tony-A]

The trick is that the control system can only react so fast - suddenly disconnect an entire town, and the load drops, causing the power in to spin the generator too fast. If the control system overcorrects, then you'll get too low of a frequency. If a far-away generator drops out and you've got to supply more current to your local region, then the demand has gone up, slowing the frequency.

Huh? The only control system has to do with needing to be pretty much in phase when you connect to the grid with big generators. After connection, anything that wants to go a bit faster has to supply power to all of the grid. Anything that wants to go slower isn't applying any power to the grid. In normal operation all the generators are in locked phase with each other. Short of something massive which gets something out of phase the system is completely self-regulating. I'm not sure what happens if something bit slips, but I suspect that the voltage doubles and the generators try to melt the transmission lines and facilities. I would assume they have stuff in place to disconnect fast in such a case.

Re:The blame game

[Mashiki]

Now you get into the fun stuff you see, this is called "politics".

I'm a real fan of wondering "what if", in this case "what if" Jean and idiot boy(that would be the minister), hadn't opened their mouths and blamed the US first. Would NY having been first target by Canada for the cause have reacted quite so in blaming Ontario? Maybe, maybe not.

But, with less then friendly relations between Canada and the US well I'm not surprised that the first reaction out of Ottawa...I mean from our Priminister sitting in Quebec with Hydro at his cottage while Ontario was without power and not saying a word except "blame the US".

Re:The blame game

[Mashiki]

Neither were justified, and all could be considered childish. Neither knew what was going on, and neither country still really knows what "exactly happened"; as it stands the statements were tit for tat. Or politics, nothing more nothing less.

Re: The Blame Game

[antiMStroll]

I hear in Canada they use cosine waves. ;-)

Only in Quebec. It's required to power the horizontal traffic lights.

Re:The blame game

[Anonymous Coward]

Priminister of Canada was the first one along with one of his ministers to directly blame the US

The blame game is so much better to play when you are right though... :)

Re:The blame game

[drinkypoo]

They don't use AC generators in solar plants? I can see why they wouldn't if they're using PV cells (since PV sells output DC) but there are a number of solar plants which reflect sunlight at a boiler. Is the turbine not connected to an AC generator?

Re:The blame game

[Alien Being]

Notice that New England (almost as good as the old one), true to form, separated and remained intact. I'm not sure which politico to thank for putting the kaibosh on merging grids with NY.

Re:The blame game

[Anonymous Coward]

The Canadian reaction wasn't justified either. It was a wildass guess, not based on fact.

Re:The blame game

[Anonymous Coward]

What's tat, where do you get it, and how do you trade it for the other thing?

Re:The blame game

[uncoveror]

It wasn't the Canadians or the U.S. The Martians turned out the power to hide their invasion! Read more.

Re:The blame game

[Mashiki]

A good point mate, 90% of the problem around the great lake triangle is the hodge podge of 60 year old technology. A real problem with the grid to say the least, I can't remember if I read or I heard from one of my friends who works as a lineman down in the US that when NY tripped they were cycling at 62Hz or 63Hz, the grid was supposed to auto-disconnect at 60.3Hz. I do remember reading that it was Ontario that was pushing the phase higher into NY tho, something to do with the grid becoming highly unstable(not surpising with deregulation here). I'm guessing that the auto-disconnect was a delay of some kind or hadn't been upgraded or some other crap.

Going to be very interesting to see how this all plays out in the end, all around the triangle tho there is alot of blame for the grid going down.

Re:The blame game

[doozer]

The easiest way to cause frequency variation: change the load on the generator.

In a very short period of time, (approximately zero) the draw on the generators
may have jumped. This very fast ramp up would have effectively stalled the generator
for a very short period of time. Then, the motor would start spinning again, and most
likely overshoot the proper rotational speed, and frequency would over what it should be.

Then, over time the system that regulates the output of the generators would slowly slew
the frequency back to what it's supposed to be.

Re:The blame game

[Anonymous Coward]

Tit for Tat.

Well, in that case, Tat

Re:The blame game

[madbastd]

The really bizarre part that has not been explained is that the frequency deviated.

I'll try for an even simpler examplation of this one.

If you suddenly increase the amount of power drawn from a generator, it will try to slow down. Usually, the other generators in the system will stop this, by generating more power, and nothing much will happen. On the other hand, if the generator does slow down (by pulling the whole grid with it), then the grid frequency will drop. The opposite will happen if you suddenly decrease the amount of power drawn from a generator.

That sort of thing was happening as the grid shook itself to bits, and the frequency would have been wavering all over the place.

Future Prevention

[LamerX]

Hopefully this is something that they can actually use to learn from. Seems to me that they should put safeguards in between each one of these events. Because if just ONE of these events could have been stopped, it would have stopped the whole east coast from losing power.

Re:Future Prevention

[the_2nd_coming]

here is the solution:

Build more power plants!!!!

Re:Future Prevention

[CoffeeCrusader]

yeah, but that would have meant, that the power providers had to provide money too. All of this could have been prevented with a bit more modern equipment. And modern equipment is expensive. In a deregulated market the power companies don't have the money to buy it. (well, usually) And to refer to a point made a bit further up: yes, this stuff matters, because there are people who care about what the masters of the power are doing. Because i like my screen to have another color than a forced black. Thus I like power. And I want to know why people cut me off of my power.

Re:Future Prevention

[Aczlan]

Point of information: The whole east coast DID NOT lose power. Only the northeastern united states did

Re:Future Prevention

[Anonymous Coward]

This was classic Domino Effect...

You _want_ the alternate paths to trip instead of dealing with melting transmission lines. Circuit breakers are better than pennies in your fuse box.

The (transmission) system worked as designed. The east coast needs more sooty power plants near the load.

IANAPE

Re:Future Prevention

[Detritus]

The trick is to build the power plants in the same area that the load is located in. This reduces the amount of power that must be imported or exported from any given area. Unfortunately, some people and politicians prefer to have the generating capacity, and its associated pollution, somewhere else (see NIMBY).

Re:Future Prevention

[AKnightCowboy]

The trick is to build the power plants in the same area that the load is located in. This reduces the amount of power that must be imported or exported from any given area.

No, the trick is to USE the power plants in the same area that the load is located in. First Energy has at least 2 nuclear power plants in Northern Ohio and one of them was shut down. Energy companies are finding it cheaper to buy electricity on the open market instead of generating their own. Yes kids, de-regulation was a horrible mistake when you do it to public utility monopolies.

Re:Future Prevention

[Anonymous Coward]

see NIMBY

For those unfamiliar, Nimby was Gumby's uglier, retardeder little brother.

Re:Future Prevention

[afidel]

You are right about deregulation sucking, but the reason the David Besse plant is shut down is that it is bar none the worst run plant in the country. They had a FOOTBALL size hole in the reactor containment unit. Furthermore First Energy is losing a LOT of money having that plant down, check their most recent SEC report. The problem is that most of us within the 50 mile death zone do NOT want to see it restarted unless there is a wholesale change of management and lots of government oversight. Ralph Nadar says that it should be closed permenantly, I disagree but I think a lot has to change before I am willing to let them operate a nuclear reactor near me.

Re:Future Prevention

[dirc]

Energy companies are finding it cheaper to buy electricity on the open market instead of generating their own.

If a supplier finds it cheaper to buy from someone else rather than produce his own, it indicates either: (a) the supplier is a less efficient producer. Someone else is able to do it more cheaply., or (b) the cost of building new capacity is so high that the cash flow from the new capacity does not justify the cost to build it.

If the problem is (a), then as a consumer, you want the supplier to outsource the work to someone else, because you get a cheaper product.

If the problem is (b) and the demand for the product is increasing in the long run, then as a consumer you will likely start to face spot shortages and price spikes caused by those shortages, since the outside supplier will eventually have less and less excess capacity to sell.

California has had experience with (b) in two markets, the electric power market and the gasoline market. In the former, the power outages were caused by high demand and insufficient generation capacity (only part of which was due to attempts to manipulate the spot price of energy). The major culprit was the length of time it took to get State regulatory approval to build new power plants. When Gov. Davis finally realized there was a problem, he relaxed the approval process and dozens of new plants were built.

The gasoline market suffers from regulatory restraint as well. Since 1985, not a single new oil refinery has been built in CA. Ten have closed. (See this for info. Now, refineries are running at over 90% capacity, rather than the 65% capacity they ran at in the mid-80s. Every time a refinery has a problem, the supply is out-stripped by demand and prices spike. The refiners, needless to say, love this. The are making huge profits because the government has imposed an almost complete barrier to new competition. (Presumably there is some price of gas at which it would be profitable to run the regulatory gauntlet.) The situation will get worse, since it is expected that in the new 5 years about 10% of CA refining capacity will be closed down.

As the two examples suggest, regulation of the market between suppliers and consumers is not the issue. The issue is government regulation by of the suppliers which restricts capacity. The government has some legitimate reasons for those regulations (such as protecting the environment), but their citizens must accept the consequences of growing demand and static supply. Those consequences will include outages, shortages and higher prices.

Re:Future Prevention

[the_2nd_coming]

well, we citizens do not accept this crap and the unfortunate thing is that the eco-jerks who made this policy crap happen are to fucking loud to talk over.

Re:Future Prevention

[cbreaker]

Agreed.

I'm all for the environment protection crap, but at the same time - we're here now and we need power and gas to live in this world.

We can't go back to hunting and gathering as a means of survival. Our survival requires the environment to be damaged. So let us damage it - not wrecklessly mind you (like dumping waste into a river) - but let us do what we need to do to live in this world, and hope that something more "clean" comes along to help in the future.

Personally, I'm a big fan of Nuclear power. Sure, we don't have a good method of disposing the waste, but burying it isn't a bad option and we have plenty of land for it. Coal and other fossil feul plants damage the air supply; much more threatening then some deeply buried radioactive if you ask me. At least we can decide what to do with it, whereas fossil feuls dump waste directly into the air.

Re:Future Prevention

[DerekLyons]

Hopefully this is something that they can actually use to learn from. Seems to me that they should put safeguards in between each one of these events. Because if just ONE of these events could have been stopped, it would have stopped the whole east coast from losing power.

Maybe. Probably not.

Electrical lines under load can be thought of as water pipes under pressure. If you gag a relief valve shut to prevent one portion from going under, the pressure is still there and it's going to go somewhere, and probably cause trouble when it gets there to boot. Contrairiwise, if you open a valve to let pressure out, you are going to change pressures and flows in the remainder of this system, possibly causing by your local protective action global problems, or local problems elsewhere.

It's not going to be a trivial fix, as the problems are far deeper than simply upgrading some equipment in few places.

Re:Future Prevention

[cpeterso]

I live in California. PG&E has plans to build two new power plants near central California (i.e. the boonies). The local Sierra Club is having a hissyfit because the power plants would be "near" some protected park lands. So apparently, the Sierra Club does not want power plants build in these three places: protected park lands, the boonies, or (presumably) near cities. So where would they like their new power plants to be built?

Hey they forgot...

[Cytlid]

... 5pm EST, Sept 12, 2003, the website goes down for some reason...

Re:Hey they forgot...

[tomhudson]

6pm EST, Sept 12, 2003, after repeated nmappings and traceroutes (as curious /.ers try to find out why their site melted, what they were running, etc.), Department of Homeland Security claims that site is under cyber-attack.

The last entry:

[Faust7]

5:08 PM - U.S./Canada Power Outage Task Force web server goes out.

Re:The last entry:

[LinuxTek]

Re:text version (aka karma whoring) (Score:5, Funny)
by connsmythe96 on Friday September 12, @04:05PM (#6947442)

You left out:
Friday September 12, @03:58PM: Server goes down due to unusually high traffic.

The last entry: (Score:5, Funny)
by Faust7 on Friday September 12, @04:05PM (#6947445)

5:08 PM - U.S./Canada Power Outage Task Force web server goes out.

And two equal jokes at the exact same time, and with the same score... wow... first redundant stories, now reduntant posts...

this is becomming reduntant

Timeline

[Prince Cyph0r]

"12:05:44 - 1:31:34 PM - Four Generator trips

2:02:00 - 2:02:00 PM - Transmission line disconnects in southwestern Ohio

3:05:41 - 3:41:33 PM - Transmission lines disconnect between eastern Ohio and northern Ohio

3:45:33 - 4:08:58 PM - Remaining transmission lines disconnect from eastern into northern Ohio

4:08:58 - 4:10:27 PM - Transmission lines into northwestern Ohio disconnect, and generation trips in central Michigan

4:10:00 - 4:10:38 PM - Transmission lines disconnect across Michigan and northern Ohio, generation trips off line in northern Michigan and northern Ohio, and northern Ohio separates from Pennsylvania

4:10:40 - 4:10:44 PM - Four transmission lines disconnect between Pennsylvania and New York

4:10:41 - 4:10:41 PM - Transmission line disconnects and generation trips in northern Ohio

4:10:42 - 4:10:45 PM - Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection

4:10:46 - 4:10:55 PM - New York splits east-to-west. New England (except Southwestern Connecticut) and the Maritimes separate from New York and remain intact.

4:10:50 - 4:11:57 PM - Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out."

4:12:30 PM - ??? - Millions of random people from the northeast US yelling "OH SHIT!!!"

the timeline...

[fuckfuck101]

12.04 - power on
14.11 - power off

Obligatory Line

[Eberlin]

5:00:00 - Profit!!!

Re:Obligatory Line

[KentoNET]

Don't forget these:

5:24PM - In Soviet Russia, electricity uses up YOU

and

5:24PM - All your electrons are belong to us...

Do the Right Thing

[leishen]

So I wonder if this whole thing will finally give people who "just want it done" the concept that getting it done "right" is really worthwhile.

Except MS, of course. They'll make the way they do it the right way. Sorry, had to throw that in. I've been fighting with MS products all week and I'm kind of sick of them and glad the week is over.

MSBlaster.exe

[devphaeton]

Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K) getting nailed by the Blaster worm? The timing is right, and there are a number of indications thereof:
See:

this or
this or
this.

Re:MSBlaster.exe

[Anonymous Coward]

"the power grid's SCADA systems (which habitually run a stripped down Win2K) "

Isn't that TREASON ?!!!!

Re:MSBlaster.exe

[insecuritiez]

A lot of people suspect that. I personally think it's the best explanation. However, even if Blaster caused the outage and every "expert" at the plant knew it, it would NEVER be published that way. That would open up a whole new can of worms in the public eye. A security and publicity nightmare. No, if Blaster caused anything that issue will be quietly swept under the rug. Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

Re:MSBlaster.exe

[Anonymous Coward]

Does correlate with a slight lag ... check the graphs:
http://isc.sans.org/port_details.html?por t=135

Re:MSBlaster.exe

[00420]

The best part is they decided it wasn't the Blaster worm before they even started the investigation.

Re:MSBlaster.exe

[gregmac]

Has anyone followed up or concluded anything regarding the possibility of the power grid's SCADA systems (which habitually run a stripped down Win2K)

I've still never understood this. I think most systems are actually based on NT, but maybe they are migrating to 2k now. Either way, the fact that these automation systems are based on a system like windows is very strange to me. OPC (the protocol used to communicate between sensors and databases) is based on DDE (or OLE), which seems so incredibly strange to me.

I've been developing a linux-based SCADA system. I took a look at quite a few systems, and I just didn't feel comfortable running any of them for a number of reasons. Stability and security being two major issues. Another was cost - these are being deployed in small installations, mostly for remote monitoring, which wouldn't typically have a SCADA system due to the cost. Between a mixture of existing open source software, some nice hardware, and in-house development (mostly me), the system has cost us about $20k to develop, which is less than it would cost to licence most software per site.

Anyways, that was a bit OT, but the point is, very early on we decided that deploying on windows would be a pain. These are all remote installations, with no one on site that can service them. If something goes down, I want to be able to remotely fix it as much as possible. I just don't feel comfortable deploying a remote windows system and relying on it to stay running, not to mention the fact that people's health could be affected (water treatment). To me, windows is not the proper platform to be using for this situation.

Re:MSBlaster.exe

[Nucleon500]

Well, there isn't much evidence, but would there be? That's the type of thing that gets covered up. (Notice how the media is careful to call msblaster.exe blaster, stripping the ms?) Yeah, my guess is someone brought an infected laptop onto the SCADA net. What would happen?

The systems are designed to run without monitoring. However, without human intervention, the systems aren't very good at staying up in exceptional circumstances. My guess is a computer failure made the grid much more vulnerable, to the extent where something routine brought it down.

Something else to add: a while ago in Ohio, a nucler power plant had its control systems down for a while as a result of msblaster.exe. Thankfully, the plant was off anyway, but it shows that sysadmins are just as bad in power infrastructure as they are in the rest of the commercial sector.

Re:MSBlaster.exe

[MagikSlinger]

Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

Obligatory response: Woo-hoo! Linux all the way baby!

BTW, the EULA specifically prohibits NT and up from being used in "critical" situations where life and environmental damage are on the line. So it would be the utilities' fault, not Microsoft's.

Re:MSBlaster.exe

[bill_mcgonigle]

I had a story rejected about this last week.

It's in my journal.

The important bits are:
Win2K/XP
Commercial Internet used on these systems
- and now: -

I didn't write it at the time, but a major power generator in Akron, Ohio got bit in the ass by SQL.Slammer earlier this year. Now that we know this came out of Ohio, it's more significant. Follow the link from my story for more.

Re:MSBlaster.exe

[MagikSlinger]

Isn't that TREASON ?!!!!

You'd think so (used Ann Coulter's definition of the word). To be fair, the SCADA systems are not officially considered part of the safety or reliability system. If they were, they'd have to run a grown up operating system that is certified for use in power generation and distribution. Wasn't this QNX's niche?

Re:MSBlaster.exe

[Orne]

If MSBlaster got into the SCADA network, then more than just First Energy would have been hit by high network traffic... yet only First Energy reports network issues. Besides, "habitually" you use a Unix mainframe to communicate on the SCADA network, and and Win* machine to pull the data from the mainframe into a form you can view it.

I find it entirely more plausable that they were having network issues on their local networks that communicated with their mainframe, not the mainframe communicating with SCADA. (Occam's Razor) Due to the large number of alarms going on, I find it even more plausable that the human operators were simply overwhelmed by the amount of new data coming in.

Re:MSBlaster.exe

[jabber01]

Are you suggesting that a piece of software could actually have caused a consequence in the physical world?? That's crazy talk! Madness I tell you.

Re:MSBlaster.exe

[homer_ca]

SCADA does monitoring only, no control. It's possible that an outage of the SCADA systems could have contributed to the miscommunication and confusion in Ohio from 2PM-4PM, but the power grid is supposed to be able to run without SCADA.

Re:MSBlaster.exe

[GoofyBoy]

I can't believe this would happen. Unless the number of people were tiny and were convinced that it was in the public interest to cover it up.

Usually the "experts" are responsible professionals, perhaps engineers. They know they have a responsibility to the public good and let them know that systems out there (even if now their own systems are fixed) are vulnerable to this "flaw". It would be worth it to protect the X millions of dollars to the economy.

How hard would it be to say "Yes we got hit just like everyone else in the world. It was a mistake and unless the entire system changes it could happen again." annoymously to some reporter. Word would get out and things would/might change.

I just don't like the fact that you are implying that people who take their jobs and reponsiblities seriously would lie to cover their own butts.

Re:MSBlaster.exe

[Attaturk]

A lot of people suspect that. I personally think it's the best explanation. However, even if Blaster caused the outage and every "expert" at the plant knew it, it would NEVER be published that way. That would open up a whole new can of worms in the public eye. A security and publicity nightmare. No, if Blaster caused anything that issue will be quietly swept under the rug. Maybe Microsoft will suddenly not get a contract with the power generators anymore, but that's as far as we'll ever hear of it.

In case you missed it, Newsforge has a very interesting and relevant article.

A quote:

In that article, [Kevin] Poulsen offers a detailed description of how another Microsoft worm, Slammer, crashed two Unix-based control systems at the Davis-Besse nuclear power plant in Northern Ohio.

It may be Microsoft's fault but it's not just Microsoft's operating systems that suffer.

Re:MSBlaster.exe

[nizmogtr]

I thought power companies usually run QNX photon or some other OS instead of windows inorder to avoid the blaster scenario.

Re:MSBlaster.exe

[pi_rules]

OPC? On Linux? Will you share?

I don't have to work with OPC but I started up a venture of sorts once with a guy that knew the horrors of it and let me tell you I hated trying to design our idea in Windows. It just didn't fly. I got sick of COM objects talking to COM objects talking to COM objects. It just got really really bad. This was three years ago, so maybe now it wouldn't be such a problem for me but I just hated the code we had to write.

I'd just love to see a portable, open-sourced SOAP OPC bridge of sorts. Is this an in-house closed up development thing or are you free to speak about it? Can some of the source be released under an OSI friendly license? I'd love to get my fingers back into that pie, develop the snot out of a cool assed open-architecture system and then set to work consulting on implementing the thing.

Re:MSBlaster.exe

[gregmac]

OPC? On Linux? Will you share?

I didn't actually write any OPC code, though that would have been handy. What I wrote is basically a messaging server, that uses a fairly simple protocol to talk over TCP sockets. Clients hook in, and can provides inputs, outputs and variables (there's really not much distinction between them, except variables have the option of being stored in an SQL database, and inputs can only be set by the client that provided them). Clients can request any value at any time, or request to be notified of certain values.

I have a client written to gather input from my hardware (opto22), provide output, another that logs values to a database, one that graphs values to rrdtool, one that compares values against alarm setpoints, one that dispatches to people based on time-of-day when an alarm is activated, etc.

Writing an OPC bridge would not be too difficult, I don't think, depending on how hard it is to use the OPC protocol. I started on a modbus interface (never finished it since I don't need it right now), and have plans to make a Profibus interface.

The system is pretty simple, but at the same time it's pretty powerful. I started developing it in March, and have had the vast majority of it running for a couple months now. Since it's so modular, it's very easy to test incrementally, a lot of the design was based on a rapid development time.

The user interface is web-based, and it's designed from the start to be remotely configurable and viewable. Alarms, variables, settings, logging and graphing options, etc, can all be set from the GUI. Some of the system setup is also done from the GUI, but control programs (ie, the things that actually 'do' system control) have to be written and set up manually, although I'm sure that could be done through GUI as well, I just haven't bothered yet.

I've been too busy lately to put any effort into thinking about how to release it to open source, but it's quite possible I will. Now- most of the code I've written is in PHP, which I'm sure will turn a lot of people off. In reality, using PHP has allowed me to write solid code very rapidly, and running on linux on modern hardware, performance is not an issue at all. Python would be my next choice, but I didn't know it well enough and thought it would take to long to learn and develop the system at the same time. C is out mostly for the fact that it would take at least twice as long to write the same quality code (and I'm sure some people will aruge that point).

PuffinPLC has quite a bit of work done, and I really did consider using and contributing to their stuff, but last I checked, they had been going for couple years, and were still only predicting the first usable release would be in a year. I was looking for a SCADA system for a specific task - they're trying to build one that can replace big commercial packages.

Anyways, if there is enough interest, I would definately be willing to open source it. Likely there could be some performance boosts by rewriting some stuff, espessaily the server, in another language, but I'm going for a stable, working solution first.

Re:MSBlaster.exe

[babyrat]

SCADA stands for Supervisory Control and Data Acquisition. It is possible (not desirable, but possible) to lose all your 'Supervisory' Control for extended periods of time and have everything keep running fine in the 'controllers' themselves.

As a side note, OPC is based upon DCOM, not DDE and most SCADA systems (that run on MS Platforms) are now (and have been for quite some time) certified on Windows 2000.

Re:MSBlaster.exe

[Skapare]

It might not have even been necessary for the SCADA systems to be directly affected. If other computers had been affected in some way, and if there were shared networks involved (the current trend, so as to reduce costs), the overload in networks due to Blaster effects, could have delayed measurements or responses by other systems. A few lost packets can do things like cause TCP to delay recovery equally as long as the period of the problem (because it doubles the time interval between each retry).

So even if the control systems were running highly secure and reliable systems, simply deploying it over a load vulnerable network can make the whole system itself vulnerable to a denial of service attack (as Blaster effectively was). These are cases where total isolation and mandatory quality of service are required.

Re:MSBlaster.exe

[SagSaw]

BTW, the EULA specifically prohibits NT and up from being used in "critical" situations where life and environmental damage are on the line. So it would be the utilities' fault, not Microsoft's.

Nothing, of course, stops utilities from negotiating their own licence agreement with Microsoft. I'd imagine, though, that such a licence would compleatly absolve Microsoft of any bad things which might result from some form of software error/bug/malfunction.

Curious

[JediTrainer]

Anyone know the time that idiot Bloomberg tried to blame the whole thing on Canada?

Bloomberg

[oZZoZZ]

Mayor of NYC decided to publicly blame Canada for this before any facts surfaced.. while Lastman, the mayor of Toronto said something along the lines of "Do you expect the US to take blame for anything?", after *some* facts surfaced
Neither responses were politically acceptable, however the media coverage of the blame game seemed to evaporate as soon as it was clear that it wasn't Canada's fault.
I found that more than a little interesting.

Re:Bloomberg

[Otter]

Both are generally regarded as idiots, anyway, and there were plenty of more pressing practical problems to be dealt with.

Getting ticketed for sitting on a curb sucks, but at least it beats getting SARS...

Re:Bloomberg

[Anonymous Coward]

Blame Canada!

Blame Canada!
Shame on Canada!

The smut we must stop
The trash we must smash
Laughter and fun
must all be undone
We must blame them and cause a fuss
Before someone thinks of blaming us!

Matt and Trey -- America's Imported Wisdom

Re:Bloomberg

[notcreative]

There's no "U" or "S" in "BLAME."

Re:Bloomberg

[ergo98]

To be fair, before Bloomberg made his idiotic statement (which included the words "absolutely certain") various levels of the Canadian government had blamed it on a New York power station, and then a Pennsylvania nuclear power plant fire (information they got from the US Department of Defense, as a sidenote, but nonetheless it was irresponsible to repeat it verbatim so early on). I think a big difference though is that when I hear that it's a New York power plane or a nuclear station in Penn, most Canadians don't think "Those damn Americans!", or any "us versus them" nonsense. Instead it's a power plant going down, and why the hell is this grid so fragile? I suspect for Bloomberg and crew, though, it's nothing of the sort: Not only is it not "Ontario", it's "Canada" (which gives you an idea of the perception right there), but there's a definite slant of "Well that explains it right there!". This is par for the course for the various levels of government in New York state, though: Hillary Clinton has made countless nonsensical statements about how Canada is to blame for every fault in her little fantasy world.

Re:Bloomberg

[MagikSlinger]

Matt and Trey -- America's Imported Wisdom

Since when did Colarado become a foreign country?

Re:Bloomberg

[Anonymous Coward]

I relieved it wasn't in Canada. Since our idiot politicians couldn't keep their mouths shut when they didn't know WTF they were talking about either. If it had turned out it had begun in Canada (and it could have, our power grid has to withstand the constant assault of Canadian winters!) it just would have blown the whole issue wildly out of proportion ...

Re:Bloomberg

[canfirman]

Well, it didn't help with CNN reporting it was Canada's fault - without a retraction!

Besides, CNN was showing people in NYClining up for groceries, batteries, etc. Then, they showed Canada - people lining up at the beer store!!!!

Well...could this mean we have our priorities?

Re:Bloomberg

[Anonymous Coward]

Matt and Trey were from Canada

Re:Bloomberg

[GoofyBoy]

But there are the letters "M" and "E"!

Re:Bloomberg

[antiMStroll]

...various levels of the Canadian government had blamed it on a New York power station, and then a Pennsylvania nuclear power plant fire (information they got from the US Department of Defense, as a sidenote, but nonetheless it was irresponsible to repeat it verbatim so early on).

Was it blame, or did Canadian officials re-distribute the best information available, given to them by a disinterested authorative source, at the time of a major emergency? If the latter then not doing so would be irresponsible. On what basis did Bloomberg make his statements?

Re:Bloomberg

[Guy Harris]

Matt and Trey were from Canada

Your Canada includes Colorado and Texas? (If it includes Texas, do we at least get to blame you for Dubya?)

Re:Bloomberg

[Anonymous Coward]

Heh, folks were lining up at the gas stations, grocery stores, getting batteries, etc etc here in Toronto as well as lining up for beer.

Hey, it was an unexpected long weekend and we wanted to make sure we had the appropriate supplies in.

Re:Bloomberg

[Anonymous Coward]

We'll take Dubya if you'll do something with Jean Poutine

Re:Bloomberg

[Prune]

Mod parent up!

Re:Bloomberg

[linzeal]

Most americans don't even know any canadian states, hell I knew people in Washington that did not know about BC, heh.

Re:Bloomberg

[gfreeman]

Canada has states now?

Re:Bloomberg

[smallpaul]

Hillary Clinton has made countless nonsensical statements about how Canada is to blame for every fault in her little fantasy world.

What???

Re:Bloomberg

[TheMidget]

Provinces, you nitpick

Re:Bloomberg

[ergo98]

"or did Canadian officials re-distribute the best information available, given to them by a disinterested authorative source, at the time of a major emergency?"

The irresponsible part was that they didn't attribute their source, leading one to believe that they had independently verified these facts. i.e. If Joe tells you that the sky is blue, and Bob tells you that the sky is blue, there are two sources confirming that the sky is blue, yet what if Bob is merely repeating what Joe told him and isn't indicating this?

Re:Bloomberg

[Guy Harris]

We'll take Dubya if you'll do something with Jean Poutine

One fictitious president for another - sounds like a good trade to me....

Slashdotted?

[packethead]

Or maybe it's another power outage.

Late Night "Wacko" Talk Radio

[GSpot]

A couple of days after the blackout, I was staying up late and unfortunatley listed to some late night radio. Two different kooks were speculating on the cause of the blackout. One was positive that the Federal government was testing out some "advanced" weapon and the other freak was convinced that the culprit was the power companies looking for an edge to justify raising rates to build more infrastructure.

just my .02$

Re:Late Night "Wacko" Talk Radio

[Anonymous Coward]

The guy I heard blamed it on the Trilateral Commission. He also attached some significance to the fact that 4:11 in the Eastern Time Zone was 9:11 GMT.

Re:Late Night "Wacko" Talk Radio

[red floyd]

Except the US was on Daylight Savings Time. And 4:11 EDT is only 8:11 GMT.

Actually, of those three conspiracy theories, I'd pick #2 (jack up prices). Remember California!

Re:Late Night "Wacko" Talk Radio

[bill_mcgonigle]

the power companies looking for an edge to justify raising rates to build more infrastructure

nah, never

Re:Late Night "Wacko" Talk Radio

[base3]

Turns out the second freak was right, based on the pap the Pres and Secretary of Energy are spouting about ratepayers being "part of the solution" for a "modernized grid." I wouldn't be surprised if the blackout were engineered just for that purpose. Worked in CA.

WELCOME TO THE UNITED STATES.

[Anonymous Coward]

This is CNN

This is Fox News

From Department of Energy Secretary: +1, Patriotic

[Anonymous Coward]

"It won't happen again".

That was a reassuring comment in the stream of
idiotic statements emanating from The Stacked Deck

Thanks for nothing,
W00t

What they *are* confident of...

[Not_Wiggins]

"We're can't say for certain what happened and when, but we can say with almost 90% certainty that when the power went out, people went without power. We think it might be related to some electrical do-hickey thingy that someone was supposed to be watching carefully in case it broke, but we're not confident enough to make that bold a statement... nor to claim that anything actually 'broke'.

"But what we *can* say is that we feel strongly that we feel something different should have happened. An appropriate amount of blame will be laid... oh yes... and we'll make the bad people pay."

Re:What they *are* confident of...

[Nucleon500]

"We'd also like to add that while we have no idea what the cause is, we're certain it isn't terrorism, or even msblaster. Absolutely sure. It must have been corperate culture or something. This interview is over."

Stuff that matters?

[stratjakt]

This is stuff that matters.

You guys wanna talk about dual use? Tommy Chong is off to the slammer for 9 months for selling pipes. Not drugs, but pipes.

choice quote:
He admitted once having "a drug problem with marijuana" but said he beat it by redirecting his energy to salsa dancing.

Fuckin lame war on drugs. Yeah, Cheech and Chong are dangerous lunatics and definately need to be taking up cell space.

Re:Stuff that matters?

[xchino]

While it most certainly sucks Chong got busted, it has nothing to do with the technical nature of slashdot. That's something you find on fark.com. Also, if he wouldn't have geared his marketing gimmicks towards indicating these devices were to be used for marijuana, he wouldn't be in the mess he is in now. All it would have taken was a "For tobacco use only" disclaimer and he could have joined the hundreds of legal water pipe dealers.

Aside from that, the measure of the sentence is ridiculous, apart from the idiocy of anti marijuana laws in the first place.

I know what really happened...

[Stingr]

"it does paint a pretty good picture of what happened and when it happened."

4:10:50 - 4:11:57 PM
Technician 1: "Hey I wonder what this big red button does?" {Click}

Technician 2: "NOOOOOO!!!!"

Re:I know what really happened...

[sharkey]

Technician 2: "NOOOOOO!!!!"

Probably more like, "Darers go first!"

Canada To US

[Anonymous Coward]

Canada say to You Ess:

aLL yOUR pOWER aRE bELONG tO uS!!!

Imagine if...

[ajiva]

Imagine if someone *really* wanted to cause problems. They could attack a number of our power generators (say 10-15), cause them to go down. Now you have this nice cascading effect where all of the power plants in the US go down. We're left in the dark. *SHUDDERS*

Re:Imagine if...

[NerveGas]

I'm all for it.

I have a propane camping stove, plenty of propane, and plenty of canned food around. The servers I'm responsible for will stay running, the data center has its own generators, but our office will be dead.

That means I'd get to sit at home and play Monopoly all day. If I get bored of that, there are about a hundred things I can do for fun that don't require a bit of electricity.

Shoot, it would probably even be beneficial to people in my neighborhood. I'll bet that the park down the street from me would be teeming with people outside, enjoying wholesome activities and human interaction.

Instead of neighbors walking around the block looking for code violations to report to the city, they'd probably be actually interacting with each other, maybe even solving their problems without running to a baby-sitting city government!

I'm sure there would be consequences. Analysts would talk about how many hundreds of trillions of dollars were lost, but in the end, we'd all go back to work, take care of the stuff that didn't get done, and we'd have had a good time while it lasted.

Now, if the power outtage also included me being somewhere like the island of Manhatten or on a subway when it hit, that might be a bit less enjoyable, but unbeknownst to New Yorkers and Californians, the rest of the country works a whole lot differently than they do.

steve

Re:Imagine if...

[tomhudson]

They don't have to attack the generators (which is hard to do), just the transmission lines somewhere in the bush (a couple of shotguns aimed at the insulators will do fine).

Done near peak load, everything goes bye-bye again :-(

Re:Imagine if...

[Anonymous Coward]

Why go to that trouble? Just blow up a few transformer yards, and electrically isolate the plants from the grid. Same effect, less effort.

Re:Imagine if...

[temojen]

That was tried in the early 1980's on Vancouver Island by the Squamish Five

Re:Imagine if...

[CycleMan]

neighbors walking around the block looking for code violations to report to the city

Your city has its own Bugzilla?

Cool...

Re:Imagine if...

[cant_get_a_good_nick]

I agree with this man, 100%. All people should be as prepared as this man. Our country would be much better off.

Signed:
Hank Hill, seller of propane and propane accessories.

Re:Imagine if...

[Anonymous Coward]

If I get bored of that, there are about a hundred things I can do for fun that don't require a bit of electricity.

I only can think of one thing, and it surely isn't playing Monopoly...

Sorry ;-)

Re:Imagine if...

[kmilani2134]

Actually, here in NYC the lack of power actually did have a silver lining in that everybody was out talking to their neighbors. People don't really interact much anymore with all of their TV and computers and other entertainment available to them. So it was pretty amazing to see people getting to know their neighbors and having a shared experience.

When the power finally came on, one of the news stations interviewed a marriage counselor who said many of her clients had called her to tell her how the blackout resulted in an improved relationship with their spouse. This makes me wonder if the Northeast will have an increase in birth rates in 8 months.

The timeline maybe should have: May 2004 - Historic rise in birth rates in the Northeast

Re:Imagine if...

[Anonymous Coward]

I seriously doubt that the birth rates will be affected. This is long since been proven to be urban legend.

Re:Imagine if...

[linzeal]

Hey I'm in way northern California and have enough canned food, camping gear, and know how to survive any catastrophe.

Actual timeline

[Anonymous Coward]

1. Power companies build infrastructure
2. Government regulations increase
3. Power company stops building infrastructure
4. Power outtage

Future events
5. Regulations relaxed
6. Power grid infrastructure increased

Re:Actual timeline

[fpp]

Huh? In a deregulated market, the power companies STOP building infrastructure, because it costs them huge amounts of money that they no longer have. With increased competition comes cost cutting, and the first thing usually to go is the building of more infrastructure.

Re:Actual timeline

[garrulous]

PROFIT???

Re:Actual timeline

[Anonymous Coward]

What you miss is that the regulations, epa, environmental impact lawsuits, permiting process, getting right of way, etc... costs a lot of time and money and creates significant barriers to anyone wanting to install any power lines over long distances.

A large number of minor or major incremental regluatory costs add greatly to the final price to build it.

Derregulation only seperated the power generation from the transmission and those two from the energy trading.

The local governments get around this by using eminent domain to force unwilling sellers to sell their land as well as immunity from lawsuits.

Opposing a power line due to environmental reasons may help the environment, but will increase everyone's cost to live there. This especially hurts the poor people since the vast majority of their disposable income is spent for housing, electricity, phone, water, heat, cooling, and lots of thisgs tied to those costs.

BitTorrent link

[mskfisher]

I've started up a BitTorrent mirror of the PDF here:

http://www.mskf.org/BlackoutSummary.torrent

Re:BitTorrent link

[Anonymous Coward]

Ah, 0x0D, my mortal enemy, we meet again.

And I fear our path will cross once more before our lives have finished their course.

Timeline of today's slashdotting...

[Ikeya]

12:05:44 - 1:31:34 PM - energy.gov publishes timeline

2:02:00 - 4:50 PM - Business as usual. Interested parties are viewing the published timeline.

04:58:00PM - Existence of timeline is revealed to slashdot.org

05:00:03PM - energy.gov? what's that?

I wonder if it's bad when we slashdot effect gov't sites...

Re:Timeline of today's slashdotting...

[zurab]

In related news, attorney general John Ashcroft is expected to announce tonight that justice department has successfully identified and are in the process of bringing to justice yet another terrorist organization. Online community known as "slash/dot" has been widely recognized as an organization with differing views from current White House administration. Most recently members of this organization have been responsible for hacking and bringing down the U.S./Canada Power Outage Task Force website containing the report on timeline of events during recent blackout in Northeastern United States.

"This investigation and compromized information is crucial to the security of our power supply and and our country," - said justice department spokesperson - "we are currently working with the FBI and various local law enforcemnet officials to identify and bring to justice responsible members of the underground organization." Attorney general is expected to give out more details in his address.

The "slash/dot" organization has been identified in the past as a source of criticism towards America's entertainment companies, the U.S. PATRIOT Act, Digital Millenium Copyright Act, and other laws of the land. This un-American behavior coupled with anti-White House sentiment and now, recent terrorist actions holding the joint task force website hostage has landed "slash/dot" on government's terrorist organizations list.

"That's fine," - said one Florida resident who wished to remain anonymous - "everybody has the right to free speech but if you go against the government and are accused of terrorism, then, as they say - you do the crime, you do the time, right?"

Attorney general John Ashcroft is expected to speak during primetime tonight to reach wider audience. Some skeptics argue that this address and timing is related to justice department's push to pass PATRIOT ACT II that will give law enforcement even more tools to combat terrorism. But justice department has hinted in the past that anyone expressing skepticism on the battle against terrorism may be classified as a terrorist as well.

What really happened

[segment]

12:05:44 - 1:31:34 PM - script kiddiot downloads SoBig source code and opens up a jolt

2:02:00 - 2:02:00 PM - sciprt kiddiot gets frustrated cuz he cant gcc -o sobig sobig.c "Fuck how do I do this shit" drinks more jolt

3:05:41 - 3:41:33 PM -sends message to securityfocus.com `echo "3y3 j4m th3 k1ng 0f th3 w0rld"|mail -s help security-basics@securityfocus.com

3:45:33 - 4:08:58 PM - script kiddiot finds out he can't gcc a damn virus so he jumps on his leet Windows98 machine

4:08:58 - 4:10:27 PM - virus makes its way through insecure machines on the internet

4:10:00 - 4:10:38 PM - script kiddiot jerks off to jenna jameson while looking at news on television

4:10:40 - 4:10:44 PM - power goes *boom*

4:10:41 - 4:10:41 PM - script kiddiot wonders why he can no longer watch porn

4:10:42 - 4:10:45 PM - script kiddiot jumps off the bed gets dressed because mom is knocking on the door

4:10:46 - 4:10:55 PM - mom "I wonder what happened to the power"

4:10:50 - 4:11:57 PM - drinks another jolt

Re:What really happened

[Anonymous Coward]

script kiddiot jerks off to jenna jameson

Well, if you're gonna do it, that's as good of way as any!

First Energy the real problem

[glassesmonkey]

First Energy gets informal SEC request for data
Fri September 12, 2003 08:17 AM ET
WASHINGTON, Sept 12 (Reuters) - Ohio utility First Energy Corp. FE.N on Friday said the Securities and Exchange Commission had sent it a voluntary, informal request for data regarding its recent 2002 financial restatements. "First Energy intends to fully comply with the request and does not anticipate any adverse consequences," the Akron, Ohio, company said in an SEC filing.

The SEC request was made on Thursday, First Energy said, when the Ohio utility revised three regulatory filings, citing tens of millions of dollars in "typographical and minor computational errors."

The company's transmission lines have been cited as a possible source of the massive Aug. 14 blackouts.

Oh yeah, these guys run a nUUclear powerplant that um.. is like a college landlord of mine. Cheap and dirty as possible.

Re:First Energy the real problem

[red floyd]

No, no, no!

<RANT type="paranoid>
The power outage occurred after the SEC requested anything. The outage was First Energy's way of asking, "Are you sure you want to mess with us?"
</RANT>

Re:moron smoke&mirrors, deep voodoo & chic

[chiph]

Moderation: +1 Unintelligible Rant

New Task Force appointed...

[Anonymous Coward]

New Task Force appointed Monday, Sept. 15, 2003 to investigate web server outage beginning on Friday, Sept. 12, 2003.

Re:Details slowly emerge:

[Anonymous Coward]

No, the DEA targetted him because he had information that could have exonerated Tommy Chong.

Disturbances Before Outage Itself

[shoemakc]

Did anyone else notice a strobing effect in their fluorescent lighting in those 20-30 seconds before the full power outage? My understanding is that any sort of arc lamp (fluorescent, metal halide) will extinguish if the voltage sags beyond a certain point, so I doubt it could have been a voltage sag before the full blackout.

It almost seemed as if the power frequency itself had gone unstable...say from a nominal 60Hz to like 5Hz. Then again, with the modern electronic ballasts used today, who knows how they respond to a voltage sag. Maybe they strobe. Any one have any thoughts on this?

-Chris

Re:Disturbances Before Outage Itself

[Anonymous Coward]

I definitely saw my lamp on my desk still glowing while the other stuff was out. So there was a brown out.

Re:Disturbances Before Outage Itself

[Anonymous Coward]

Yes. In fact, the strobing effect happened multiple times, giving me enough warning to save and close what I was working on. I didn't shut down the first time because I didn't notice power fluctuations on the machine, and I assumed (!) that it was isolated to the branch that the lighting is on. It happened for the third time, and I was just reaching for the shutdown button when the grid fell over.

I also observed that all the fans we had running in the office were still turning, and the Capslock and Numlock LEDs were still lit, even though the machines all went down. Apparently there was enough voltage left on the line to spin the fan motors, and power those LEDs.

I didn't find out that it wasn't localized to my building for quite a while though, since 20 million people all flipped open their cell phones at the same time.

Re:Disturbances Before Outage Itself

[quacking duck]

I was at my home computer. Speakers started making odd noises, like it was suffering a power drop before coming back up (it makes a whoomph whenever I turn it on or off). This happened a couple times in the half-minute before the oddest thing (in my power outage experiences anyway) happened: first my computer shut off, but the monitor, speakers and DSL modem and router were still running (monitor wasn't receiving a signal anymore of course, but was nevertheless still "on" because the power light was still lit). Then the monitor switched off, speakers made a bottoming-out sound, and the modem and router lost power about the same time.

That suggested to me a voltage sag first, because the systems using the most power cut out first, while those using the least remained to the end; usually they all cut out at the same time if there's a power outage. All these components were connected to a surge protector.

Re:Disturbances Before Outage Itself

[SpaceLifeForm]

So, when are you getting your new computer?

Re:Disturbances Before Outage Itself

[Elwood P Dowd]

Yeah. See this post about frequency variation before the power outage.

Re:Disturbances Before Outage Itself

[gid]

I live an North Eastern Ohio. I had two CRT monitors on and plugged into normal power. Both of my computers are on a UPS. I specifically remember my monitors dimming and then comming back a few times, all the awhile my UPS is going nuts, supplying extra battery power to keep the voltage up. My lights remained on, but dimmed. Then a few seconds later, maybe 10-15, the power totally went out.

So of course I shut down my Windows machine as fast as I could, as this isn't a beastly UPS by any means, and two computers on it won't last long. I'm skilled in the art of windows keyboard shortcuts, so shuting down was no problem. As far as writing code goes, I probably save the file I'm working on once at least every 60 seconds out of habbit, ctrl-s is your friend. :) So then a few minutes later my UPS starts really going nuts, so I shutdown my linux box, by ctlr-alt-backspacing out of x, and then giving it the three finger salute. Anyway, enough rambling. :)

Kind of odd, when the power came back, the next day my central air conditioner wasn't working after I noticed it being hotter than blazes in my computer room. The circuit breaker for that had been tripped, none of the others were.

Re:Disturbances Before Outage Itself

[quacking duck]

All my equipment made it thru okay. If it didn't I might be knocking on the door of the company that made the surge protector--guaranteed to protect $25000 worth of equipment.

Actually maybe it's too bad it didn't fry my computer, I could've assembled a decent G5 rig with that money... ;-)

Re:Disturbances Before Outage Itself

[tomhudson]

Re: powering down linux boxes: If you're using a journaling filesystem, as long as it's not doing a sync, you won't hose the fs by pulling the plug. (Doing a sync is the last thing you want to do when the power is sagging - you won't write dependably to disk). Want to save your fs and your box - cut the power w/o doing a save.

Always freaks Windows users when I say - look, you want proof linux is better? just reach over there and pull the plug out of the wall. They refuse to, and look at me like I'm crazy when I do it, then show them the 1-second extra lost when I reboot as the journal is played back, compared to scandisk.

Re:Disturbances Before Outage Itself

[drinkypoo]

It is mu understanding that a flourescent lamp is not an arc lamp. An arc lamp produces light specifically from the spark of electrical discharge. A HID lamp which is commonly used in high-end automotive applications (there are conversion kits available for many if not all vehicles, many of them are general fit and there are only so many headlight housing shapes) operates in this fashion, for example. A flourescent lamp works by using AC power, frequently at more than 60Hz (the more modern fuller-spectrum less-flickery lights often run at 400Hz or so) and by stimulating the flourescent gas - which then flouresces.

Re:Disturbances Before Outage Itself

[gid]

Well sure you CAN do it, but that doesn't mean it's to be done. I run reiserfs, so yeah, my filesystem will remain in tact, but there's a chance that the data might not be. I've dealt with enough UPSes to know they give you enough time to totally shut down before the battery dies, it only took all of 10-15 seconds to shutdown properly.

And actually, I don't think Windows XP Pro does scandisk anymore with NTFS5, but I could be wrong, my XP box doesn't crash that much. My linux box actually goes tits up more often than XP, but that's because of a flakey hard drive or something. My secondary IDE interface with the 200 gig hard drive likes to reset occasionally, the drive never comes back, and basically any applications that try to access the drive after that lock up solid, a power cycle brings it back... rather frustrating... But that's ok, the stuff that I'm storing on that 200 gig drive I wouldn't really miss. :) My primary 120 gig drive has a nice safe back up. :)

Re:Disturbances Before Outage Itself

[Grog6]

A florescent tube is still an arc in the the sense of it being a column of ionized gas carrying electricity, but it isn't an 'Arc Lamp'.

The working 'gas' in your typical 40W tube is actually a small amount of mercury, heated and vaporized by the high voltage generated by the ballast and starter. (about 10kV at low current to start it off)

The mercury vaporizes and ionizes, dropping the voltage across the tube to about 80V; it flashes 120 times a second, as the 60Hz is full-wave rectified to make the strobe effect less obvious, as well as keeping everything warm.

The mercury gives off mostly UV light, which is absorbed by the coating inside the tube, and that gives off the visible light we see.

HPS lamps work much the same way, but sodium is used as the phosphor material instead ;it takes awhile to vaporize, which is why the light doesn't come on for a few minutes; the UV which is most of the output is absorbed by the inner glass. (thankfully)

Argon is the fill gas for both.

High frequency ballasts exist, but are expensive compared to the old standard, but are coming into wider use. If its running from dc, it is likely running at higher frequency for greater efficiency.

BTW, the outer envelope on a HID or HPS lamp is to block the UV emission; if it breaks, the lamp will still work, it will just dust all your plastic stuff (as well as you) with UV exposure. (UV is bad for plastic.)

So; None of our modern, indoor lights is an arc lamp, those are too inefficient, as well as burning the electrodes as they burn.

Most all are all electric column discharge lamps with the resulting UV converted to visible by a phosphor of some kind.

Re:Disturbances Before Outage Itself

[tomhudson]

If the fire alarm goes off, I'm hitting the power button and yanking my backup drive out of the box. Sure, I might lose the last 30 secs of code I was writing, but that's the seast of my worries at that point ... :-)

One of these days I'm going to try it on an older machine, w. a drive that contains stuff I don't mind losing, and see if I can get away with doing it w/o powering down ... (don't know unless you try :-)

Duh?!?!?

[Anonymous Coward]

slammer for 9 months for selling pipes. Not drugs, but pipes.

Selling bongs, not pipes, is against the law.

He admitted once having "a drug problem with marijuana" but said he beat it by redirecting his energy to salsa dancing.

So what was the pound of dope, that was found in his home, for? Medicinal purposes?

Underfrequency Relaying

[Anonymous Coward]

Finally someone gets to the heart of the matter. As someone who works in the power industry, and knows about UFR's, I can say that they do clear the lines quite quickly in an event. :)

"A place I worked" went dark a few years ago in a similar event. A large generator's main breaker tripped, system went unstable and underfrequency, and UFR's (which are set in multiple stages) cleared the lines. Took 8 hours to recover the 2000 MW that were lost (light load).

It's all part of standard protection equipment doing its job. You see 59 Hz? Open the breakers! 58 Hz? Open more breakers! 0 Hertz? Uh oh, here come the guys in suits...

Alternate Source

[Hal The Computer]

http://www.nrcan-rncan.gc.ca/media/documents/Black out_Summary.pdf

I think I will be fair and equitable and allow Slashdot to take out a Canadian website as well. Please be kind to Natural Resources Canada.

Re:Alternate Source

[quacking duck]

Still downloaded fine. Canadian servers hold up better against the slashdot effect, eh? ;-)

Re:Alternate Source

[Pig Hogger]

Just in case, here is my own mirror of it.

So who the hell...

[GeneralEmergency]

...declared this to be an "SCO Nonsense Free Friday"?

Stumped, I am!

Re:Draft Wesley Clark for President

[Anonymous Coward]

I do not recall some parts of it. But anyway, look at how ignorant Bush is. A Rhodes scholar with a masters in economics from Oxford cannot be worse than Bush.

Canada topic/icon?

[Mr. Neutron]

This is a US and Canada story. I notice it has the USA topic/icon, but not a Canada topic/icon. Is there a topic icon for Canada? What should the Canada icon look like?

-The Canadian flag?
-A maple leaf?
-A Mountie hat?
-A hockey stick/puck?
-Celine Dion's head?

Any other ideas?

Re:Canada topic/icon?

[Megaslow]

A Tim Horton's donut? A Labatt's Blue? Jean Chretien's Butt Crack?

Re:Canada topic/icon?

[Anonymous Coward]

A South Park style flipping head.

Re:Canada topic/icon?

[SEE]

The USA topic/icon. Canada's a protectorate allowed to play at having its own foreign and defense policies, not a soverign country of its own.

Re:Canada topic/icon?

[Anonymous Coward]

Canada's a protectorate allowed to play at having its own foreign and defense policies, not a soverign country of its own.

True, but by that argument the proper topic/icon should be the Union Jack.

Re:Canada topic/icon?

[antibryce]

A mountie carrying the Canadian flag while beating Celine Dion with a hockey stick?

Re:Canada topic/icon?

[geekoid]

you don't want them to use the canadian flag, they'll screw it up.

Re:Canada topic/icon?

[Anonymous Coward]

Hmm, only one country has ever invaded and tried to annex Canada, and guess who that was? We haven't felt the need to support a large, expensive military since American troops discovered it was just too fucking cold up here to be worth their bother =)

Re:Canada topic/icon?

[quacking duck]

A USA flag preceded by a "!" symbol.

As in "NOT USA" ;-)

Re:Canada topic/icon?

[Anonymous Coward]

You got that in reverse. The USA is Canada's biatch. We're bigger, we're on top, and if you look at those maps of southern Ontario, it's obvious who's dick is sliding into who's skanky arse.

Re:Canada topic/icon?

[SoTuA]

-Celine Dion's head?

If it's mounted on a pike and dripping, I'm all for it!

Re:Canada topic/icon?

[CoyoteGuy]

How about sexy women wearing hiking boots, and lumberjack jackets with matching skirts, drinking beer.. as in free.. as in good beer...

Re:Canada topic/icon?

[Anonymous Coward]

A picture of Dubious Bush smoking a Cuban cigar with Saddam.

Re:Canada topic/icon?

[Anonymous Coward]

For a serious answer (unlike the others), I'd say that a Canadian flag, or just the maple leaf, would be nice.

Note that I'm not Canadian myself, but I think Canada has one of the nicest looking flags in the world.

Re:Draft Wesley Clark for President

[Anonymous Coward]

Hitler was a scholar, a faithful husband and a decorated war hero.

(I'm aware of "Godwin's law". It's mere opinion and cannot dismiss valid observation)

Blame timeline

[glassesmonkey]

CNN blames lightning in Canada

Canada Dept. of Natl Defense says NO

Canadian Prime Minister says it was a fire at a Con Edison power plant in New York

Canadian Defense Minister says it's the nuclear power plant in Pennsylvania

PA's Emergency Management Agency says NO

6pm NYC Mayor Michael Bloomberg holds a news conference

Ohms and amps

[Anonymous Coward]

Most writeups of this event blather about "power flow" and "electricity sloshing". I wish someone would explain this in terms of resistance, impedance, and current.

"Suddenly the impedance in Michagan dropped. With Ontario as a constant-current source, the current through Niagara increased ..."

That'd make more sense, no?

last paragraph - small generators

[morcheeba]

In the last paragraph, I was refering to small gas-powered generators that people buy for power outages or use on their camper. Boy did that thing get pissed when I'd turn on the air conditioning!

It couldn't be greed, could it?

[FreeMars]

Meanwhile, my local electric utility stands acused of pulling power off-line in an attempt to jigger the price of power. Pioneer Press Article

Next on slashdot.org

[phorm]

East coast switches to linux powered servers

All east coast power generating stations, from nuclear reactors to hydroelectric dams, have suddenly decided to switch to Linux. When asked for the reason behind this decision, the response was:
"Well, erm, it was because of bl*cough* erm, I mean just a financial decision from the boys upstairs, I don't know anything about it"

Re:Draft Wesley Clark for President

[SteveAstro]

A Rhodes scholar with a masters in economics from Oxford cannot be worse than Bush.
Clinton was a Rhodes Scholar wasn't he.....
Steve

Energy.Gov seems to be down

[Trillian_1138]

tomhudson said, "6pm EST, Sept 12, 2003, after repeated nmappings and traceroutes (as curious /.ers try to find out why their site melted, what they were running, etc.), Department of Homeland Security claims that site is under cyber-attack."

As of 5 PM CST (6 PM EST) I'm unable to connect to energy.gov and am getting a IE "This page cannot be displayed" error. A quick hop over to netcraft shows that no uptime data is available on energy.gov, which may or may not mean anything.

Did we manage to slashdot a gov't website?

-Trillian

Re:Energy.Gov seems to be down

[tomhudson]

Yep. nmap (even nmap -P0, just in case they're dropping ping probes) reports it down. traceroute doesn't go all the way either. It's dead, Jim. Or a pretty damn good simulation.

Re:Energy.Gov seems to be down

[tomhudson]

The funny thing was I posted the Department of Homeland Security cyber-attack warning thing at 5:45 pm EST, just picked 6pm as a convenient time... So, what are the odds that a bunch of other /.ers read the joke, ran nmap and traceroute, and f$cked the server? When I tried it for real (15 minutes later) it was dead(no nmap, no traceroute).

Re:Energy.Gov seems to be down

[Cytlid]

Yea, and *I* said 5pm... so I still get 0, Funny? Geesh, I predicted it an hour beforehand!

Re:Energy.Gov seems to be down

[tomhudson]

So We're both gonna get vists from a "Mr. Smith"? (I just put the time down so that anyone looking back would realize it was a joke, including any gov't lurkers^H^H^H^H^H^H^workers).

Snipers

[axxackall]

It's happened same weekend (august/18) right after August/14. In Georgia (not american state - the former Soviet republic in Caucasian mountains between Black and Caspian Seas) the separatists used the sniper rifle to hit few wires in the inter-regional grid lines. Boom. The whole country was without electricity for a couple of days.

I guess those georgian (or perhaps chechen) snipers used August/14 as an idea for their attacks. Or did they wait until their American brothers will do the job first?

So what was the reason of wire cuts in Ohio again? No sniper bullets found around yet? No traces of bullet on broken isolators? Keep looking - you may find some more food for your brains.

The deal with the frequency

[Spamalamadingdong]

Others have kind of poked at this, but they haven't really explained it for the neophyte. I've had some education in electrical power engineering, so I'll try to fill that gap.

There are two things you need to keep in mind here. The first is that phase in AC systems performs much the same function as voltage in DC systems; just as power flows from higher voltage to lower voltage across a DC connection, power flows from leading phase to lagging phase along an AC connection. (This has to do with reactance; all power lines are inductive.) Counterintuitively, voltage helps move power but it mostly balances VARs (volt-amperes reactive); if you have a local low-voltage situation, you can connect a capacitor to add some VARs and the voltage will come up. This is part of why big inductive loads cause line voltage to dip.

The second thing is that frequency variation is just a phase change over time. If the local frequency falls for a bit, it means that the local phase is moving behind the rest of the grid. This is what you would expect if some large load was added (or a generator lost) and more power had to come from elsewhere on the grid; the delta-phase across the interconnecting lines has to shift to allow more power to flow. What little energy buffering there is is mostly the rotational energy of generators and motors, so phase changes don't quite happen instantaneously.

If you had a serious local power shortage leading to shutdown, under-frequency is exactly what you would expect. Generators trip off-line, and the phase of the local grid backs off to pull more power from outside. It would take a full second at 59 Hz to shift one cycle, so this can go on for a fair fraction of a second. If the phase change over a transmission line increases past 90 degrees it will have to trip off-line, and once the local grid is an island you can have just about any frequency that the system will try to operate at. It's my understanding that most generators trip off-line at more than a fractional Hz off 60, if for no other reason than that they aren't designed or certified to operate on a grid that's obviously malfunctioning and such a condition means trouble. Mechanical resonances at off-operating rotational speeds are another reason to shut down.

Last, I suspect your conclusion is correct.

Re:The deal with the frequency

[EuropeanSwallow]

Hello

Just to correct you:

1. Not all power lines are inductive. Underground cables (like NY underground high voltage power network) are mainly capacitive.
2. Frequency variation has nothing to do with phase shifts. As it says, it has to do with the fact that, in a transient state of the network after a load change, the generators have to balance the new load (electrical torque the generator "sees") with a change on its mechanical torque (acting on the prime mover). This doesn't happen in all generators, only the ones equipped with a speed governor, which are said to "contribute to frequency regulation". This adjustment is not instantaneous, and causes oscilation in the frequency value, until all the generators stabilize in the new operating point. Should the load change be too big, it might trip some of the generators, either by absolute deviation from the 60Hz (generally around 1Hz), or by rate of frequency change.

The What?!?!

[ewhenn]

The U.S./Canada Power Outage Task Force

We ACUTALLY have a task force dedicated to this?

So long for a timeline?

[kosibar]

It took this long to get an official timeline? Hmm. Makes me think that maybe there's some of this going on.

The blame game

[Mashiki]

Your forgetting that the Priminister of Canada was the first one along with one of his ministers to directly blame the US, that being New York(first) and Pennsylvania(second), NY blaming Ontario(third) for the blackout.

Tit for Tat.

Nobody wants to do the work of prevention

[Spamalamadingdong]

It would have been a cinch to handle such problems if we had good DSM (demand-side management). If you could dump load incrementally as required to maintain margins, the massive outages (theoretically) wouldn't be possible (DOS attacks against the communications for the DSM notwithstanding). However, DSM is still "out there" despite at least a decade of talking about it. If you couldn't buy an air conditioner, electric water heater or other big load without DSM hardware, we'd be far less vulnerable than we are now. Unfortunately we are still in the situation of giving discounts for some customers to let their systems be controlled by the power company, and the program is still so little-known that electricians have even bypassed the relays when people complained because their central air wasn't running full blast on the hottest day of the year.

The problem is the "one rate" billing system which gives no incentive to cut back. If you had to pay 25 cents a KWH on the hot afternoons and 4 cents a KWH overnight, you could have an A/C system that makes ice all night and consumes no power at all during the afternoon. But even though the system would benefit hugely from lots of people doing this, it doesn't happen... because the billing system gives all the benefit to the utilities and none to the consumer. That needs to change.

Re:Nobody wants to do the work of prevention

[davburns]

Okay, I'm clueless on this... DSM seems like an obvious solution for short-term (few miniutes) shortages, but I don't think people want their large-load appliances to stop working just because the power company decided they could make more if we didn't use them today. Maybe it would have to be more fine-grained than "shut off now" but more like "increase AC thermostats 3 degrees." Anyway, how are the air conditioner, electric heater and other big loads (and the power meter) supposed to "know" when to conserve? Would the power company send some signal? Or would they just use voltage/frequency of the lines?

At a bigger scale, I kept thinking, as I read the report, that the grid needs a "smarter" way of isolating itself. Relays that turn off lines and generators based on only local information can act like dominoes. They protect infrastructure from damage, but the emergent property is that, when the grid gets unhealty, it all shuts down. It seems like it would be better for a computer to decide (for example) that Ohio is going out, so everyone disconnect them at the same time, to save the rest of the grid from whatever is wrong there. (I guess politics aren't going to allow that, though.) Another way might be for computers to be working to find the "islands" that have ballanced generation and load, then make the relays at the edges of the islands more sensitive, and within the island less sensitive. That would allow more (and larger?) islands to remain with power, if the grid were to get unstable. It would also be a major motovation against NIMBYism. (I remember that the Bush Energy Policy noted that NYC needed a lot more power generation within the city. I wonder if any has been built.)

We've seen people do time-of-day based conservation in California and in Oregon,you can get a special power meter that will charge differently based on different times of day. That's a step away from "one rate" billing, but I don't see how that could help prevent or contain major power outages.

Out of curiosity, how much would DSM cost? If it's more than the cost of conservation, then it's easy to see why it never goes anywhere -- because more efficient appliances shed load every day, not just when the grid is unstable.

the real question is

[Anonymous Coward]

Is this more then a coincedence that it happens as James Ksansnak leaves the board of PJM?

Re:Draft Wesley Clark for President

[Anonymous Coward]

Hitler was a scholar, a faithful husband and a decorated war hero.

Hmm? Eva Braun and Hitler weren't married until the last week of April, 1945. Unusual definition of "faithful husband", that.

We had a flourescent tube that had been...

[AzrealAO]

We had a flourescent tube that had been "burned out" for about 6 months suddenly come on about 2 minutes before we lost power here in Toronto.

Re:We had a flourescent tube that had been...

[evilviper]

Okay, now why did you have a "burned out" flourescent tube plugged-in for the past 6 months?

Correction: MONITORING system went down

[Spamalamadingdong]

Something else to add: a while ago in Ohio, a nucler power plant had its control systems down for a while as a result of msblaster.exe.

The article you cites says it was a safety monitoring system (the backup digital one, not the old analog one running in parallel) that went down. This had nothing to do with control systems.

While it is worrisome that any system at a nuke plant could be disabled by worm traffic (appalling breach of good network operating practices), the criticality of a backup radiation monitoring system is very, very low.

you didn't see...

[Anonymous Coward]

his film career. That dude obviously needs to be medicated!

the twelve oclock effect

[Jack Schitt]

i wonder what would happen if you called Time in New York right after the power came back on... "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP "At the tone, Eastern standard time will be, 12:00 Exactly... BEEP" Click

Power

[Orne]

No. The Voltage component of Power is important when managing the health of the bulk power grid. For those that don't remember, Power is Volts Amps, and because current is directional, power is directional also. This is why people often use "power flow"...

The impedence in line in a function of the amount of energy flowing through it; as current increases, capacitive losses increase, causing the voltages at the ends to drop. This is sometimes called surge impedence loading. Impedence across a power line is constantly changing, and it is easier to wrap both variables into Power.

Next, the use of transformers makes amps by themselves meaningless. Power is near constant across a transformer, so High Amps Low Voltage can become Low Amps High Voltage. By talking about everything in the form of Power, then you can easily measure the transfer of energy between the various voltage levels of your system, which eases explaining the system.

Finally, Power is an easily understood market concept. If I run a generator at a low voltage (13kV), and produce 10 Amps, I'm generating 130 kW. I pipe that through a large number of transformers & lines, and deliver it to a load running at 23kV, and maybe tomorrow I sell it to someone at 9kV. By keeping everyone running in Power notation, we can all agree that money is exchanged for work, and the proper energy is delivered and paid for.

...the class system?

[anthonyrcalgary]

No thanks.

Re:Linux - equally brainded

[Anonymous Coward]

BAHAHAHAHAHAHAHAHAHA *thud* AHAHAHAHA *cough* *cough* heh... heh.... yeah Plan 9

Re:1st event...

[forgetful_ca]

-1? knuckleheads.
at worst, score 0 funny.

Re:Split rate tariffing

[paulc]

They have something like that in the UK. It used to be called Economy 7 amongst other names and was commonly used for stored energy heating. At 11pm or midnight, a relay would click and your storage heaters would start sucking power to heat some bricks inside them. 7 hours later, the relay clicks off and the heaters release their heat through the course of the day. Electricity consumed during this period was cheaper per unit than daytime power consumption.

A lot of appliances like dish washers and washing machines had timers that would delay the start of the cycle so you could put stuff in at 10pm, head off to bed, then later on the device would do it's stuff when power was cheaper.

Re:Are you down?

[dosius]

Betty Bowers reflects Christianity like Osama bin Laden reflects Islam. Get a life. Read what God really said.

Now for the topic, this should never have happened... I heard rumors about Win2K systems controlling the failsafes, that got MSBLASTER, anyone can confirm or deny? ;) If so, hey, I'd like to pummel someone in the dust for this...and it hit a little close to home, remember, Niagara Falls is home to the hydroelectric plant run by the New York Power Authority that supplies a lot of juice to NYC. Not surprising that NYC was affected. Thankfully I was only affected for half an hour! Ghad.

-uso.

Quebec's seperate and distinct power grid

[Anonymous Coward]

Did you notice that the effect stopped at the Quebec border? They also have a seperate and distinct power grid (in addition to a seperate and distinct culture) as a remnant of the repairs after the 98(?) Ice storm. They were quite pleased that the effect could not propogate through the DC buffers at the NY and Ontario borders

Re:Quebec's seperate and distinct power grid

[Anonymous Coward]

Funny. The Phase Angle Regulators and DC ties between Hydro Quebec and the other power pool members were put in place as a penalty because HQ had such a high frequency of blackouts due to ice storms. The surrounding territories were tired of being dragged down with HQ's poor control, and HQ was forced to isolate by going to DC.

Ironically, it helped HQ stay up while the southern zones fell, because they were unaffected by the AC frequency shifts of the system separation of Aug 14th.

Re:Quebec's seperate and distinct power grid

[Anonymous Coward]

high frequency of blackouts due to ice storms? What are you talking about? This is pure FUD. The HQ system was very stable before the Big Ice Storm of 98, it was even better afterwards. AFAIK, we have the best power system in the world. As soon as the blackout happened, we were able to offer about 7000 Mw to rebuild the grid, and we did, as this excerpt from the document shows:

One relatively large island remained in operation serving about 5700 MW of demand, mostly in western New York. This service was maintained by generating stations south of Lake Ontario with Ontario generators at Niagara and St. Lawrence as well as the 765 kV and DC interties with Quebec. This island formed the basis for restoration in both New York and Ontario.

So where are the thank yous?? Are they lost as the thank yous Canada deserved when we took care of all those airplanes that were redirected to us on 09/11?

Re:Quebec's seperate and distinct power grid

[gfreeman]

Where are the Thank Yous? Hmm, where were the official Thank Yous from Quebec to Ontario and the other provinces for bailing out your asses in 98?

CNN

[Anonymous Coward]

It was a laugh tuning in the American networks. Canadian media was saying that there was a large power outage, CNN was saying 'We have no confirmation that this is a terorist attack' LOL They just love their hype and sensationalism

I know who I am going to blame (for fun)...

[theycallmeB]

Even though I am sure he had nothing to do with it.

A friend of mine works at NASA's Glenn Research Center near Cleveland (about where this whole mess started). GRC is where NASA has a very large, supersonic wind tunnel (large as in can accommodate full size mock-ups of fighter planes). When they run it during peak hours (rarely), it costs upwards of a million dollars per hour and causes localized blackouts in Cleveland. So I will have to make some sort of comment along the lines of "You just HAD to push to big, green button, didn't you?"

I also know that it wasn't the tunnel I work with in Chicago; we can only cause localized brownouts from the transient currents when we start the motor (rated for 1.5MW, starting power-draw is several times that). On an unrelated note, the sound of the capacitors charging is very similar to the noise you would expect from an evil villain's death ray.

bleh

[Orne]

There should be a Slashdot "law"... the number of times you Preview your message is directly proportional to the number of errors you make in your post...

"The impedence in a line is a ..."

Re:Sad news ... John Ritter dead at 54

[ThisIsFred]

Now, he died because of some torn artery. And unlike your Stephen King post the other day, this one is for real.

Could it have been an EMP?

[asscroft]

could it have been an electro magnetic pulse, like in that movie where they break into the vault in the vegas hotel. Only it went a little haywire and gave them more than a 20 second window. That was my first thought. Probably a little imaginative, but it's still always nice to hear someone explain why this isn't the case.

yes, yes, I know with that attitude we'd have to hear people explaining about why it's not aliens, and not due to sun spots and any other thing some wacko can think up. But at least my idea was in a recent movie, so that gives explaining it away some entertainment value.

Or are they not explaining this away for the same reason they haven't explained away the crash over pennsylvania NOT being caused by an ordered shot. I.E., we'd rather not lie to you so we'll just not answer that question.

Anyone missing some money following that blackout?

Re:Could it have been an EMP?

[asscroft]

Holy Shit. I just tied something entirely unrelated to 9/11. I suck. Mod me down. Damn, I hate that.

Re:Could it have been an EMP?

[Anonymous Coward]

What did you say you were smoking again?

Roadmap?

[dpeltzm1]

Is it my imagination or did we just give the bad guys a roadmap on how to do this on purpose? I understand the public 'needs to know' but on the other hand i need to be able to sleep at night! and while im sure this report will fade into memory as most things in our country seem to do, i'll bet someone else will remember it just fine.

Re:Roadmap?

[Anonymous Coward]

i need to be able to sleep at night!

Me to. So please, mr terrorist... I want a blackout at 11:00pm every friday so I won't hear my neighbour's fucking music! (Well actually the only thing I hear is BOOM BOOM BOOM BOOM...)

And the best part is I may finally be able to see some stars without having to go 200 miles away from my home!

Seriously... Are you really terrorized byblackouts?

Re:Roadmap?

[DerekLyons]

Is it my imagination or did we just give the bad guys a roadmap on how to do this on purpose?

Short answer: No.

Long answer; Such cascades are very sensitive to input conditions, it would be very difficult to replicate.

bull

[Anonymous Coward]

"Dassault - who make the core avionics software in every modern military aircraft, including U.S."

There is NO "core" avionics in modern military aircraft. And the Avionics code I saw (and helped write) used by USAF had "NOFORN" all over it (NOFORN = no foreigners may see it, even NATO, even if they have top secret clearance).

Re:bull

[Anonymous Coward]

mostly because they stole it from foreigners.

Why your A/C tripped

[Anonymous Coward]

Your A/C is a 230V circuit. It draw power actually from two different, out of phase supplies. If the phase angle between those two lines (called legs) varies from normal, you get enormous current flows between the legs and neutral.

So likely one of the legs in your house either dropped or went out of phase and caused the A/C breaker to trip.

Re:Why your A/C tripped

[gid]

Interesting, I knew it drew off of both legs that come into my house, didn't know why it tripped. Makes sense tho.

Re:Why your A/C tripped

[shoemakc]

Your A/C is a 230V circuit. It draw power actually from two different, out of phase supplies

Most residential service (here in the US anyway)is in fact 3 wire single phase service. Each of the two wires is 120V to ground, while from hot to hot is 240V. If the two legs were simply 2 of the 3 legs of a 3-phase sytem, the voltage from hot to hot would be 208V (120*sqrt(3)). If the two legs were a 2 phase system, each leg would be 180 degrees out of phase, and the phases would cancel each other out.

-Chris

Re:Why your A/C tripped

[Skapare]

Motors initially draw an excess of current as they start up (they would also do this if the mechanical load were suddenly increased). If the voltage drops so low as to not allow the motor actually get going, it would be "stuck" in the high current situation for quite some time. This is why you see the lights dim briefly in many places when motors like A/C compressors start up.

In 2-phase circuits, the transformer is center tapped on the secondary, and the primary is on just one phase of the three phase source. You get 120v between either leg and the center (ground/neutral), and 240v between the opposing legs. A phase difference would certainly lower the voltage, but that's not going to happen on a 2-phase circuit unless you have other reactive loads there.

But, it may have actually been a 208v circuit between 2 legs of a 3-phase circuit. Most modern A/C systems are rated to operate even on 208 volts, so this kind of thing is not uncommon. It's cheaper to supply 208v instead of 240v to comemrcial buildings which already have a need for 3-phase motors (larger motors and A/C systems) than it is to supply genuine 240v (because this would require making the 3 transformers center tapped, and running 7 wires (6 point star and neutral) rather than 4 (3 point star and neutral). If a leg changed phase relative to the others, it could certainly upset the voltage balance, and the A/C could get undervoltage or overvoltage.

A frequency change can also cause problems, especially with 3-phase motors. A sudden phase shift (a brief frequency change) can cause a motor to go out of sync and draw more current to get back in sync.

Not the mayor of NYC!!! He wouldn't do that!!

[Evil MarNuke]

Come on, this is a guy that had an approval rating of 24 percent in June. Get with the program!

Second blackout happening!

[Mercury2k]

Friday September 12, @03:58PM: Server goes down due to unusually high traffic.
Friday September 12, @04:01PM: Server load causes electric breakers to trip power in whole building.
Friday September 12, @4:02PM: Power surge from LAN going offline and slashdot visitors surge cause second blackout

hehe, well, It sounded funny in my head :)

Re:Draft Wesley Clark for President

[Excen]

Hey fucknuts! Clinton wasn't a half-bad president. We weren't running around beating up people wearing turbans, we actually had foreign policy that wasn't based on greed (Operation Iraqi Liberation, you figure out the acronym), people had jobs, and THE FEDERAL BUDGET HAD A SURPLUS FOR ONCE IN A HUNDRED YEARS. How dare you badmouth the one president that actually took time to deal with domestic issues with anything besides a frigging tax cut.

"No beer until you finish your tequila"
-Leela's Dad

Vulnerability

[Databass]

from the summary:

4. Stuart - Atlanta 345 kV
This line is part of the transmission pathway from southwestern Ohio into northern Ohio. It
disconnected from the system due to a brush fire under a portion of the line. Hot gases from a fire can ionize the air above a transmission line, causing the air to conduct electricity and short-circuit the conductors.

So twenty guys with twenty gas cans could start 20 brush fires under 20 high-transmission lines and cause something like this to happen again? I wonder if Al Queda read this article?

Seems to be simple.

[cheeseflan]

I am unsure about the obsession with the blaster worm in this thread.

From a fairly careful read of the .pdf it seems that the outage was a simple cascade failure.

A few issues that happened on the same area of the grid forced a change in the whole grid. Add to that, a lack of central (or coherent) control and you have an inability to rebalance the network. That's how (I believe) the system failed.

This is plain and simple a failure of the regulatory regime (a.k.a. deregulation that is anything but). Nowhere near enough reserve capacity and interconnection is available, and more importantly, too many people are in charge of planning. This means there is no thought applied to the wide area effects of implementation decisions. A clear example of this is interconnects that try to re-join the grid automatically - rather than waiting for intelligent or coherent systems control to allow reconnection.

Grids in the rest of the world know this, and so retain central control even when the implementation is through the private sector. The central control is often private, although it is in charge of overall planning and provisioning.

Perhaps the US regulators and legislators could actually learn something here?

FIRST SEVEN-YEAR THEORY POST

[Spooge Knight]

I KILLED PAC!

Re:"Required" email

[Anonymous Coward]

We do use cosine waves here in Canada, smartass. But we run it 90 degrees out of phase.

You hit the nail on the head

[Spamalamadingdong]

DSM seems like an obvious solution for short-term (few miniutes) shortages, but I don't think people want their large-load appliances to stop working just because the power company decided they could make more if we didn't use them today.

If I understand correctly, the typical program was to run things on something like a 15 minutes on, 15 off basis. I didn't read about actual programs other than for peak-shaving during the summer heat, so the only appliances I've read about being so controlled in real time were central air conditioners. Electric water heaters on timers are another matter; my electric company has had discounts for timed usage for as long as I can remember seeing their rate schedules.

Maybe it would have to be more fine-grained than "shut off now" but more like "increase AC thermostats 3 degrees." Anyway, how are the air conditioner, electric heater and other big loads (and the power meter) supposed to "know" when to conserve? Would the power company send some signal? Or would they just use voltage/frequency of the lines?

As you concluded, it would require hardware at the appliance itself. This presents a big problem of infrastructure, which means there has to be a standard for its function. When I was keeping tabs on it the communications seemed to be tilting toward a more comprehensive SCADA system built around ASN.1 format messaging, but that was 8 years ago.

It seems like it would be better for a computer to decide (for example) that Ohio is going out, so everyone disconnect them at the same time, to save the rest of the grid from whatever is wrong there.

It would have been even better if Ohio had shed load (plants, neighborhoods) as the feeders went down. The rolling blackouts would have been quite newsworthy, but they would have been a local problem.

The ideal situation would have been for Ohio to command non-critical loads down to half power or even less as the situation deteriorated. There are variable-speed A/C compressors which can run at pretty much any power level you command up to 100%, and dimmable fluorescent fixtures could back lighting load down to fractional power as well. If the communications link went all the way down to each outlet, your computer could even do its part by backing down its clock speed. The possibilities are myriad. Unfortunately, we have not even begun to explore them.

Out of curiosity, how much would DSM cost? If it's more than the cost of conservation, then it's easy to see why it never goes anywhere -- because more efficient appliances shed load every day, not just when the grid is unstable.

That's a non-sequitur. As a customer, I want DSM so that the other guy can't overload the grid and take the power down. Conservation saves me a ton of money, but my own use of CF lamps, thrifty computer and open windows instead of A/C doesn't help me if everyone else's load brings the system down. The tragedy of the commons is how we got here.

You're wrong. One hint

[Anonymous Coward]

RTFP

Probably because building management are lazy.

[AzrealAO]

It was an outside office with windows on one wall, so lots of active lights aren't really necessary, and building managment is reluctant to be bothered changing the lights unless all the lights in an office are burned out and we complain for a few days.