NERC Releases Interim Report on Aug 14th Blackout

will writes "The North American Electric Reliability Council has released four documents concerning the August 14th power outage power outage in the North East. The blackout investigation homepage lists all NERC's documents relating to this event. Press coverage is at The Washington Post, CNN, and CBS News. The take home message: FirstEnergy did it. The are, of course, denying it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."

Comments

No, Sue the End User (or at least the analyst)!

[FreeUser]

Good, let's sue SCO!

heh! :-)

Alas, as Microsoft apologists are wont to point out (even in the many cases where the crash or security flaw doesn't stem from mismanagement or configuration errors), a misconfigured system ins't going to work regardless of what OS it is running.

In this particular case, quoth the article

One of MISO's monitoring systems required technical repairs that afternoon, but the technician who fixed it forgot to turn on an automatic feature that updated information every five minutes, preventing it from operating normally, the report said. "Thinking the system had been successfully restored, the analyst went to lunch."

The technician forgot to restart the monitoring software. Oops.

Following in $CO's illustrious footsteps, I think perhaps it is time we sued the poor schmuck who forgot to restart the monitoring program. Or better yet, the company dumb enough to hire him, the electrical company. After all, according to Our Lord and Master Darl McBride, End Users should pay (and pay heavilly).

Re:No, Sue the End User (or at least the analyst)!

[MrResistor]

I can't help but think, if the process has to be started by someone, it isn't automatic, is it? And if it's so damned important, why wasn't it started automatically?

Seems to me the stupidity predates the technician.

Re:The XA/21 Did Not Break Down

[Troed]

Mod parent up. Windows/Blaster WAS one of the causes of the blackout, something that has been known for quite some time. Nice to see official confirmation. The submitter has taken too much SCO crack me thinks.

SCADA failed to transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack.

What caused the boxes to crash?

[AJWM]

It may have been a Unix box that crashed, but I've seen and heard of cases where a 'nix box crashed because of the high network load of a Windows-based worm/virus epidemic. Was this the case here?

(There should have been better firewalling in place if so, of course.)

Re:Now we know...

[JoeBuck]

The biggest difference is that France puts their power lines underground. Here in Silicon Valley, some French colleagues were shocked at the frequent power disruptions during the winter. Well, of course the power was out, the locals tell them, there was a big storm! (The "big storm" consists of an inch of rain and a short period of 40 mph winds, which knocks down all the power lines).

What were the failed systems running?

[mabu]

After browsing all the comments thus far, there doesn't seem to be a definitive answer as to what OS these failed systems were running. Let's find this out... it's also of interest that in the Windows license agreement, it specifically states that Windows shouldn't be used for mission-critical applications involving scenarios like this, so it would be further hilarity if they were running Windows in contrast to the products' own licensing agreement.

Power outage traced to dim bulb in White House

[TheRealStyro]

Investigative journalist Greg Palast knows what caused the power outage - deregulation. Here are some excerpts from his article on the subject...

I can tell you all about the ne're-do-wells that sent us back to the Dark Ages last week. I came up against these characters -- First Energy and the Niagara Mohawk Power Company -- some years back. You see, before I was a journalist, I worked for a living, as an investigator of corporate racketeers.

The power outage began in First Energy's Ohio operation. This company was the model for the film, "China Syndrome." Really. Then First Energy's Pennsylvania unit fumbled the power ball. These are the very same Homer Simpsons who melted Three Mile Island.

...

Is last week's black-out a surprise? Heck, no, not to us in the field who've watched Bush's buddies flick the switches across the globe. In Brazil, Houston Industries seized ownership of Rio de Janeiro's electric company. The Texans (aided by their French partners) fired workers, raised prices, cut maintenance expenditures and, CLICK! the juice went out so often te locals now call it, "Rio Dark."

So too the free-market British buckaroos controlling Niagara Mohawk raised prices, slashed staff, cut maintenance and CLICK! -- New York joins Brazil in the Dark Ages.

The short of it? Bush and cronies deregulated the energy industry (and continuing even more this week) and the industry responded just as any greedy entity would - raise prices, fire workers, slack off on maintenance and pocket the savings.

FDR enacted regulation of certain industries to insure that consumers would not get ripped-off. Bush reversed these regulations possibly because he doesn't know history, and/or he and/or friends/relatives have stock in these industries, or some other overlooked reason. Bush and his administration heavily contributed to the power outage, and is making sure that plenty more are bound to happen. Remember this on the next election day.

Re:SCADA systems are DCOM based

[Anonymous Coward]

Yes,

SCADA systems are running a stripped down version of Windows 2000 or NT for the nodes, and the
front ends are UNIX.

The front end did not crash, it was not getting data.

Thanks to a few MS worms running around, I'll wager.

Check out this link:
http://itco.web.cern.ch/itco/Projects-Servi ces/JCO P/ProjectTeam/Minutes/1999/24-11-99/scada_unix.pdf

SCADA Definition

[pjkundert]

For those of you that might be interested, SCADA is an acronym for

Supervisory Control And Data Acquisition

It defines nothing about whether or not COM, DCOM, OPC (Ole for Process Control), or any other proprietary communication framework is used (contrary to some other highly moderated statements you are likely to see in this discussion)

The bulk of serious SCADA systems in place are probably legacy systems of some kind, including many variants of UNIX systems (we have old micro VAX systems still in use). Many of the newer systems are Windows based, and are obviously subject the standard Windows viri, worms, etc.

Worse yet, these systems are very difficult to upgrade or patch, due to the critical nature of their duties. It is not unlikely that a large portion of the Windows based SCADA systems in the world remain unpatched, and are "safely" firewalled off from the internet.

Of course, the problem is that much of the monitoring gear used to diagnose network issues is also Windows based, and carries with it the standard retinue of Windows viruses and worms, right into the heart of the control center.

These UNIX systems have run (and will contiue to run) uninterrupted for years at a time. We have calendar alerts in place to tell us to go manually reboot unix machines after months of uptime, just to ensure that their SCSI drives will spin back up (in case of a control center power outage, etc.)

Somehow, I don't think that is an issue for some vendor's SCADA systems based on more popular OS's, but I might be wrong...

Re:Now we know...

[JayBlalock]

Simple scientific method-approach to a problem. A Problem exists. If you cannot logically derive the solution beforehand, you try Solution A. If, given adequate time, Solution A fails to solve the problem, you move on to Solution B.

Our power grid has not been upgraded significantly in decades. The power companies have shown quite well that, without direct regulation, they are not going to willingly do so. (especially as the costs of upgrading go up every year technology advances and the grid goes up) Ergo, at this point it's time to try a new solution.

I'm not talking about centrally socializing the entire grid or something. In a country our size, that would be impractical (as opposed to someplace much smaller like France). But the fact remains, it is a gigantic problem for the country - both in terms of security AND the sheer common sense social benefits to having a stable power grid. And it's getting worse every year thanks to entropy.

My proposal would be along the lines of: Form an INDEPENDENT committee. (IE no industry folks acting as anything but advisors) It decides what needs to be upgraded for us to have a power grid roughly on par with Europe's. It gives the power companies adequate time (a decade? Years anyway) to achieve this. And it gives them some money to do so. (as I pointed out in another post, since this is very much a matter of national security, I can see no reason why we can't take a billion or so normally meant for building bombs and use it for the power grid instead)

Canada would undoubtedly help us out in this, as our grids are (we know now) so closely tied together.

And at the end of that time limit, if the power companies have failed to upgrade themselves, THEN we start talking about getting draconian and collectivizing everything - since at that point the power companies would have well proven beyond reasonable doubt they are putting their own financial profits ahead of *national security.* Such an entity would have no right to exist within the US, as far as I'm concerned. (or at least not to own and control such an important chunk of our infrastructure)

(and sadly, I can easily see them pocketing the money and then doing nothing. Just look at how much Verizon has stolen from various states with promises of fiber-optic upgrades that never happened)

Re:Report vindicates First Energy

[Melantha_Bacchae]

Yep, Ohio's infamous Hole-In-The-Head reactor was First Energy's baby. Enough primary coolant (the stuff that cools the rods and glows a pretty blue around them from the radiation) was leaking out the top of the reactor head to eat a hole big enough to put a gallon milk jug through. The only thing between America and Chernobyl was a one-sixteenth inch thick stainless steel plate on the outside. Enough coolant did make its way to the outside that the top had to be wiped off periodically. This went on for six years, with several bungled inspections failing to catch it, and a cover up to attempt to let First Energy to continue running the thing. First Energy was still trying to get the government to let them patch and restart it when the blackout happened.

Funny thing about the blackout: nuclear plants and dirty fossil fuel plants were affected. Clean plants, like the Niagara Falls hydroelectric plant, blinked, then went right on running.

Another funny thing about the blackout: five days before the lights went out, "Godzilla X Megagiras" had had its US premiere at the Egyptian Theatre. In this movie, Godzilla's attacks on nuclear plants cause Japan to ban all nuclear energy and look to clean energy. The bad guy keeps building fusion plants (fobbing them off as "clean energy"), but they always leak radiation, and Godzilla always finds them and knocks their lights out.

You might want to stock up on flashlights. "Godzilla X Megagiras" is coming out on DVD in the US January 6th.

"Is Godzilla showing his hatred toward man-made energy?"
Shinoda, "Godzilla 2000 Millennium" (Japanese version)

Cleveland eye witness to FirstEnergy

[peptidbond]

Well, I do not post much, but I thought this was worth it.

The day of the black out, I was heading home to my house in Cleveland. Naturally, we were hit first. It was beautiful that night because my neighbor had his 12" telescope out in his yard.

However, what is really intriguing is what happened over the next week. Everyday, driving to and from work I would notice all of these FirstEnergy trucks out on the street. No, they were not restoring power. THEY WERE TRIMMING TREES.

The NERC report states that FirstEnergy failed to trim trees to protect the grid. FirstEnergy knew that they had caused the problem and they tried to cover it up.

Also keep in mind that FirstEnergy owns the troubled Davis-Bessie Nuclear plant in Sandusky, Ohio. The nuke plant was shut down almost 2 years ago because boric acid had eaten a football sized hole in the reactor lid. It has been repaired but not re-started due to "safety culture concerns".

Finally, my G/F is a field biologist. She is constantly doing field surveys for FirstEnergy. She always speaks of their poor land use and inability to maintain their lines. She has even seen over-heated lines drooping almost to the ground.