NERC Releases Interim Report on Aug 14th Blackout

will writes "The North American Electric Reliability Council has released four documents concerning the August 14th power outage power outage in the North East. The blackout investigation homepage lists all NERC's documents relating to this event. Press coverage is at The Washington Post, CNN, and CBS News. The take home message: FirstEnergy did it. The are, of course, denying it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."

Comments

No, Sue the End User (or at least the analyst)!

[FreeUser]

Good, let's sue SCO!

heh! :-)

Alas, as Microsoft apologists are wont to point out (even in the many cases where the crash or security flaw doesn't stem from mismanagement or configuration errors), a misconfigured system ins't going to work regardless of what OS it is running.

In this particular case, quoth the article

One of MISO's monitoring systems required technical repairs that afternoon, but the technician who fixed it forgot to turn on an automatic feature that updated information every five minutes, preventing it from operating normally, the report said. "Thinking the system had been successfully restored, the analyst went to lunch."

The technician forgot to restart the monitoring software. Oops.

Following in $CO's illustrious footsteps, I think perhaps it is time we sued the poor schmuck who forgot to restart the monitoring program. Or better yet, the company dumb enough to hire him, the electrical company. After all, according to Our Lord and Master Darl McBride, End Users should pay (and pay heavilly).

Re:No, Sue the End User (or at least the analyst)!

[MrResistor]

I can't help but think, if the process has to be started by someone, it isn't automatic, is it? And if it's so damned important, why wasn't it started automatically?

Seems to me the stupidity predates the technician.

Re:Now we know...

[JoeBuck]

The biggest difference is that France puts their power lines underground. Here in Silicon Valley, some French colleagues were shocked at the frequent power disruptions during the winter. Well, of course the power was out, the locals tell them, there was a big storm! (The "big storm" consists of an inch of rain and a short period of 40 mph winds, which knocks down all the power lines).

Power outage traced to dim bulb in White House

[TheRealStyro]

Investigative journalist Greg Palast knows what caused the power outage - deregulation. Here are some excerpts from his article on the subject...

I can tell you all about the ne're-do-wells that sent us back to the Dark Ages last week. I came up against these characters -- First Energy and the Niagara Mohawk Power Company -- some years back. You see, before I was a journalist, I worked for a living, as an investigator of corporate racketeers.

The power outage began in First Energy's Ohio operation. This company was the model for the film, "China Syndrome." Really. Then First Energy's Pennsylvania unit fumbled the power ball. These are the very same Homer Simpsons who melted Three Mile Island.

...

Is last week's black-out a surprise? Heck, no, not to us in the field who've watched Bush's buddies flick the switches across the globe. In Brazil, Houston Industries seized ownership of Rio de Janeiro's electric company. The Texans (aided by their French partners) fired workers, raised prices, cut maintenance expenditures and, CLICK! the juice went out so often te locals now call it, "Rio Dark."

So too the free-market British buckaroos controlling Niagara Mohawk raised prices, slashed staff, cut maintenance and CLICK! -- New York joins Brazil in the Dark Ages.

The short of it? Bush and cronies deregulated the energy industry (and continuing even more this week) and the industry responded just as any greedy entity would - raise prices, fire workers, slack off on maintenance and pocket the savings.

FDR enacted regulation of certain industries to insure that consumers would not get ripped-off. Bush reversed these regulations possibly because he doesn't know history, and/or he and/or friends/relatives have stock in these industries, or some other overlooked reason. Bush and his administration heavily contributed to the power outage, and is making sure that plenty more are bound to happen. Remember this on the next election day.

SCADA Definition

[pjkundert]

For those of you that might be interested, SCADA is an acronym for

Supervisory Control And Data Acquisition

It defines nothing about whether or not COM, DCOM, OPC (Ole for Process Control), or any other proprietary communication framework is used (contrary to some other highly moderated statements you are likely to see in this discussion)

The bulk of serious SCADA systems in place are probably legacy systems of some kind, including many variants of UNIX systems (we have old micro VAX systems still in use). Many of the newer systems are Windows based, and are obviously subject the standard Windows viri, worms, etc.

Worse yet, these systems are very difficult to upgrade or patch, due to the critical nature of their duties. It is not unlikely that a large portion of the Windows based SCADA systems in the world remain unpatched, and are "safely" firewalled off from the internet.

Of course, the problem is that much of the monitoring gear used to diagnose network issues is also Windows based, and carries with it the standard retinue of Windows viruses and worms, right into the heart of the control center.

These UNIX systems have run (and will contiue to run) uninterrupted for years at a time. We have calendar alerts in place to tell us to go manually reboot unix machines after months of uptime, just to ensure that their SCSI drives will spin back up (in case of a control center power outage, etc.)

Somehow, I don't think that is an issue for some vendor's SCADA systems based on more popular OS's, but I might be wrong...

Cleveland eye witness to FirstEnergy

[peptidbond]

Well, I do not post much, but I thought this was worth it.

The day of the black out, I was heading home to my house in Cleveland. Naturally, we were hit first. It was beautiful that night because my neighbor had his 12" telescope out in his yard.

However, what is really intriguing is what happened over the next week. Everyday, driving to and from work I would notice all of these FirstEnergy trucks out on the street. No, they were not restoring power. THEY WERE TRIMMING TREES.

The NERC report states that FirstEnergy failed to trim trees to protect the grid. FirstEnergy knew that they had caused the problem and they tried to cover it up.

Also keep in mind that FirstEnergy owns the troubled Davis-Bessie Nuclear plant in Sandusky, Ohio. The nuke plant was shut down almost 2 years ago because boric acid had eaten a football sized hole in the reactor lid. It has been repaired but not re-started due to "safety culture concerns".

Finally, my G/F is a field biologist. She is constantly doing field surveys for FirstEnergy. She always speaks of their poor land use and inability to maintain their lines. She has even seen over-heated lines drooping almost to the ground.