NERC Releases Interim Report on Aug 14th Blackout

will writes "The North American Electric Reliability Council has released four documents concerning the August 14th power outage power outage in the North East. The blackout investigation homepage lists all NERC's documents relating to this event. Press coverage is at The Washington Post, CNN, and CBS News. The take home message: FirstEnergy did it. The are, of course, denying it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."

Comments

Go get 'em!

[grub]

"According to the product specs, it is a Unix system with X Windows."

Good, let's sue SCO!


ps: f1st pr05t

Re:Go get 'em!

[orthogonal]

'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows.

Darl McBride had repossessed the system, leaving a bill for $699 and a subpoena in its place.

Re:Go get 'em!

[tekspot]

"According to the product specs, it is a Unix system with X Windows."

The reason they use X is because their systems are managed by MCSEs.

Microsoft sucks

[Meor]

[Slashdot]
You shouldn't use MS products for such a critical system.

What? It was a unix system?

Must have been a hardware failure then.
[/slashdot]

Memo to power company:

[SuperBanana]

Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.

Memo to power company:

Put power-system controlling servers on UPSes :-)

(yeah yeah, I know, it wasn't because they lost power. Its a joke :-)

Now we know...

[zeux]

I think it's really hard to perform that kind of investigation and looking at the results I think they did a good job making the results widely available. Anyway, it's still unacceptable that such things happens in the most powerful country in the world.

Flame me if you want but France is known to have one of the best electrical system in the world. The government owns it and some laws do protect the costumer. It allows France to export a lot of power to other European countries thus making a lot of money from it and playing a big role in Europ energy.

I think France made good choices for power:
- nuclear (or nuculear whatever). Much cheaper and cleaner than oil. France is very strong on nuclear technology and do export its knowledge to other countries. To date we didn't have any major incident and I think it's pretty secure as long as enough money is spent.
- public service. There is a law in France stating that every citizen, wherever he lives, as the right to have access to electricity for free. Of course we pay bills but if you live in the country the government MUST bring you access to electricity even if you live far from everything. It's a law so sometimes it's not really followed but most of the time the government sticks to it.

When I arrived in the USA I was shocked by the poor quality of your electric system. Many outages, expensive bills and thousand of wires in the sky ! I think that it's a very advanced country with an outdated power system. I've seen on the TV recently that many companies as starting to produce their own power. I really think it's pretty bad since the installations needed are dangerous, potentially explosive and very expensive.

What happened on August the 14th also suggests that a country really depends on its energy distribution system and that it could be, in the near future, target for terrorists.

Re:Now we know...

[(startx)]

It's all about early adopter. We invested huge amounts of money to wire up the whole country before nuclear was an option, and there's no way we can spend that much again to rip it all out and replace it with newer/better/safer alternatives. Same thing with our telephone system, cable tv, internet, etc. We're trying to squeeze every last ounce of usefulness out of the existing system, while so-called "3rd world" countries are getting the latest and greatest tech because they have no prior investments.

Re:Now we know...

[JoeBuck]

The biggest difference is that France puts their power lines underground. Here in Silicon Valley, some French colleagues were shocked at the frequent power disruptions during the winter. Well, of course the power was out, the locals tell them, there was a big storm! (The "big storm" consists of an inch of rain and a short period of 40 mph winds, which knocks down all the power lines).

Disaster again

[Camel Pilot]

If I remember correctly wasn't a "Unix system" also implicated in the chaotic havoc that was Jurassic Park?

The XA/21 Did Not Break Down

[Anonymous Coward]

GE's XA/21 system "controls generation and the high voltage transmission network for optimal generation and transmission of power," as provided on GE's corporate web site:

http://www.gepower.com/prod_serv/products/scada_ so ftware/en/xa21.htm

On page 99 of the report the task force states specifically that,"although there were a number of worms and viruses impacting theInternet and Internet-connected systems and networks in North America before and during the outage, the SWG's preliminary analysis provides no indication that worm/virus activity had a significant effect on the power generation and delivery systems."In other words, the power generation and delivery systems (GE's XA/21) running Unix were not affected. SCADA, the alarm monitoring system, however, was affected. SCADA failedto transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack. The net result is that the operators were not aware of serious trouble with the grid before it was to late.

No, Sue the End User (or at least the analyst)!

[FreeUser]

Good, let's sue SCO!

heh! :-)

Alas, as Microsoft apologists are wont to point out (even in the many cases where the crash or security flaw doesn't stem from mismanagement or configuration errors), a misconfigured system ins't going to work regardless of what OS it is running.

In this particular case, quoth the article

One of MISO's monitoring systems required technical repairs that afternoon, but the technician who fixed it forgot to turn on an automatic feature that updated information every five minutes, preventing it from operating normally, the report said. "Thinking the system had been successfully restored, the analyst went to lunch."

The technician forgot to restart the monitoring software. Oops.

Following in $CO's illustrious footsteps, I think perhaps it is time we sued the poor schmuck who forgot to restart the monitoring program. Or better yet, the company dumb enough to hire him, the electrical company. After all, according to Our Lord and Master Darl McBride, End Users should pay (and pay heavilly).

Re:No, Sue the End User (or at least the analyst)!

[MrResistor]

I can't help but think, if the process has to be started by someone, it isn't automatic, is it? And if it's so damned important, why wasn't it started automatically?

Seems to me the stupidity predates the technician.

You got it in reverse

[just fiddling around]

FYI, US governement have been engaged in a nice deregulation wave for 10 years. That deregulation has been poorly tought out, because of a "The Market Cures All" mindset. What did the deregulation bring? Gigantic power outages and California-style power crunches.

The deregulated US power grid is overstressed, has little or no margin for extra demand, and has outdated and poorly maintained emergency systems. And thanks to Enron and friends, artificial scarceness has been created to crank up prices.

In a nutshell, you pay more and get less(safety, avaliability, quality) than when the market was regulated. Ain't The Market grand?

Electric power is a basic necessity(in our post-industrial world), and should be managed by the state, just like water works and law enforcement.

Re:You got it in reverse

[Vainglorious Coward]

Electric power is a basic necessity...

Absolutely, and it's interesting to note that planned US spending on electrical infrastructure to 2005 is 71 cents per American per year. Meanwhile, the US gov. plans to spend $255 per citizen in Iraq! (Health spending plans are also illuminating : $38 per capita on hospitals in Iraq versus $3.30 in the US).

If I was a US citizen, I would be furious about this failure to invest my tax dollars in my own country's infrastructure.

Who cares about the OS or the provider?

[swordgeek]

First of all, let's be perfectly clear on this.

Any carefully managed OS (inluding Windows) can be stable and predictable.

Any badly managed OS (including Unix) WILL be unstable and unpredictable.

Got it? Good.

Now as far as pointing fingers at the guilty parties, understand that the infrastructure is really at fault far more than any individual company. Look at this from a broader perspective: One company was able to take down how wide of an area? The whole system is too fragile, too interdependent, and maintained too close to full capacity. Worse, there's absolutely no incentive for a company to maintain a large reserve capacity, since it costs a lot, and brings in no extra income.

But of course when it happens the next time (and make no mistake--it will), we'll have another commission to once again figure out the single company that broke a rotten and unstable infrastructure, instead of fixing the root problem.

First Energy is a horrible corporation

[puppetluva]

Here's a timely link. . .

FirstEnergy used to be CEI. . . a horrible company in Cleveland that allowed a nuclear core failure through negligence and then tried to bully the government of Cleveland into selling its municipal power supply to it thereby granting it monopoly status in the region.

Check out the story here:
http://kucinich.us/powertothepeople.htm

The presidential candidate Dennis Kucinich was Mayor of Cleveland at the time and endured a browbeating when he refused to sell the Municipal power company to these scoundrels.

I'm not necessarily pro-Kucinich, but I am VERY anti-FirstEnergy. They have a track record of irresponsibility and dishonesty that should be enough to have the government dismantle them.

Power outage traced to dim bulb in White House

[TheRealStyro]

Investigative journalist Greg Palast knows what caused the power outage - deregulation. Here are some excerpts from his article on the subject...

I can tell you all about the ne're-do-wells that sent us back to the Dark Ages last week. I came up against these characters -- First Energy and the Niagara Mohawk Power Company -- some years back. You see, before I was a journalist, I worked for a living, as an investigator of corporate racketeers.

The power outage began in First Energy's Ohio operation. This company was the model for the film, "China Syndrome." Really. Then First Energy's Pennsylvania unit fumbled the power ball. These are the very same Homer Simpsons who melted Three Mile Island.

...

Is last week's black-out a surprise? Heck, no, not to us in the field who've watched Bush's buddies flick the switches across the globe. In Brazil, Houston Industries seized ownership of Rio de Janeiro's electric company. The Texans (aided by their French partners) fired workers, raised prices, cut maintenance expenditures and, CLICK! the juice went out so often te locals now call it, "Rio Dark."

So too the free-market British buckaroos controlling Niagara Mohawk raised prices, slashed staff, cut maintenance and CLICK! -- New York joins Brazil in the Dark Ages.

The short of it? Bush and cronies deregulated the energy industry (and continuing even more this week) and the industry responded just as any greedy entity would - raise prices, fire workers, slack off on maintenance and pocket the savings.

FDR enacted regulation of certain industries to insure that consumers would not get ripped-off. Bush reversed these regulations possibly because he doesn't know history, and/or he and/or friends/relatives have stock in these industries, or some other overlooked reason. Bush and his administration heavily contributed to the power outage, and is making sure that plenty more are bound to happen. Remember this on the next election day.

SCADA Definition

[pjkundert]

For those of you that might be interested, SCADA is an acronym for

Supervisory Control And Data Acquisition

It defines nothing about whether or not COM, DCOM, OPC (Ole for Process Control), or any other proprietary communication framework is used (contrary to some other highly moderated statements you are likely to see in this discussion)

The bulk of serious SCADA systems in place are probably legacy systems of some kind, including many variants of UNIX systems (we have old micro VAX systems still in use). Many of the newer systems are Windows based, and are obviously subject the standard Windows viri, worms, etc.

Worse yet, these systems are very difficult to upgrade or patch, due to the critical nature of their duties. It is not unlikely that a large portion of the Windows based SCADA systems in the world remain unpatched, and are "safely" firewalled off from the internet.

Of course, the problem is that much of the monitoring gear used to diagnose network issues is also Windows based, and carries with it the standard retinue of Windows viruses and worms, right into the heart of the control center.

These UNIX systems have run (and will contiue to run) uninterrupted for years at a time. We have calendar alerts in place to tell us to go manually reboot unix machines after months of uptime, just to ensure that their SCSI drives will spin back up (in case of a control center power outage, etc.)

Somehow, I don't think that is an issue for some vendor's SCADA systems based on more popular OS's, but I might be wrong...

Re:SCADA systems are *NOT ALWAYS* DCOM based

[elbarsal]

OK, before everybody gets carried away with what SCADA systems are based on, let me point out - SCADA stands for "Supervisory Control and Data Acquisition". No single company makes SCADA any more than any single company makes any other type of application software.

SCADA systems exist that will run on QNX, Linux (See VSystem), Windows NT/2000, DOS (Yes, DOS - i.e. FactoryLink with DesqView) as well as other older platforms like VAX/VMS and PDP-11.

So, SCADA systems do not necessarily rely on OPC or DCOM - some do, some don't. Some still use DDE (trust me, not the most reliable means of communication).

Industry uses all sorts of SCADA systems, from several vendors - Rockwell Software makes RSView, Wonderware makes InTouch, Siemens makes WinCC, you get the picture. The GE package involved in the systems at First Energy is only one example of SCADA, in the same way that EMACS is only one example of a text editor.

Sorry for the rant, but IAAAE (I am an automation engineer).

It's deregulation

[laird]

"It's all about early adopter. We invested huge amounts of money to wire up the whole country before nuclear was an option, and there's no way we can spend that much again to rip it all out and replace it with newer/better/safer alternatives. Same thing with our telephone system, cable tv, internet, etc. We're trying to squeeze every last ounce of usefulness out of the existing system, while so-called "3rd world" countries are getting the latest and greatest tech because they have no prior investments."

I disagree completely -- until the US deregulated, we had an extremely reliable power system that was able to expand dramatically for many decades to keep up with demand while keeping prices low. Then some lunatics convinced enough people that it was a good idea to make power companies unregulated monopolies (i.e. they were no longer required to invest in maintenance and infrastructure, or to maintain excess capacity to cover emergencies or power spikes, and removed the cap on profit margin), after which the power companies did what you'd expect -- raised rates like crazy while slashing spending on infrastructure maintenance and expansion. The result is that a bunch of investors and CEO's made tons of money while the capacity and quality of service that they provided suffered. The problem was not just the big outage -- the number of instabilities in the power grid that didn't result in actual outages has been increasing dramatiaclly for decades.

See http://www.ncpa.org/iss/ene/2003/pd081503a.html for some more info.

Who do I blame? The politicians who allowed themselves to be bought off (or conned) by the power companies, to the detriment of the entire country.

Cleveland eye witness to FirstEnergy

[peptidbond]

Well, I do not post much, but I thought this was worth it.

The day of the black out, I was heading home to my house in Cleveland. Naturally, we were hit first. It was beautiful that night because my neighbor had his 12" telescope out in his yard.

However, what is really intriguing is what happened over the next week. Everyday, driving to and from work I would notice all of these FirstEnergy trucks out on the street. No, they were not restoring power. THEY WERE TRIMMING TREES.

The NERC report states that FirstEnergy failed to trim trees to protect the grid. FirstEnergy knew that they had caused the problem and they tried to cover it up.

Also keep in mind that FirstEnergy owns the troubled Davis-Bessie Nuclear plant in Sandusky, Ohio. The nuke plant was shut down almost 2 years ago because boric acid had eaten a football sized hole in the reactor lid. It has been repaired but not re-started due to "safety culture concerns".

Finally, my G/F is a field biologist. She is constantly doing field surveys for FirstEnergy. She always speaks of their poor land use and inability to maintain their lines. She has even seen over-heated lines drooping almost to the ground.