Slashdot Mirror


NERC Releases Interim Report on Aug 14th Blackout

will writes "The North American Electric Reliability Council has released four documents concerning the August 14th power outage power outage in the North East. The blackout investigation homepage lists all NERC's documents relating to this event. Press coverage is at The Washington Post, CNN, and CBS News. The take home message: FirstEnergy did it. The are, of course, denying it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."

13 of 426 comments (clear)

  1. X Windows? by loconet · · Score: 1, Informative

    X Window not X Windows

    --
    [alk]
    1. Re:X Windows? by Anonymous Coward · · Score: 1, Informative

      #if defined PEDANTIC_ASSHOLE

      Either 'X' or 'X Windowing System'.

      #endif /* PENDANTIC_ASSHOLE */

      And just so I can be 'labeled' a troll, anyone who calls the parent informative is a fuck-tard.

  2. The XA/21 Did Not Break Down by Anonymous Coward · · Score: 3, Informative

    GE's XA/21 system "controls generation and the high voltage transmission network for optimal generation and transmission of power," as provided on GE's corporate web site:

    http://www.gepower.com/prod_serv/products/scada_ so ftware/en/xa21.htm

    On page 99 of the report the task force states specifically that,"although there were a number of worms and viruses impacting theInternet and Internet-connected systems and networks in North America before and during the outage, the SWG's preliminary analysis provides no indication that worm/virus activity had a significant effect on the power generation and delivery systems."In other words, the power generation and delivery systems (GE's XA/21) running Unix were not affected. SCADA, the alarm monitoring system, however, was affected. SCADA failedto transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack. The net result is that the operators were not aware of serious trouble with the grid before it was to late.

    1. Re:The XA/21 Did Not Break Down by Anonymous Coward · · Score: 1, Informative

      As quoted in the Yahoo! news wire here:

      "Not only did the software that controls audible and visual alarms stopworking at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too. More than an hour would pass after that before anyone in FirstEnergy's control room would realize that the software -- General Electric's GE Harris XA21 system --wasn't working."

      You know why the XA/21 "wasn't working?" Here's why:

      A close inspection of the time lines presented on pages 22 and 23 of the DOE report reveals that the Emergency Management System alarm functions (SCADA) first fails at 14:14 EDT. These alarm monitoring systems are the computers running on the Windows Servers.

      14:20 EDT: FirstElectric loses half their remote consoles as the wormspreads.

      14:41 EDT: the Primary EMS server (the Windows Primary Domain Controllerrunning SADA) fails.14:54 EDT: the backup EMS server (the Backup Domain Controller) serverfails.

      15:08 EDT: FE primary EMS Server Restarts.

      15:46 - 15:59 EDT: FE IT rebooting H4,H1 (GE XA/21)

      Take a look from the same article:

      "FirstEnergy's information technology staff belatedly became aware ofthe problem and tried to fix it. In fact, they thought they had shortly after 3 p.m. -- at which point two of the three power lines had already tripped. "Just as you may have to reboot your desktop computer, which often curescomputing problems, that's what they did," said Jimmy Glotfelty, amember of the government task force that investigated the blackout." They had the indication that they fixed the alarm problem, but that was erroneous."

      "They did not adequately test the system after the reboot to be surethey had fixed it," said David Meyer, another task force member."

      So it appears that after the Primary EMS server restarted at 15:08, they looked at the SCADA software on the system and found that it seemed to be working correctly, that is, the PDC system running SCADA came up and communicated with the XA/21 system shortly before the DDOS attacked the PDC all over again. Given that I have seen a down PDC freeze a BDC (usually while the BDC is updating itself with the primary server) I think they said, "we'll worry about the BDC later, we have bigger fish to fry."

      So what do they do? It appears that they rebooted the XA/21! Here we have a DDOS on the NT alarm monitoring server and they decide to bounce the XA/21! Worse, it apears the technicians rebooted the XA/21 without even so much as performing an analysis on the XA/21 or--and this is a fact as well as a big"no-no"--telling the control staff monitoring the grid that they were doing it.

  3. I call bullsh**... by mobiux · · Score: 2, Informative

    Actually the problem was the DE-regulation of the power grid, ie, the ability to use "backup" power lines for the wholesale selling and transfer of power.

    These emergency lines were never meant for this, and since the companies that were buying and selling power didn't own the lines, they had no motivation to upgrade the capacity.

    This article explains alot.

  4. Re:Disaster again by Anonymous Coward · · Score: 1, Informative

    Actually, they were SGI systems running Irix, intermixed with Powermacs.

    The GUI they used is available for download, I believe it was a /. article about a year ago... It was a tried-and-failed approach to 3D filesystem navigation.

    The 'joke' interface for the magic word appeared to be running on MacOS, I would guess 7.

  5. SCADA systems are DCOM based by stock · · Score: 1, Informative

    "According to the product specs, it is a Unix system with X Windows."

    Thats a outcry lie :
    http://www.automationtechies.com/sitepages/pid6 41. php

    SCADA are OPC (OLE for Process Control) based.

    So the story is a false report. Microsoft is again the company to blaim.

    Robert

  6. First Energy is a horrible corporation by puppetluva · · Score: 4, Informative

    Here's a timely link. . .

    FirstEnergy used to be CEI. . . a horrible company in Cleveland that allowed a nuclear core failure through negligence and then tried to bully the government of Cleveland into selling its municipal power supply to it thereby granting it monopoly status in the region.

    Check out the story here:
    http://kucinich.us/powertothepeople.htm

    The presidential candidate Dennis Kucinich was Mayor of Cleveland at the time and endured a browbeating when he refused to sell the Municipal power company to these scoundrels.

    I'm not necessarily pro-Kucinich, but I am VERY anti-FirstEnergy. They have a track record of irresponsibility and dishonesty that should be enough to have the government dismantle them.

  7. Population density viz of the Eastern USA by DAldredge · · Score: 2, Informative

    http://oceanexplorer.noaa.gov/explorations/deepeas t01/background/dumping/media/dumping2.html

  8. Re:SCADA systems are *NOT ALWAYS* DCOM based by elbarsal · · Score: 4, Informative

    OK, before everybody gets carried away with what SCADA systems are based on, let me point out - SCADA stands for "Supervisory Control and Data Acquisition". No single company makes SCADA any more than any single company makes any other type of application software.

    SCADA systems exist that will run on QNX, Linux (See VSystem), Windows NT/2000, DOS (Yes, DOS - i.e. FactoryLink with DesqView) as well as other older platforms like VAX/VMS and PDP-11.

    So, SCADA systems do not necessarily rely on OPC or DCOM - some do, some don't. Some still use DDE (trust me, not the most reliable means of communication).

    Industry uses all sorts of SCADA systems, from several vendors - Rockwell Software makes RSView, Wonderware makes InTouch, Siemens makes WinCC, you get the picture. The GE package involved in the systems at First Energy is only one example of SCADA, in the same way that EMACS is only one example of a text editor.

    Sorry for the rant, but IAAAE (I am an automation engineer).

  9. Re:You got it in reverse by HomerJayS · · Score: 2, Informative
    Actually, California never had a power generation shortage

    Yes, indeed there was a generation shortage. CA de-reg worked just fine until 2000-2001. A heat wave, and below average hydro power output (drought) contributed. Several new power plants have come online since this time to alleviate the shortages.

    I agree that the price manipulation was also a major contributing factor. Utilities could not afford to purchase all the power they needed at 'market' rates (the rates they charged the customers were fixed by the CPUC). All very much legal, BTW.

    All in all a classic case in how government enforced price controls don't work.

  10. Re:It's deregulation by Orne · · Score: 2, Informative

    And to this, I disagree completely. Until the US deregulated, we had a vertically-integrated system that was completely comfortable at over-charging the end users (we people in residential load) when the true price of electricity was much lower. Did you know the average locational price of energy in PA last year was $0.027/kWh on the bulk market? How much are you paying per hour to your local utility?

    Deregulation simply gave the utilities the opportunity to operate in three parts: Transmission, Generation and Load Serving Entities. Thinking they could make a quick buck, some utilities sold their plants to the highest bidder... nothing wrong with that. But the governments wouldn't let anyone build new generators, so there's a fixed supply of energy with rising demand. Then, a vast majority of local governments went and capped the price which the LSE could pass on to their customers, and freed the prices the Generators could charge. So what happened? The governement regulations drove the LSEs out of business (price of supply >> price recouped), and the result is crashed companies and billions in debt passed on to the residents.

    However, Controllers like PJM, NYISO and NEISO adopted Locational Pricing, which did a great job in pinpointing where new generation was needed by raising prices in that area. What do we see today? Lower spot market prices than ever before. An over-capacity glut, where the market made it so desireable we'll have excess power in the northeast for another 4 years.

    I blame the politicians out west who did a half-assed job in only deregulating half the industry, because the parts they regulated caused all the problems. Funny thing, the areas of the country that tried to hang on to the policies of their vertical monopolies (California ISO, First Energy) were the ones that faulted in the deregulated system. This should be a warning to the Dominion's, TVAs and the Southern companies who lag behind...

  11. Re:Mod parent down... by pVoid · · Score: 2, Informative
    Well, the real issue, as I understand it, and as everyone seems to want to obfuscate, is the Unix system *did* in fact malfunction, and that the warnings were not seen *because* of the worms taking down the Windows servers.

    If anyone has some tangible evidence to the contrary, I would sure like to hear it/read the quote... As far as reading this blog goes though, I'm not too interested in hearing people quote one line, and make a completely orthogonal conclusion.

    That's what I was saying... The grandparent post used the crucial "in other words" construct which, contrary to appearances, is actually just a guess (even if it is educated). I'm not defending microsoft here. I just want to make that crystal clear.