Slashdot Mirror

← Back to Stories (view on slashdot.org)

NERC Releases Interim Report on Aug 14th Blackout

Posted by michael on from the candlelight-vigil dept.

will writes "The North American Electric Reliability Council has released four documents concerning the August 14th power outage power outage in the North East. The blackout investigation homepage lists all NERC's documents relating to this event. Press coverage is at The Washington Post, CNN, and CBS News. The take home message: FirstEnergy did it. The are, of course, denying it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."

46 of 426 comments (clear)

  1. Go get 'em! by grub · · Score: 5, Funny


    "According to the product specs, it is a Unix system with X Windows."

    Good, let's sue SCO!


    ps: f1st pr05t

    --
    Trolling is a art,
    1. Re:Go get 'em! by orthogonal · · Score: 4, Funny

      'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows.

      Darl McBride had repossessed the system, leaving a bill for $699 and a subpoena in its place.

      --
      Opinions on the Twiddler2 hand-held keyboard?
    2. Re:Go get 'em! by tekspot · · Score: 5, Funny

      "According to the product specs, it is a Unix system with X Windows."

      The reason they use X is because their systems are managed by MCSEs.

  2. Oh boy, here it comes by Stonent1 · · Score: 2, Funny

    it is a Unix system with X Windows

    Send in..... the Trolls.

  3. Are they into warez? by Anonymous Coward · · Score: 2, Funny
    http://www.nerc.com/~filez/

    just wondering if their webmasters are into the warez business...

  4. Flame bait by gnuLNX · · Score: 2, Funny

    My god was the unix comment WITH X WINDOWS really necassary. ok so I will bite. SHoulda been a linux computer then it would not have crashed. HA!

    --
    what?
  5. Microsoft sucks by Meor · · Score: 5, Funny

    [Slashdot]
    You shouldn't use MS products for such a critical system.

    What? It was a unix system?

    Must have been a hardware failure then.
    [/slashdot]

  6. Memo to power company: by SuperBanana · · Score: 4, Funny
    Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.

    Memo to power company:

    Put power-system controlling servers on UPSes :-)

    (yeah yeah, I know, it wasn't because they lost power. Its a joke :-)

    --
    Please help metamoderate.
  7. Now we know... by zeux · · Score: 4, Insightful

    I think it's really hard to perform that kind of investigation and looking at the results I think they did a good job making the results widely available. Anyway, it's still unacceptable that such things happens in the most powerful country in the world.

    Flame me if you want but France is known to have one of the best electrical system in the world. The government owns it and some laws do protect the costumer. It allows France to export a lot of power to other European countries thus making a lot of money from it and playing a big role in Europ energy.

    I think France made good choices for power:
    - nuclear (or nuculear whatever). Much cheaper and cleaner than oil. France is very strong on nuclear technology and do export its knowledge to other countries. To date we didn't have any major incident and I think it's pretty secure as long as enough money is spent.
    - public service. There is a law in France stating that every citizen, wherever he lives, as the right to have access to electricity for free. Of course we pay bills but if you live in the country the government MUST bring you access to electricity even if you live far from everything. It's a law so sometimes it's not really followed but most of the time the government sticks to it.

    When I arrived in the USA I was shocked by the poor quality of your electric system. Many outages, expensive bills and thousand of wires in the sky ! I think that it's a very advanced country with an outdated power system. I've seen on the TV recently that many companies as starting to produce their own power. I really think it's pretty bad since the installations needed are dangerous, potentially explosive and very expensive.

    What happened on August the 14th also suggests that a country really depends on its energy distribution system and that it could be, in the near future, target for terrorists.

    --
    Iraq: war to save the U
    1. Re:Now we know... by (startx) · · Score: 4, Insightful

      It's all about early adopter. We invested huge amounts of money to wire up the whole country before nuclear was an option, and there's no way we can spend that much again to rip it all out and replace it with newer/better/safer alternatives. Same thing with our telephone system, cable tv, internet, etc. We're trying to squeeze every last ounce of usefulness out of the existing system, while so-called "3rd world" countries are getting the latest and greatest tech because they have no prior investments.

    2. Re:Now we know... by JoeBuck · · Score: 3, Interesting

      The biggest difference is that France puts their power lines underground. Here in Silicon Valley, some French colleagues were shocked at the frequent power disruptions during the winter. Well, of course the power was out, the locals tell them, there was a big storm! (The "big storm" consists of an inch of rain and a short period of 40 mph winds, which knocks down all the power lines).

    3. Re:Now we know... by Ktulu_03 · · Score: 2, Insightful

      Both sides are screwing us, in my opinion. The right needs to realize that we must plan for the future to minimize our dependence on oil (especially foreign sources). the left needs to realize that some short-term solutions (more nuclear plants, drilling in alaska) need to take place until the longer-term solutions are in place.

      I also feel that we should use some solar in a reasonable fashion. especially in cities where they are consistently hotter, because of all the concrete. Out in the burbs where I am, I've seen a house that has some PVC pipes and a solar cell on their garage roof, must be some solar water heating system. Every little bit helps.

    4. Re:Now we know... by JayBlalock · · Score: 2, Interesting
      Simple scientific method-approach to a problem. A Problem exists. If you cannot logically derive the solution beforehand, you try Solution A. If, given adequate time, Solution A fails to solve the problem, you move on to Solution B.

      Our power grid has not been upgraded significantly in decades. The power companies have shown quite well that, without direct regulation, they are not going to willingly do so. (especially as the costs of upgrading go up every year technology advances and the grid goes up) Ergo, at this point it's time to try a new solution.

      I'm not talking about centrally socializing the entire grid or something. In a country our size, that would be impractical (as opposed to someplace much smaller like France). But the fact remains, it is a gigantic problem for the country - both in terms of security AND the sheer common sense social benefits to having a stable power grid. And it's getting worse every year thanks to entropy.

      My proposal would be along the lines of: Form an INDEPENDENT committee. (IE no industry folks acting as anything but advisors) It decides what needs to be upgraded for us to have a power grid roughly on par with Europe's. It gives the power companies adequate time (a decade? Years anyway) to achieve this. And it gives them some money to do so. (as I pointed out in another post, since this is very much a matter of national security, I can see no reason why we can't take a billion or so normally meant for building bombs and use it for the power grid instead)

      Canada would undoubtedly help us out in this, as our grids are (we know now) so closely tied together.

      And at the end of that time limit, if the power companies have failed to upgrade themselves, THEN we start talking about getting draconian and collectivizing everything - since at that point the power companies would have well proven beyond reasonable doubt they are putting their own financial profits ahead of *national security.* Such an entity would have no right to exist within the US, as far as I'm concerned. (or at least not to own and control such an important chunk of our infrastructure)

      (and sadly, I can easily see them pocketing the money and then doing nothing. Just look at how much Verizon has stolen from various states with promises of fiber-optic upgrades that never happened)

      --
      Bush: He's Liberal in all the wrong ways.
    5. Re:Now we know... by Zathrus · · Score: 2, Insightful

      One A/C system where I live but no heating devices. In France I had no A/C but a heating device. Quite the same.

      Uh, no. Heating is considerably cheaper to do than cooling, even if you do both by electrical means. In most areas of the world where heating is a concern you don't do it electrically -- because burning gas is much more effective. It's also considerably easier to insulate against cold than it is against heat.

      Since I arrived 6 months ago: none in Florida.

      And exactly where do you live in Florida? Most of it has daily thunderstorms. Florida is the lightning strike capital of the world.

      Many places in America still don't have any access to the powergrid. Ok the country is VERY BIG, but hey do you REALLY think it's normal ?

      Yes, I do. Do you realize that France is only a bit bigger than twice the size of Colorado? Or that the US has nearly as many square kilometers of water as France has square kilometers of land? The US has to provide power to roughly 5x the number of people on roughly 20x the amount of land.

      Take a look at Russia, the Ukraine, China, India, or any other large, geographically diverse country. How much of it is wired? Frankly, the US probably has more wired than any other country. Heck, if you want a fun comparison, go look at Canada -- they have communities off the grid too. Sure, they're way the heck up North, but that's where most of the unpowered commnities are in the US too (largely in Alaska, some in the Rockies -- good luck finding anyone off the grid not by their own choice in the Northeast, which is a much more accurate comparison to France).

      France doesn't have to deal with a lot of the issues that the US does. Like ants that will happily chew through electrical insulation (fire ants in the SE US; between that and clay soil it makes burying cable extremely expensive)? No permafrost either. Tornados? Hurricanes? Nope.

      Tell me what to envy to such a system in such an advanced country

      I challenge you to show me another country that has anywhere close to the same issues and is doing better than the US. We could be doing better on fossil fuels (particularly oil -- we have absurd reserves of coal and natural gas; but even for oil we only import roughly twice as much as we produce), and I would certainly like to see the US move back toward nuclear power. But we have some of the cheapest power in the world, and the cost of power is a baseline for everything else in the economy. It's one of the key reasons that we have such a strong economy, and have had one since the advent of industrialization.

    6. Re:Now we know... by zeux · · Score: 2, Insightful
      First of all, thanks for being kind in your answer, it's worth the read and it's not a flame like some other answers to my topic.
      It's also considerably easier to insulate against cold than it is against heat.

      But wait a minute, isn't it the same problem ? When you need to cool the place you need to prevent cold air to go out. When you need heating you need to prevent cold air to go in, right ? I hardly see the difference. I'm ok that it's harder to cool than to heat but are you sure it's a so big difference ?

      Look at that. Do you think the difference between heating and cooling does explain that ?

      The US has to provide power to roughly 5x the number of people on roughly 20x the amount of land.

      Yup, but there are some big spaces uninhabited like in Texas for instance. I'm looking for statistics about percentage of people living in cities because I think France has much more small villages isolated than US. I'll try to find that kind of statistics and post it.

      Take a look at Russia, the Ukraine, China, India, or any other large, geographically diverse country. How much of it is wired?

      Unlike USA, those countries can't really afford it...

      France doesn't have to deal with a lot of the issues that the US does. Like ants that will happily chew through electrical insulation (fire ants in the SE US; between that and clay soil it makes burying cable extremely expensive)? No permafrost either. Tornados? Hurricanes? Nope.

      I didn't know about the ants. I've read a book where the author explains that ants rule the world... BTW, we have some strong thunderstorm in France and some flooding too but our electricity grid uses redundant connections much more than the American power grid so it's rare that we have a problem.
      But we have some of the cheapest power in the world, and the cost of power is a baseline for everything else in the economy. It's one of the key reasons that we have such a strong economy, and have had one since the advent of industrialization.
      I agree on this but what's the point to have cheap power if it fails ? I mean I think you will have more and more problems in the future if you don't upgrade your power grid.

      Thanks
      --
      Iraq: war to save the U
    7. Re:Now we know... by coupland · · Score: 2, Funny

      Anyway, it's still unacceptable that such things happens in the most powerful country in the world.

      What, Canada? Don't say such things, we're really very shy.

      The government owns it and some laws do protect the costumer.

      Wow that's some country, laws to protect costumers no less. What about mimes and jugglers?

      When I arrived in the USA I was shocked by the poor quality of your electric system

      Well, you're definitely French, I'll grant ya that.

  8. Disaster again by Camel+Pilot · · Score: 4, Funny

    If I remember correctly wasn't a "Unix system" also implicated in the chaotic havoc that was Jurassic Park?

    1. Re:Disaster again by Enry · · Score: 2, Funny

      No, just bad writing. Ugh.

    2. Re:Disaster again by mcc · · Score: 2, Funny

      Yes, but the Unix systems were running the now-widely-discredited "AH AH AH YOU DIDN'T SAY THE MAGIC WORD" security system product.

      It was a third-party-application issue, not an OS issue. Unix itself is not to blame.

      --
      Irritable, left-wing and possibly humorous bumper stickers and t-shirts
  9. Report vindicates First Energy by gordguide · · Score: 2, Funny

    In a press release First Energy spokesmen said that the report "totally vindicates us" and the blame "clearly lies with the uncontrolled and inherently dangerous actions of something called 'trees'. We are calling on Government Agencies to immediately ban the use, propegation, and distribution" of all trees, which "apparently reproduce unaided" by some "poorly understood virus-like activity".

    From CNN Money's link:
    " ... He noted that Ohio-based FirstEnergy (FE: Research, Estimates)'s failure to trim trees growing alongside major transmission lines contributed to the massive blackout. ..."

    1. Re:Report vindicates First Energy by Melantha_Bacchae · · Score: 2, Interesting

      Yep, Ohio's infamous Hole-In-The-Head reactor was First Energy's baby. Enough primary coolant (the stuff that cools the rods and glows a pretty blue around them from the radiation) was leaking out the top of the reactor head to eat a hole big enough to put a gallon milk jug through. The only thing between America and Chernobyl was a one-sixteenth inch thick stainless steel plate on the outside. Enough coolant did make its way to the outside that the top had to be wiped off periodically. This went on for six years, with several bungled inspections failing to catch it, and a cover up to attempt to let First Energy to continue running the thing. First Energy was still trying to get the government to let them patch and restart it when the blackout happened.

      Funny thing about the blackout: nuclear plants and dirty fossil fuel plants were affected. Clean plants, like the Niagara Falls hydroelectric plant, blinked, then went right on running.

      Another funny thing about the blackout: five days before the lights went out, "Godzilla X Megagiras" had had its US premiere at the Egyptian Theatre. In this movie, Godzilla's attacks on nuclear plants cause Japan to ban all nuclear energy and look to clean energy. The bad guy keeps building fusion plants (fobbing them off as "clean energy"), but they always leak radiation, and Godzilla always finds them and knocks their lights out.

      You might want to stock up on flashlights. "Godzilla X Megagiras" is coming out on DVD in the US January 6th.

      "Is Godzilla showing his hatred toward man-made energy?"
      Shinoda, "Godzilla 2000 Millennium" (Japanese version)

  10. The XA/21 Did Not Break Down by Anonymous Coward · · Score: 3, Informative

    GE's XA/21 system "controls generation and the high voltage transmission network for optimal generation and transmission of power," as provided on GE's corporate web site:

    http://www.gepower.com/prod_serv/products/scada_ so ftware/en/xa21.htm

    On page 99 of the report the task force states specifically that,"although there were a number of worms and viruses impacting theInternet and Internet-connected systems and networks in North America before and during the outage, the SWG's preliminary analysis provides no indication that worm/virus activity had a significant effect on the power generation and delivery systems."In other words, the power generation and delivery systems (GE's XA/21) running Unix were not affected. SCADA, the alarm monitoring system, however, was affected. SCADA failedto transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack. The net result is that the operators were not aware of serious trouble with the grid before it was to late.

    1. Re:The XA/21 Did Not Break Down by Troed · · Score: 2, Interesting

      Mod parent up. Windows/Blaster WAS one of the causes of the blackout, something that has been known for quite some time. Nice to see official confirmation. The submitter has taken too much SCO crack me thinks.

      SCADA failed to transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack.

      --
      it's in my head
  11. No, Sue the End User (or at least the analyst)! by FreeUser · · Score: 4, Interesting
    Good, let's sue SCO!

    heh! :-)

    Alas, as Microsoft apologists are wont to point out (even in the many cases where the crash or security flaw doesn't stem from mismanagement or configuration errors), a misconfigured system ins't going to work regardless of what OS it is running.

    In this particular case, quoth the article

    One of MISO's monitoring systems required technical repairs that afternoon, but the technician who fixed it forgot to turn on an automatic feature that updated information every five minutes, preventing it from operating normally, the report said. "Thinking the system had been successfully restored, the analyst went to lunch."


    The technician forgot to restart the monitoring software. Oops.

    Following in $CO's illustrious footsteps, I think perhaps it is time we sued the poor schmuck who forgot to restart the monitoring program. Or better yet, the company dumb enough to hire him, the electrical company. After all, according to Our Lord and Master Darl McBride, End Users should pay (and pay heavilly).
    --
    The Future of Human Evolution: Autonomy
    1. Re:No, Sue the End User (or at least the analyst)! by MrResistor · · Score: 3, Interesting

      I can't help but think, if the process has to be started by someone, it isn't automatic, is it? And if it's so damned important, why wasn't it started automatically?

      Seems to me the stupidity predates the technician.

      --
      Under capitalism man exploits man. Under communism it's the other way around.
  12. I call bullsh**... by mobiux · · Score: 2, Informative

    Actually the problem was the DE-regulation of the power grid, ie, the ability to use "backup" power lines for the wholesale selling and transfer of power.

    These emergency lines were never meant for this, and since the companies that were buying and selling power didn't own the lines, they had no motivation to upgrade the capacity.

    This article explains alot.

  13. What caused the boxes to crash? by AJWM · · Score: 2, Interesting

    It may have been a Unix box that crashed, but I've seen and heard of cases where a 'nix box crashed because of the high network load of a Windows-based worm/virus epidemic. Was this the case here?

    (There should have been better firewalling in place if so, of course.)

    --
    -- Alastair
    1. Re:What caused the boxes to crash? by AJWM · · Score: 2, Insightful

      Do we really care how the box crashed?

      If we don't want it to happen again, yes we do. It was more than one box, which points to something more systemic than a random hardware glitch or failure.

      --
      -- Alastair
  14. You got it in reverse by just+fiddling+around · · Score: 5, Insightful

    FYI, US governement have been engaged in a nice deregulation wave for 10 years. That deregulation has been poorly tought out, because of a "The Market Cures All" mindset. What did the deregulation bring? Gigantic power outages and California-style power crunches.

    The deregulated US power grid is overstressed, has little or no margin for extra demand, and has outdated and poorly maintained emergency systems. And thanks to Enron and friends, artificial scarceness has been created to crank up prices.

    In a nutshell, you pay more and get less(safety, avaliability, quality) than when the market was regulated. Ain't The Market grand?

    Electric power is a basic necessity(in our post-industrial world), and should be managed by the state, just like water works and law enforcement.

    --
    You're not old until regret takes the place of your dreams.
    1. Re:You got it in reverse by Anonymous Coward · · Score: 2, Insightful

      Look at Texas for a model in deregulation. Their energy prices are low, and they have a 20% surplus on even the hottest days in summer. Texas, BTW, has its own power grid separate from the other main grids in the US.

      Deregulation can work, but as you seem to imply, only if it is properly thought out. This is a very rare occurance in politics and government.

    2. Re:You got it in reverse by HomerJayS · · Score: 2, Informative
      Actually, California never had a power generation shortage

      Yes, indeed there was a generation shortage. CA de-reg worked just fine until 2000-2001. A heat wave, and below average hydro power output (drought) contributed. Several new power plants have come online since this time to alleviate the shortages.

      I agree that the price manipulation was also a major contributing factor. Utilities could not afford to purchase all the power they needed at 'market' rates (the rates they charged the customers were fixed by the CPUC). All very much legal, BTW.

      All in all a classic case in how government enforced price controls don't work.

    3. Re:You got it in reverse by Vainglorious+Coward · · Score: 3, Insightful
      Electric power is a basic necessity...

      Absolutely, and it's interesting to note that planned US spending on electrical infrastructure to 2005 is 71 cents per American per year. Meanwhile, the US gov. plans to spend $255 per citizen in Iraq! (Health spending plans are also illuminating : $38 per capita on hospitals in Iraq versus $3.30 in the US).

      If I was a US citizen, I would be furious about this failure to invest my tax dollars in my own country's infrastructure.

      --
      My next sig will be ready soon, but subscribers can beat the rush
  15. Power out again @ FirstEnergyCorp by kidgenius · · Score: 2, Funny
    Well, I clicked on the link to FirstEnergyCorp and I got a 500 Error.

    Seems that their server went down. I guess this means the northeast will be experiencing another blackout.

    Blackouts by ./ Must be a first

  16. Who cares about the OS or the provider? by swordgeek · · Score: 4, Insightful

    First of all, let's be perfectly clear on this.

    Any carefully managed OS (inluding Windows) can be stable and predictable.

    Any badly managed OS (including Unix) WILL be unstable and unpredictable.

    Got it? Good.

    Now as far as pointing fingers at the guilty parties, understand that the infrastructure is really at fault far more than any individual company. Look at this from a broader perspective: One company was able to take down how wide of an area? The whole system is too fragile, too interdependent, and maintained too close to full capacity. Worse, there's absolutely no incentive for a company to maintain a large reserve capacity, since it costs a lot, and brings in no extra income.

    But of course when it happens the next time (and make no mistake--it will), we'll have another commission to once again figure out the single company that broke a rotten and unstable infrastructure, instead of fixing the root problem.

    --

    "People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
  17. First Energy is a horrible corporation by puppetluva · · Score: 4, Informative

    Here's a timely link. . .

    FirstEnergy used to be CEI. . . a horrible company in Cleveland that allowed a nuclear core failure through negligence and then tried to bully the government of Cleveland into selling its municipal power supply to it thereby granting it monopoly status in the region.

    Check out the story here:
    http://kucinich.us/powertothepeople.htm

    The presidential candidate Dennis Kucinich was Mayor of Cleveland at the time and endured a browbeating when he refused to sell the Municipal power company to these scoundrels.

    I'm not necessarily pro-Kucinich, but I am VERY anti-FirstEnergy. They have a track record of irresponsibility and dishonesty that should be enough to have the government dismantle them.

  18. What were the failed systems running? by mabu · · Score: 2, Interesting

    After browsing all the comments thus far, there doesn't seem to be a definitive answer as to what OS these failed systems were running. Let's find this out... it's also of interest that in the Windows license agreement, it specifically states that Windows shouldn't be used for mission-critical applications involving scenarios like this, so it would be further hilarity if they were running Windows in contrast to the products' own licensing agreement.

  19. Power outage traced to dim bulb in White House by TheRealStyro · · Score: 4, Interesting
    Investigative journalist Greg Palast knows what caused the power outage - deregulation. Here are some excerpts from his article on the subject...
    I can tell you all about the ne're-do-wells that sent us back to the Dark Ages last week. I came up against these characters -- First Energy and the Niagara Mohawk Power Company -- some years back. You see, before I was a journalist, I worked for a living, as an investigator of corporate racketeers.

    The power outage began in First Energy's Ohio operation. This company was the model for the film, "China Syndrome." Really. Then First Energy's Pennsylvania unit fumbled the power ball. These are the very same Homer Simpsons who melted Three Mile Island.

    ...

    Is last week's black-out a surprise? Heck, no, not to us in the field who've watched Bush's buddies flick the switches across the globe. In Brazil, Houston Industries seized ownership of Rio de Janeiro's electric company. The Texans (aided by their French partners) fired workers, raised prices, cut maintenance expenditures and, CLICK! the juice went out so often te locals now call it, "Rio Dark."

    So too the free-market British buckaroos controlling Niagara Mohawk raised prices, slashed staff, cut maintenance and CLICK! -- New York joins Brazil in the Dark Ages.
    The short of it? Bush and cronies deregulated the energy industry (and continuing even more this week) and the industry responded just as any greedy entity would - raise prices, fire workers, slack off on maintenance and pocket the savings.

    FDR enacted regulation of certain industries to insure that consumers would not get ripped-off. Bush reversed these regulations possibly because he doesn't know history, and/or he and/or friends/relatives have stock in these industries, or some other overlooked reason. Bush and his administration heavily contributed to the power outage, and is making sure that plenty more are bound to happen. Remember this on the next election day.
    --
  20. SCADA Definition by pjkundert · · Score: 5, Interesting

    For those of you that might be interested, SCADA is an acronym for

    Supervisory Control And Data Acquisition

    It defines nothing about whether or not COM, DCOM, OPC (Ole for Process Control), or any other proprietary communication framework is used (contrary to some other highly moderated statements you are likely to see in this discussion)

    The bulk of serious SCADA systems in place are probably legacy systems of some kind, including many variants of UNIX systems (we have old micro VAX systems still in use). Many of the newer systems are Windows based, and are obviously subject the standard Windows viri, worms, etc.

    Worse yet, these systems are very difficult to upgrade or patch, due to the critical nature of their duties. It is not unlikely that a large portion of the Windows based SCADA systems in the world remain unpatched, and are "safely" firewalled off from the internet.

    Of course, the problem is that much of the monitoring gear used to diagnose network issues is also Windows based, and carries with it the standard retinue of Windows viruses and worms, right into the heart of the control center.

    These UNIX systems have run (and will contiue to run) uninterrupted for years at a time. We have calendar alerts in place to tell us to go manually reboot unix machines after months of uptime, just to ensure that their SCSI drives will spin back up (in case of a control center power outage, etc.)

    Somehow, I don't think that is an issue for some vendor's SCADA systems based on more popular OS's, but I might be wrong...

    --
    -- -pjk Perry Kundert perry@kundert.ca http://kundert.2y.net
  21. Population density viz of the Eastern USA by DAldredge · · Score: 2, Informative

    http://oceanexplorer.noaa.gov/explorations/deepeas t01/background/dumping/media/dumping2.html

  22. Re:SCADA systems are *NOT ALWAYS* DCOM based by elbarsal · · Score: 4, Informative

    OK, before everybody gets carried away with what SCADA systems are based on, let me point out - SCADA stands for "Supervisory Control and Data Acquisition". No single company makes SCADA any more than any single company makes any other type of application software.

    SCADA systems exist that will run on QNX, Linux (See VSystem), Windows NT/2000, DOS (Yes, DOS - i.e. FactoryLink with DesqView) as well as other older platforms like VAX/VMS and PDP-11.

    So, SCADA systems do not necessarily rely on OPC or DCOM - some do, some don't. Some still use DDE (trust me, not the most reliable means of communication).

    Industry uses all sorts of SCADA systems, from several vendors - Rockwell Software makes RSView, Wonderware makes InTouch, Siemens makes WinCC, you get the picture. The GE package involved in the systems at First Energy is only one example of SCADA, in the same way that EMACS is only one example of a text editor.

    Sorry for the rant, but IAAAE (I am an automation engineer).

  23. It's deregulation by laird · · Score: 4, Insightful

    "It's all about early adopter. We invested huge amounts of money to wire up the whole country before nuclear was an option, and there's no way we can spend that much again to rip it all out and replace it with newer/better/safer alternatives. Same thing with our telephone system, cable tv, internet, etc. We're trying to squeeze every last ounce of usefulness out of the existing system, while so-called "3rd world" countries are getting the latest and greatest tech because they have no prior investments."

    I disagree completely -- until the US deregulated, we had an extremely reliable power system that was able to expand dramatically for many decades to keep up with demand while keeping prices low. Then some lunatics convinced enough people that it was a good idea to make power companies unregulated monopolies (i.e. they were no longer required to invest in maintenance and infrastructure, or to maintain excess capacity to cover emergencies or power spikes, and removed the cap on profit margin), after which the power companies did what you'd expect -- raised rates like crazy while slashing spending on infrastructure maintenance and expansion. The result is that a bunch of investors and CEO's made tons of money while the capacity and quality of service that they provided suffered. The problem was not just the big outage -- the number of instabilities in the power grid that didn't result in actual outages has been increasing dramatiaclly for decades.

    See http://www.ncpa.org/iss/ene/2003/pd081503a.html for some more info.

    Who do I blame? The politicians who allowed themselves to be bought off (or conned) by the power companies, to the detriment of the entire country.

    --
    Enable 3D printed prosthetics!
    1. Re:It's deregulation by Orne · · Score: 2, Informative

      And to this, I disagree completely. Until the US deregulated, we had a vertically-integrated system that was completely comfortable at over-charging the end users (we people in residential load) when the true price of electricity was much lower. Did you know the average locational price of energy in PA last year was $0.027/kWh on the bulk market? How much are you paying per hour to your local utility?

      Deregulation simply gave the utilities the opportunity to operate in three parts: Transmission, Generation and Load Serving Entities. Thinking they could make a quick buck, some utilities sold their plants to the highest bidder... nothing wrong with that. But the governments wouldn't let anyone build new generators, so there's a fixed supply of energy with rising demand. Then, a vast majority of local governments went and capped the price which the LSE could pass on to their customers, and freed the prices the Generators could charge. So what happened? The governement regulations drove the LSEs out of business (price of supply >> price recouped), and the result is crashed companies and billions in debt passed on to the residents.

      However, Controllers like PJM, NYISO and NEISO adopted Locational Pricing, which did a great job in pinpointing where new generation was needed by raising prices in that area. What do we see today? Lower spot market prices than ever before. An over-capacity glut, where the market made it so desireable we'll have excess power in the northeast for another 4 years.

      I blame the politicians out west who did a half-assed job in only deregulating half the industry, because the parts they regulated caused all the problems. Funny thing, the areas of the country that tried to hang on to the policies of their vertical monopolies (California ISO, First Energy) were the ones that faulted in the deregulated system. This should be a warning to the Dominion's, TVAs and the Southern companies who lag behind...

  24. Blame Canada by ablair · · Score: 2, Insightful

    Looks like the headline-grabbers like Pataki and Bloomberg, amongst others, now have little to say about their quick denouncements of Canada for the whole power mess. Interesting that this is the same reaction pattern for the current Canadian internet pharmacy spat, where FDA commissioners are now publicly alleging Canadian drugs to be unsafe. Is 'mouth off first and ask questions later' now an official US political strategy?

  25. Cleveland eye witness to FirstEnergy by peptidbond · · Score: 3, Interesting

    Well, I do not post much, but I thought this was worth it.

    The day of the black out, I was heading home to my house in Cleveland. Naturally, we were hit first. It was beautiful that night because my neighbor had his 12" telescope out in his yard.

    However, what is really intriguing is what happened over the next week. Everyday, driving to and from work I would notice all of these FirstEnergy trucks out on the street. No, they were not restoring power. THEY WERE TRIMMING TREES.

    The NERC report states that FirstEnergy failed to trim trees to protect the grid. FirstEnergy knew that they had caused the problem and they tried to cover it up.

    Also keep in mind that FirstEnergy owns the troubled Davis-Bessie Nuclear plant in Sandusky, Ohio. The nuke plant was shut down almost 2 years ago because boric acid had eaten a football sized hole in the reactor lid. It has been repaired but not re-started due to "safety culture concerns".

    Finally, my G/F is a field biologist. She is constantly doing field surveys for FirstEnergy. She always speaks of their poor land use and inability to maintain their lines. She has even seen over-heated lines drooping almost to the ground.

    --
    peptidbond I was crazy once....
  26. Re:Power outage traced to dim bulb in White House by Lord+Omlette · · Score: 2, Funny

    "but, but, but, 9/11 changed everything!"

    And whoosh! All of the good points Mr. Palast raised no longer apply.

    --
    [o]_O
  27. Re:Mod parent down... by pVoid · · Score: 2, Informative
    Well, the real issue, as I understand it, and as everyone seems to want to obfuscate, is the Unix system *did* in fact malfunction, and that the warnings were not seen *because* of the worms taking down the Windows servers.

    If anyone has some tangible evidence to the contrary, I would sure like to hear it/read the quote... As far as reading this blog goes though, I'm not too interested in hearing people quote one line, and make a completely orthogonal conclusion.

    That's what I was saying... The grandparent post used the crucial "in other words" construct which, contrary to appearances, is actually just a guess (even if it is educated). I'm not defending microsoft here. I just want to make that crystal clear.