Slashdot Mirror
NERC Releases Interim Report on Aug 14th Blackout
will writes "The North American Electric Reliability
Council has released four documents concerning the
August 14th power outage power outage in the North East. The blackout
investigation homepage lists all NERC's documents relating to this
event. Press coverage is at The
Washington Post, CNN,
and CBS
News. The take home message: FirstEnergy
did it. The are, of course, denying
it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."
"According to the product specs, it is a Unix system with X Windows."
Good, let's sue SCO!
ps: f1st pr05t
Trolling is a art,
Funny how all the americans were blaming canada, well guess what? the song doesnt count for _EVERYTHING_!
Can someone tell me what X windows has to do with this story?
SCO.com uses Linux
Can anyone tell me any unix system that can't run X-windows ? I've used a few in my time, and they've all been capable of running X ....
Simon.
Physicists get Hadrons!
it is a Unix system with X Windows
Send in..... the Trolls.
just wondering if their webmasters are into the warez business...
X Window not X Windows
[alk]
"Yeah, since it was a *nix box that failed, it's all right, but if it was a MS box that was at fault you'd all be, like, 'OMG MICROSOFT SUXXORS,' you goddamed zealot hypocrits!"
My god was the unix comment WITH X WINDOWS really necassary. ok so I will bite. SHoulda been a linux computer then it would not have crashed. HA!
what?
See, if they were using MS Windows, it would have crashed LONG ago and we wouldn't be talking about it today. We'd be talking about it years ago. And then again last year. And then again this... uh nevermind.
-Valiss
[Slashdot]
You shouldn't use MS products for such a critical system.
What? It was a unix system?
Must have been a hardware failure then.
[/slashdot]
Memo to power company:
Put power-system controlling servers on UPSes :-)
(yeah yeah, I know, it wasn't because they lost power. Its a joke :-)
Please help metamoderate.
I think it's really hard to perform that kind of investigation and looking at the results I think they did a good job making the results widely available. Anyway, it's still unacceptable that such things happens in the most powerful country in the world.
Flame me if you want but France is known to have one of the best electrical system in the world. The government owns it and some laws do protect the costumer. It allows France to export a lot of power to other European countries thus making a lot of money from it and playing a big role in Europ energy.
I think France made good choices for power:
- nuclear (or nuculear whatever). Much cheaper and cleaner than oil. France is very strong on nuclear technology and do export its knowledge to other countries. To date we didn't have any major incident and I think it's pretty secure as long as enough money is spent.
- public service. There is a law in France stating that every citizen, wherever he lives, as the right to have access to electricity for free. Of course we pay bills but if you live in the country the government MUST bring you access to electricity even if you live far from everything. It's a law so sometimes it's not really followed but most of the time the government sticks to it.
When I arrived in the USA I was shocked by the poor quality of your electric system. Many outages, expensive bills and thousand of wires in the sky ! I think that it's a very advanced country with an outdated power system. I've seen on the TV recently that many companies as starting to produce their own power. I really think it's pretty bad since the installations needed are dangerous, potentially explosive and very expensive.
What happened on August the 14th also suggests that a country really depends on its energy distribution system and that it could be, in the near future, target for terrorists.
Iraq: war to save the U
With UNIX, you get a long power cut once in a few years. With Windows, you'll have a 5 minute power cut to reboot the servers, every 1 hour. You pick
First Energy has been a major concern of Congressman Kucinich for a while now. There is a long standing feud between Kucinich and First Energy-at this point it looks like Dennis was right about these folks.
If I remember correctly wasn't a "Unix system" also implicated in the chaotic havoc that was Jurassic Park?
It was a unix box: "Stupid admin"
It was a microsoft box: "OMG M$ SUXORS!"
According to the product specs, it is a Unix system with X Windows."
I told you to switch to windows. Call 1-800-SELLURSOUL now! Our operators are standing by. Except during their hourly reboot that is.
or X11 or X11R6.
Ah yes, using X Windows, that way, people who don't the time will associate the "failure" with something having to do with "Windows" Yea, start the flames now. IT was a Unix box... thats the all that needed to be said.
A Unix box with X Windows Changes to
A box with X Windows Changes to
A Box with XP Windows or perhaps
just a Box with Windows...
It will get lost in the translation somewhere....
Nice flamebait. The antithesis is actually support by the facts in this case.
In a press release First Energy spokesmen said that the report "totally vindicates us" and the blame "clearly lies with the uncontrolled and inherently dangerous actions of something called 'trees'. We are calling on Government Agencies to immediately ban the use, propegation, and distribution" of all trees, which "apparently reproduce unaided" by some "poorly understood virus-like activity".
... He noted that Ohio-based FirstEnergy (FE: Research, Estimates)'s failure to trim trees growing alongside major transmission lines contributed to the massive blackout. ..."
From CNN Money's link:
"
I can just see good ol Steve and Bill "coming to the rescue" with an NT based solution. The press would love it, "Microsoft replaces aging UNIX system".
Ugh.
Seriously, though, the failure of a SCADA system **SHOULD NOT** bring down the grid. This is just passing the buck.
Click here to enter the GOLDEN BET ONLINE CASINO !
Let's all wget that link at 2:00 pm EST, and see how the spammer's server handles a Slashdoting.
Opinions on the Twiddler2 hand-held keyboard?
According to the product specs, it is a Unix system with X Windows. Some times I think slashdot instigates the flame wars.
It's interesting to read the First Energy "denial." It's not really a denial, more of a "yeah, we had a problem, but hey! look over there!" Honestly, it reads like an attempt to distract, rather than deny. For instance:
"We recognize that our computer system experienced problems that day, which we discussed publicly immediately following the outage."
Is followed a paragraph later by:
"By focusing its analysis on a few selected events, the conclusions the Task Force reached don't address the complexity and magnitude of operations on the interconnected grid."
Translation: yeah, we had a problem, but the fact that our problem could take down the whole grid isn't our fault. Honestly, it sounds like a pretty stupid excuse, but then, that's just my reading of it.
Now we all know why SCO needs bodyguards.
Skipping Econ 101 to post on slashdot again? shame on you!
The same kind of problems happen in systems that are unregulated, because private power companies have no financial motiviation to keep their systems modern, and no motiviation to build the sorts of multiple redundancies that are clearly needed.
Though the govt sucks too. Damned if we do, damned if we dont.
ad logicam Claiming a proposition is false because it was presented as the conclusion of a fallacious argument.
Are these cascading failures becoming the status-quo for America? Is apathy the new buzzword in corporate circles? How is it possible for this domino effect to occur on something as critical as a power grid?! You'd think that systems like these would have a better QoS system in place, especially with the "impending threat of terrorism" that we've been on about lately...
Give it a break will ya...accidents happen. Capitalists stub their toes just as friggin much as socialists do, only when capitalists stub their toes, they blame government for it, and when socialists stub their toes, they blame the lack of government mandates requiring soft fuzzy stripes on the bottom two inches of every wall in the country.
Don't get me wrong, I am definitely much more in favour of less government regulation in all sorts of things, but accidents will still happen, regardless of market forces, and / or lack thereof.
This only happened after government DEREGULATED the power system. Democrats had been trying to pass a bill to force the power systems to be upgraded (because the deregulated companies were too cheap to do so) saying that this was going to happen any day now--and of course, Pres. Bush was told by his corporate advisors that nothing would happen and this was just a pork barrel bill.
no... you know that power companies sell each other electricity. Heck, electric companies will buy electricity from those that produce. Think wind energy and lots of fields.
..
This isn't a case of stupidity of the gov't. It's probably stupidity of not testing their systems periodically for failover. And not the entire grid testing for failover, which would be ideal, but just one company. Or so says the report
-s
-
ping -f 255.255.255.255 # if only
This would not have happened in a market unencumbered by overzealous regulations and socialist mindsets
Then do something about it. Name-calling slashdot readers isn't going to start any sort of political discussion. However, it will make you look like a dumbass.
GE's XA/21 system "controls generation and the high voltage transmission network for optimal generation and transmission of power," as provided on GE's corporate web site:
_ so ftware/en/xa21.htm
http://www.gepower.com/prod_serv/products/scada
On page 99 of the report the task force states specifically that,"although there were a number of worms and viruses impacting theInternet and Internet-connected systems and networks in North America before and during the outage, the SWG's preliminary analysis provides no indication that worm/virus activity had a significant effect on the power generation and delivery systems."In other words, the power generation and delivery systems (GE's XA/21) running Unix were not affected. SCADA, the alarm monitoring system, however, was affected. SCADA failedto transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack. The net result is that the operators were not aware of serious trouble with the grid before it was to late.
What I would want to see here is a country by country and region by region comparison of how power utilities work in various dimensions. Enron for example was a disaster, a corrupt organizations systematically attempting to defraud the public-and Enron as sold originally as a more "free market" solution.
My guess here is that there are good examples of public utilities working--and better market based solutions than Enron.
heh!
Alas, as Microsoft apologists are wont to point out (even in the many cases where the crash or security flaw doesn't stem from mismanagement or configuration errors), a misconfigured system ins't going to work regardless of what OS it is running.
In this particular case, quoth the article
The technician forgot to restart the monitoring software. Oops.
Following in $CO's illustrious footsteps, I think perhaps it is time we sued the poor schmuck who forgot to restart the monitoring program. Or better yet, the company dumb enough to hire him, the electrical company. After all, according to Our Lord and Master Darl McBride, End Users should pay (and pay heavilly).
The Future of Human Evolution: Autonomy
The same kind of problems happen in systems that are unregulated, because private power companies have no financial motiviation
Gotta disagree. People want reliability and they pay for reliability. If there were two power companies, and one of them was known for more frequent blackouts, then I would go with the other one. I chose my website hosting service in part because of their reliability (Pair Networks), and hosting services regularly advertise based on their reliability (up time). That's just an example from a familiar domain; it's generally true that people value reliability.
And therefore companies do have an incentive to provide reliability.
I seem to recall not long after the blackout, there was a report that someone had managed to hack into First Energy's network and do "something" to a SCADA system--reported, of course, with all kinds of assurances that it was pure coincidence and had nothing to do with any failures.
It also behooves us all to keep in mind the many comments and analyses that have pointed out that our current Enronesque energy policy, using interconnects for routine horse-trading of gigawatt-hours, had a lot to do with making a local failure into something much larger. These systems were simply never designed to be used in this manner. At one point, it was reported, tremoundous amounts of energy were running in circles around the Great Lakes, accomplishing nothing more than overloading lines and generating plants and forcing them to cut out.
Sounds like the folks at First Energy indeed didn't have their house in order. Making them the sacrificial lambs for this one incident is an easy way out, and allows the Administration's policy hacks and their good buddies in the industry to go back to business as usual. That is, until the next time the lights go out.
] This is Chapter 7 of the UNIX-HATERS Handbook. The X-Windows Disaster chapter was written by Don Hopkins. [
How to make a 50-MIPS Workstation Run Like a 4.77MHz IBM PC
If the designers of X-Windows built cars, there would be no fewer than five steering wheels hidden about the cockpit, none of which followed the same principles -- but you'd be able to shift gears with your car stereo. Useful feature, that.
- Marus J. Ranum, Digital Equipment Corporation
X-Windows is the Iran-Contra of graphical user interfaces: a tragedy of political compromises, entangled alliances, marketing hype, and just plain greed. X-Windows is to memory as Ronald Reagan was to money. Years of "Voodoo Ergonomics" have resulted in an unprecedented memory deficit of gargantuan proportions. Divisive dependencies, distributed deadlocks, and partisan protocols have tightened gridlocks, aggravated race conditions, and promulgated double standards.
X has had its share of $5,000 toilet seats -- like Sun's Open Look clock tool, which gobbles up 1.4 megabytes of real memory! If you sacrificed all the RAM from 22 Commodore 64s to clock tool, it still wouldn't have enough to tell you the time. Even the vanilla X11R4 "xclock" utility consumed 656K to run. And X's memory usage is increasing.
X: The First Fully Modular Software Disaster
X-Windows started out as one man's project in an office on the fifth floor of MIT's Laboratory for Computer Science. A wizardly hacker, who was familiar with W, a window system written at Stanford University as part of the V project, decided to write a distributed graphical display server. The idea was to allow a program, called a client, to run on one computer and allow it to display on another computer that was running a special program called a window server. The two computers might be VAXes or Suns, or one of each, as long as the computers were networked together and each implemented the X protocol.
[Footnote: We have tried to avoid paragraph-length footnotes in this book, but X has defeated us by switching the meaning of client and server. In all other client/server relationships, the server is the remote machine that runs the application (i.e., the server provides services, such as database service or computational service). For some perverse reason that's better left to the imagination, X insists on calling the program running on the remote machine "the client." This program displays its windows on the "window server." We're going to follow X terminology when discussing graphical client/servers. So when you see "client" think "the remote machine where the application is running," and when you see "Server" think "the local machine that displays output and accepts user input."]
The Nongraphical GUI
X was designed to run three programs: xterm, xload, and xclock. (The idea of a window manager was added as an afterthought, and it shows.) For the first few years of its development at MIT, these were, in fact, the only programs that ran under the window system. Notice that none of these program have any semblance of a graphical user interface (except xclock), only one of these programs implements anything in the way of cut-and-paste (and then, only a single data type is supported), and none of them requires a particularly sophisticated approach to color management. Is it any wonder, then, that these are all areas in which modern X falls down?
Ten years later, most computers running X run just four programs: xterm, xload, xclock, and a window manager. And most xterm windows run Emacs! X has to be the most expensive way ever of popping up an Emacs window. It sure would have been much cheaper and easier to put terminal handling in the kernel where it belongs, rather than forcing people to purchase expensive bitmapped terminals to run character-based applications. On the other hand, then users wouldn't get all of those ugly fonts. It's a trade-off.
The Motif Self-Abuse Kit
X gave Unix vendors something they had professed to want
I'll bet you the whole thing was caused by an incident involving a safety technician, hot grits in strange places, and Natalie Portman.
Repeal the DMCA!
Actually the problem was the DE-regulation of the power grid, ie, the ability to use "backup" power lines for the wholesale selling and transfer of power.
These emergency lines were never meant for this, and since the companies that were buying and selling power didn't own the lines, they had no motivation to upgrade the capacity.
This article explains alot.
See Sep 1 2003:Blaster worm linked to US blackout:
Actually, utility companies have been restricted by the EPA from cutting trees along the pole lines.
The environmentalists say the breaks in the forest inhibit deer from migrating, because they're afraid to cross open space. Anybody who has ever bagged a deer with a car knows about this issue first hand.
Just like the Alaska pipeline scares polar bear and caribou. Hell, they sleep on it for warmth.
For blaming it on us when the real blame should fall on your countries move to self-regulation of the energy industry.
:P
You can also apologize in advance for this
just = (My)Opinion.toCents();
It may have been a Unix box that crashed, but I've seen and heard of cases where a 'nix box crashed because of the high network load of a Windows-based worm/virus epidemic. Was this the case here?
(There should have been better firewalling in place if so, of course.)
-- Alastair
I'm sorry, but this "stubbed toe" is an international afair. It severly impacted Ontario's economy and we are still feeling the brunt of it. Americans should be ashamed of this.
According to GE specifications, the system is comprised of one or more "computers" running an "operating system" and a "graphical user interface" (XWin == GUI nyaaah!)
According to further documents, this computer's "operating system" was running some kind of "software program".
C'mon guys, if it was running Windows and it BSoD'ed, then it might be worth mentioning. Other than that I don't see why we should be pointing out what OS it's running. The failure could have been caused by any piece of hardware, software, sabotage, or some combination thereof.
-73, de n1ywb
www.n1ywb.com
Well, from some of the articles posted:
- 'FirstEnergy's failure to take steps to restore its system to a stable operating condition within a half-hour of one of the major power line outages violated voluntary industry guidelines issued to keep the grid operating safely, the report said. FirstEnergy also failed to provide adequate warning of the impending emergency and provided inadequate training to its operators.
Utilities do not now face penalties for guideline violations, although the energy bill now before Congress would make the guidelines mandatory and impose financial penalties for violations.'
Sounds more like a failure of capitalistic free enterprise to me.
As the air to a bird or the sea to a fish, so is contempt to the contemptible -W.B.
Check out The Industrial Physicist www.tipmagazine.com October/November 2003. The problem is not regulation, it was a stupid form of de-regulation (much like and related to the deregulation that messed up california power systems) that allowed rampant energy trading leading to unpredictable load fluctuations. The grid was not designed for this sort of trading. The grid was "gamed".
See this previous /. article
The case of the "who dun it" really doesn't matter in the end. This could have been any number of power companies in the MidWest. These unfortunate saps are the ones that got slapped first
-woog
the August 14th power outage power outage
FirstEnergy did it. The are, of course, denying it.
Do people really take so little pride in their communications that they no longer check it for grammar and spelling?
I think if you read between the lines about all of the things going on that day, it becomes more and more clear what actually happened. This is my theory, take it with however many grains of salt you wish:
1. The MSBlaster worm was widespread on that day, and was wreaking havoc on both the Internet and on corporate LANs when employees brought infected machines in to work and plugged them in behind the firewall.
2. These Unix systems are old, and are probably running on 10 mbps unswitched segments of the corporate LAN.
3. It doesn't take a lot of excess RPC traffic to completely overload the 10mbps segment. Even though the computers were not directly exploited by the MSBlaster worm, the traffic caused by all of the infected machines was enough to fully clog their 10mbps segment and effectively created a DOS attack which prevented them from monitoring the equipment out in the field which would have notified them of a failure.
4. When IT staff noticed the machines weren't responding on the network, they did the only thing they knew how to do, rebooted the servers (it says this much in the report).
5. The servers came back up just fine, but seeing as how this is probably the system administration group troubleshooting at this point and not the network administration group, they probably had no clue that their network was still overloaded, so the problem still existed.
6. Everything crashed and massive blackouts and chaos ensued.
I think it is foolish to think that MSBlaster had nothing whatsoever to do with this outage. Sure you can look through the syslog on the SCADA servers and there's probably nothing in there that would indicate a problem. That's because they were completely cut off from the rest of the network. But if they had something like MRTG that was able to monitor the bandwidth utilization on their LAN segment (highly unlikely on old equipment like this, especially 10mbps unswitched), they would have noticed near 100% utilization of the segment and collisions galore.
This is just my theory, but I'm sticking to it unless they prove otherwise.
"When the president does it, that means it's not illegal." - Richard M. Nixon
He was VERY vocal when the outage occurred that it was the fault of the Canadian power grid, although he had NO basis for that assumption.
His pissy attitude was inexcusable and Canada should request a formal apology.
MadCow.
I used to have a sig, but I set it free and it never came back.
he likes puppies!!!
"The blackout investigation homepage lists all NERC's documents relating to this event"
/~warez directory too.
http://www.nerc.com/~filez/blackout.html
Wonder if they have a
Beauty is in the eye of the beerholder.
At least they were running UNIX... if they were running Windoze, everyone in North America would be back to the Stone Age for a few months.
So sever all power ties with the US, since the ontarioans never experience accidents, no
Actually, the primary cause -- and make no mistake, it should not have brought down multiple states and provinces -- was tree limbs that had fallen on power lines. At least that is how it was presented in the half dozen or so articles I read on the issue.
The amusing thing to me is that we had a guy run for local office in Michigan on the platform of "no unnecessary tree trimming." I hope his wife and neighbors are reminding him of that now.
-- I Am Not A Terrorist.
"According to the product specs, it is a Unix system with X Windows."
:6 41. php
Thats a outcry lie
http://www.automationtechies.com/sitepages/pid
SCADA are OPC (OLE for Process Control) based.
So the story is a false report. Microsoft is again the company to blaim.
Robert
So now the score is *nix:678321287636537 M$:1
Check out The Industrial Physicist (www.tipmagazine.com) October/November 2003, for another take. A big change has occured since de-regulation hit that the grid was not designed for. Local systems are responsible for but have no control over the energy flowing through their systems for trading purposes. ENRON and other companies are "gaming" the system.
FYI, US governement have been engaged in a nice deregulation wave for 10 years. That deregulation has been poorly tought out, because of a "The Market Cures All" mindset. What did the deregulation bring? Gigantic power outages and California-style power crunches.
The deregulated US power grid is overstressed, has little or no margin for extra demand, and has outdated and poorly maintained emergency systems. And thanks to Enron and friends, artificial scarceness has been created to crank up prices.
In a nutshell, you pay more and get less(safety, avaliability, quality) than when the market was regulated. Ain't The Market grand?
Electric power is a basic necessity(in our post-industrial world), and should be managed by the state, just like water works and law enforcement.
You're not old until regret takes the place of your dreams.
what about the last major power outage that DID originate from canada? are the Canadians ashamed of that?
exactly.
Seems that their server went down. I guess this means the northeast will be experiencing another blackout.
Blackouts by ./ Must be a first
During a critical hour before a final chain reaction of power line outages began around 4 p.m. Eastern daylight saving time, FirstEnergy grid controllers were flying blind, unaware of computer and power line problems on their system, according to the investigation headed by U.S. Energy Department officials and their Canadian counterparts.
Anyone else picture Homer Simpson here?
--- I'm Green Hornet's sidekick not Inspector Clouseau's!
First of all, let's be perfectly clear on this.
Any carefully managed OS (inluding Windows) can be stable and predictable.
Any badly managed OS (including Unix) WILL be unstable and unpredictable.
Got it? Good.
Now as far as pointing fingers at the guilty parties, understand that the infrastructure is really at fault far more than any individual company. Look at this from a broader perspective: One company was able to take down how wide of an area? The whole system is too fragile, too interdependent, and maintained too close to full capacity. Worse, there's absolutely no incentive for a company to maintain a large reserve capacity, since it costs a lot, and brings in no extra income.
But of course when it happens the next time (and make no mistake--it will), we'll have another commission to once again figure out the single company that broke a rotten and unstable infrastructure, instead of fixing the root problem.
"People who do stupid things with hazardous materials often die." -- Jim Davidson on alt.folklore.urban
The Energy Bill that the House just passed will have NO EFFECT whatsoever to prevent this blackout from happening again.
Even so, the Congre$$ sold the Energy Bill as a panacea for our energy problems while stuffing it with tens of billions of dollars of tax breaks and special interest provisions. Talk about porkaroma...
In the Mississippi Delta (runs from Memphis to Vicksburg), the power company is a co-op. It's owned by the citizens. Power tends to be pretty reliable, but not very clean (due to old equipment that costs too much to replace).
The policy of the United States is worse than bad---it is insane. -- Ludwig von Mises, Economic Policy(1959)
Slashdot spewed some particularly vile FUD when the rumour came out that the outage had been caused by Microsoft Systems infected with Blaster. Time to eat our own words and apologize to MS for wrongly criticizing their products when the systems involved where actualyy *nix boxen running X.
;)
PS: For the humour-impaired: yes, this is supposed to be a joke
Sounds about right for a really decent UPS to have its batteries run dry. It would have helped had the alarms gone off, and started the backup generator.
(probably not the true scenario, but still stupid enough to be possible)
The living have better things to do than to continue hating the dead.
Get your legal pads ready, boys.
who also refuses to accept the fact that a "hacker" really does mean (as one of its meanings) a malicious hacker?
A word is defined by how most people use it, and like it or not - a hacker, to most people, is someone who does bad things with computers. Done.
Same with this - they lost the battle. It's X Windows whether they like it or not.
You can talk about black helicopters 'till you're blue in the face, but that doesn't change what's in the report. Do you really believe that you're smarter than the people who wrote that up? That they, in their obvious technical incompetence (compared to your evidently superior intellect) ignored the "fact" that increased RPC traffic is what took down the Unix server?
Their IT infrastructure is outdated and it sucks, the box crashed, get over it and take it like a man.
Here's a timely link. . .
FirstEnergy used to be CEI. . . a horrible company in Cleveland that allowed a nuclear core failure through negligence and then tried to bully the government of Cleveland into selling its municipal power supply to it thereby granting it monopoly status in the region.
Check out the story here:
http://kucinich.us/powertothepeople.htm
The presidential candidate Dennis Kucinich was Mayor of Cleveland at the time and endured a browbeating when he refused to sell the Municipal power company to these scoundrels.
I'm not necessarily pro-Kucinich, but I am VERY anti-FirstEnergy. They have a track record of irresponsibility and dishonesty that should be enough to have the government dismantle them.
I had always wondered which was the superior system, but thanks to your scientific research I now know! Thanks SlashDot!
As long as it didn't have anything to do with that Saddam Hussein fellow, or his little buddy Osama bin Laden...
...are rarely ever computer savvy enough to maintain thier own systems.
Usually these systems were put together by some integrator consultant years ago. The consultant goes on to bigger and better jobs, but the system remains pretty much as it had been set up.
The problem, as we gentle slashdot readers know too well, is that nearly all computers need security patches or updates of some sort applied. In addition, there are often configuration changes, log files which need to be looked at, and a thousand other day to day tasks which need to be done.
But there is nobody to do those things. The company usually buys a packaged turnkey system and they run around with the belief that they don't have to do anything or spend any time to keep the system in good health.
And this is why First Energy's blackout happened. Hint: it could have been any of a number of other energy distribution system operators.
Let me reiterate one very important point: operating a distribution system via SCADA is a dull job. The people doing this sort of work aren't hired because they're bright. They're hired because they can take the boredom.
Homer Simpson the cartoon character is not nearly as far fetched as most people think.
Nearly fifty percent of all graduates come from the bottom half of the class!
Hey, thanks for your id! I've already forwarded a complaint to abuse@goldenbetcasino.net.
You won't see those greens, Einstein!
After browsing all the comments thus far, there doesn't seem to be a definitive answer as to what OS these failed systems were running. Let's find this out... it's also of interest that in the Windows license agreement, it specifically states that Windows shouldn't be used for mission-critical applications involving scenarios like this, so it would be further hilarity if they were running Windows in contrast to the products' own licensing agreement.
To date we have not found a point at which the costs start increasing again (a key factor in having a free market). This causes the market pressure to favor one large company (the most efficient means of producing power). However once that happens, there is no pressure to be efficient or charge fair prices. Government regulations are intended to create that pressure.
In this case the government regulations not only failed to create that pressure but encouraged bad behavior (there are pretty complicated rules for "trading" power, what most people call "deregulation" of power was just a different, and equally bad form of government regulation).
The policy of the United States is worse than bad---it is insane. -- Ludwig von Mises, Economic Policy(1959)
The link specified in your post:
o ftware/en/xa21.htm
http://www.gepower.com/prod_serv/products/scada_s
Brings up a 404 on Ge's website.
Well.. maybe. Or Maybe not. But Definitely not sort of.
FDR enacted regulation of certain industries to insure that consumers would not get ripped-off. Bush reversed these regulations possibly because he doesn't know history, and/or he and/or friends/relatives have stock in these industries, or some other overlooked reason. Bush and his administration heavily contributed to the power outage, and is making sure that plenty more are bound to happen. Remember this on the next election day.
You can read a story I submitted on possible W32.Blaster links in my journal.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
For those of you that might be interested, SCADA is an acronym for
Supervisory Control And Data Acquisition
It defines nothing about whether or not COM, DCOM, OPC (Ole for Process Control), or any other proprietary communication framework is used (contrary to some other highly moderated statements you are likely to see in this discussion)
The bulk of serious SCADA systems in place are probably legacy systems of some kind, including many variants of UNIX systems (we have old micro VAX systems still in use). Many of the newer systems are Windows based, and are obviously subject the standard Windows viri, worms, etc.
Worse yet, these systems are very difficult to upgrade or patch, due to the critical nature of their duties. It is not unlikely that a large portion of the Windows based SCADA systems in the world remain unpatched, and are "safely" firewalled off from the internet.
Of course, the problem is that much of the monitoring gear used to diagnose network issues is also Windows based, and carries with it the standard retinue of Windows viruses and worms, right into the heart of the control center.
These UNIX systems have run (and will contiue to run) uninterrupted for years at a time. We have calendar alerts in place to tell us to go manually reboot unix machines after months of uptime, just to ensure that their SCSI drives will spin back up (in case of a control center power outage, etc.)
Somehow, I don't think that is an issue for some vendor's SCADA systems based on more popular OS's, but I might be wrong...
-- -pjk Perry Kundert perry@kundert.ca http://kundert.2y.net
There is also no evidence, nor is there any information suggesting, that viruses and worms prevalent across the internet at the time of the outage had any significant impact on power generation and delivery systems. SWG analysis to date has brought to light certain concerns with respect to: the possible failure of alarm software; links to control and data acquisition software; and the lack of a system or process for some operators to view adequately the status of electric systems outside their immediate control.
The link to blaster is implicit at best, non existen IMO.
(note: I'm not using the "in other words" tactic of the parent to introduce a whole new paragraph of 'facts')
They would just blame the Linux world for injecting bad code on to their stuff. After all, SCO has been busy getting code somewhere.
Extra space in the link. Try this: http://www.gepower.com/prod_serv/products/scada_so ftware/en/xa21.htm
http://oceanexplorer.noaa.gov/explorations/deepeas t01/background/dumping/media/dumping2.html
yeah i know its crude and mean but you made me laugh. thanks....still can't stop.
Who is this Anonymous Coward character, how does he post so much, and why is he always such a whore?
The real reason the systems crashed was because because they were installing the newest NVIDIA drivers.
Check out the last page of this. What it says is server is Compaq/Unix (which is either dec unix or SCO unix) and the desktop display is NT. All the processing is done on the client side and the servers simply hold the data. HUMMMM.
I prefer the "u" in honour as it seems to be missing these days.
Though the report is wordy, it makes it clear that the real cause of the blackout -- the reason why the normal procedures failed to do their job -- was as follows:
1. The Blaster worm was flooding the utility's network.
2. They therefore couldn't get the necessary status information.
3. They therefore failed to take the necessary corrective actions.
In other words, the real cause was a lack of security in Windows.
So why does michael's introduction try to deceive us, by implying that the Unix control system was at fault? That system didn't fail -- it just couldn't do its job due to the denial-of-service attack caused by Windows and Blaster.
OK, before everybody gets carried away with what SCADA systems are based on, let me point out - SCADA stands for "Supervisory Control and Data Acquisition". No single company makes SCADA any more than any single company makes any other type of application software.
SCADA systems exist that will run on QNX, Linux (See VSystem), Windows NT/2000, DOS (Yes, DOS - i.e. FactoryLink with DesqView) as well as other older platforms like VAX/VMS and PDP-11.
So, SCADA systems do not necessarily rely on OPC or DCOM - some do, some don't. Some still use DDE (trust me, not the most reliable means of communication).
Industry uses all sorts of SCADA systems, from several vendors - Rockwell Software makes RSView, Wonderware makes InTouch, Siemens makes WinCC, you get the picture. The GE package involved in the systems at First Energy is only one example of SCADA, in the same way that EMACS is only one example of a text editor.
Sorry for the rant, but IAAAE (I am an automation engineer).
My wife's impression, from reading a CNN blurb, was that the "root cause was poor tree trimming". Oh boy.
Friends don't help friends install M$ junk.
Our SCADA system (Nelson Mandela Metropolitan Municipality, Port Elizabeth, South Africa) rus on SCO Unixware. I used to be opposed to any "upgrades" to Win2K boxen, purely on the grounds of "Unix=stable=cool" vs. "M$=unstable=suck". Now I'm pretty blase about the whole matter.
BTW, none of our outages have been due to the SCADA system not operating properly. Most of them were due to lack of staff/maintenance and theft of conductor. The communication links between the remote outstations and the comms gateway are in general also less reliable than the computer running the SCADA system. And our operators are also in general pretty dim...
Did anyone else find it funny that the system failed at 2:14 pm? ...on August 29 1997 at 2:14 am SkyNet becomes self aware....
I'm pretty sure he did sat 2:14 am.
The PDF is about 4.4 Meg and the NERC site has been well /.ed
Check out the real specs. The poster is way off base.o ftware/en/downloads/enmac_overview.pdf
http://www.gepower.com/prod_serv/products/scada_s
The Unix server (either Dec or SCO Unix) serves mainly as DB server while the processing and display is done on MS windows.
I prefer the "u" in honour as it seems to be missing these days.
"It's all about early adopter. We invested huge amounts of money to wire up the whole country before nuclear was an option, and there's no way we can spend that much again to rip it all out and replace it with newer/better/safer alternatives. Same thing with our telephone system, cable tv, internet, etc. We're trying to squeeze every last ounce of usefulness out of the existing system, while so-called "3rd world" countries are getting the latest and greatest tech because they have no prior investments."
I disagree completely -- until the US deregulated, we had an extremely reliable power system that was able to expand dramatically for many decades to keep up with demand while keeping prices low. Then some lunatics convinced enough people that it was a good idea to make power companies unregulated monopolies (i.e. they were no longer required to invest in maintenance and infrastructure, or to maintain excess capacity to cover emergencies or power spikes, and removed the cap on profit margin), after which the power companies did what you'd expect -- raised rates like crazy while slashing spending on infrastructure maintenance and expansion. The result is that a bunch of investors and CEO's made tons of money while the capacity and quality of service that they provided suffered. The problem was not just the big outage -- the number of instabilities in the power grid that didn't result in actual outages has been increasing dramatiaclly for decades.
See http://www.ncpa.org/iss/ene/2003/pd081503a.html for some more info.
Who do I blame? The politicians who allowed themselves to be bought off (or conned) by the power companies, to the detriment of the entire country.
Enable 3D printed prosthetics!
"Our transmission system was designed and built to provide reliable service to our customers, not to be a superhighway for long-distance transactions to Canada and elsewhere," said Alexander. "Yet, on August 14, the margins built into our system for serving our customers were being drained by those transactions, with little or no reactive power support to the grid for those sales. The Task Force's solution to this problem would be to interrupt local customers in order to allow for long-distance bulk power sales.
Translation: yeah, we had a problem, but the fact that our problem could take down the whole grid isn't our fault. Honestly, it sounds like a pretty stupid excuse, but then, that's just my reading of it.
The way I read it he's actually taking responsibility for what he should. The First Energy screwed up is one issue. That their screw up produced such an blackout is a seperate issue. He's owning up to the first one, but pointing out that because of the overall grid problems anyone could have caused it, and in fact it could probably happen again in some manner.
It is a slim difference and definitely hard to nail down. As an arbitrary parallel, say 12 people leave 12 different acids out in a lab they thought would be locked overnight. During the night, some one walks in and dumps one of them down the drain messing up the entire building's plumbing. While it's easy to identify which of the 12 the acid belonged to and say 'you should have put that away', it is also true that a) if the room had in fact been locked it wouldn't have happened, and b) any one of the 12 acids could have been used.
Yes, that's a little different from the non-trimmed trees perspective, but the point remains that even if they or another company had had a legitamit, no-fault problem of the same magnatude the blackout probably would have happened.
R: That voice. Where have I heard that voice before? B: In about 365 other episodes. But I don't know who it is either.
And now the power generators think the rate-payers shoulds subsidize them by building them a new grid so they can continue their enronesque shenanigans. Grid wasn't designed for it:
m l
http://www.aip.org/tip/INPHFA/vol-9/iss-5/p8.ht
The problems would be compounded, engineers warned, as independent power producers added new generating units at essentially random locations determined by low labor costs, lax local regulations, or tax incentives. If generators were added far from the main consuming areas, the total quantity of power flows would rapidly increase, overloading transmission lines. " The system was never designed to handle long-distance wheeling," notes Loren Toole, a transmission-system analyst at Los Alamos National Laboratory.
Let's put you up as the standard of win98se's proof from hackfulness. Let's post your IP to the world as such. Then we'll see.
because, no one cares about cracking win98se. It's no fun. I could gloat if I broke my way into microsoft.com, or fsf.org, or whatever. I would then be "l33t." However, NO ONE CARES ABOUT YOU AND YOUR win98 BOX OK? If for some reason someone did, then they probably could get in.
As an aside, your strategy really is pretty good. You have a no-name computer, no reason to break in. You are on an EOL system - no one cares about breaking into 98 anymore so you are probably pretty secure from an obscurity standpoint. However, if for some reason you ever became more visible, well...let's just say that windows 98 isn't a good place to hide.
Who is this Anonymous Coward character, how does he post so much, and why is he always such a whore?
A-O-Way-2-Go OHIO.
This was OK until the wholesale prices skyrocketted due to a combination of heat wave sent demand soaring, and the combination of a lack of new power plants built over the last 10 years, a drought that cut back the amount of hydroelectric power available, and outside power generators unable to supply enough power to the state.
The utilities were forced to sell power at a loss (due to the regulated retail prices) and began to go bankrupt.
Here is a summary of the CA situation.
that this was an EMP. If I remember correctly, this whole thing started with an unexpected surge of power travelling down a line. Sure sounds like an EMP to me...
I read an article about emp's in popular science/mechanics some time ago, and it mentioned this very thing.
Would we be told if an EMP was used?
"73% of quotes on the Internet are made up" -Ben Franklin
More importantly, it may not have been Windows' fault given that tho box was under a denial of service attack. What the failure shows is relying on the Internet to deliver emergency messages is poor system design regardless of what OS is running. It also shows that the power grid can be brought down from a desktop when trees are falling due to high winds.
If this DOS wasn't intentionally targeting the power grid, I wouldn't be at all surprised that the next one does intentionally target the grid.
If it caused the loss of power to multiple states for several days... yeah, I'd be pretty ashamed.
First Energy didn't buy Penelec until the 1990's. TMI melted down way before that.
And how does deregulating the industry make trees ground a few high tension lines?
It's remarkable how easily you are letting this roll off your back. I'm telling you you guys caused a lot of problems for us. We were blamed for it, noone ever appologized for blaming us, or for causing so much trouble for us up here.
This is your reaction?!
He's also the kind of asshole who wants you to get a ticket for sitting down in public or putting "PIZZA CALZONES PASTA" on the sign over your Ray's Pizzeria.
He does let you drink in public though, as long as you are rich.
Tim
Omnia vestra castrorum habetur nobis.
Starting around 14:14 EDT, FE's control room operators lost the alarm function that provided audible and visual indications when a significant piece of equipment changed from an acceptable to problematic condition. Shortly thereafter, the EMS system lost a number of it's remote control consoles. Next it lost the primary server computer that was hosting the alarm functions and then the backup server such that all functions that were being supported on thes servers were stopped at 14:54 EDT. ... FE's system operators remaind unaware ... [and used] outdated system condition information they did have to discount information from others ...
Note also that they lost their state estimator due to poor data feeds and formats - a direct result of using M$ "standards".
I imagine that the Blaster worm worked it's way onto the network, disabled all terminals and made the poor little GE system puke. The data was available in digital form, page 29:
They kept getting good data, even at their teminals, but the alarm program was broken.
It only gets uglier from there. Remote data terminals failing, datalinks down, all sorts of Microsoft induced problems. It's very easy, even without seeing the name to know who to blame. Only Microsoft could perpetuate such a widespread and massive failure.
Friends don't help friends install M$ junk.
I think this comment was more insightful than inflammatory.
As a Quebecer, it's the worst piece of sh** I had ever read...
This story would have 1500 comments by now.
Looks like the headline-grabbers like Pataki and Bloomberg, amongst others, now have little to say about their quick denouncements of Canada for the whole power mess. Interesting that this is the same reaction pattern for the current Canadian internet pharmacy spat, where FDA commissioners are now publicly alleging Canadian drugs to be unsafe. Is 'mouth off first and ask questions later' now an official US political strategy?
First of all, I'm not the cause of the problem (and I'm canadian too, eh), but don't think that it didn't cause a lot of problems for the americans too, as it surely caused more of a problem down south that it did up here. Besides, it's done with, and now the blames been put somewhere...should we find the last person who touched the particular computer that failed and give them a good berating, or what?
Right on. Very informative. If I had moderator points, you'd get 'em.
Thanks for the info!
"According to the product specs, it is a Unix system with X Windows." that the client was running under a $23,000,000 custom conversion to win95?
http://www.securityfocus.com/archive/1/333505/2003 -08-13/2003-08-19/0
3 -08-13/2003-08-19/0
. php
e rwar/view/
http://www.securityfocus.com/archive/1/333513/200
http://www.automationtechies.com/sitepages/pid641
http://www.pbs.org/wgbh/pages/frontline/shows/cyb
The last url is a Frontline report,
"Slammer Worm to hit months after 911 attack" producers Michael Kirk, Jim Gilmore and with the aid of Detective Chris Hsung, Barton Gellman from Washington Post, Robert Cressey and Laura Wygod Website Coordinator Mountain View.
Inside this program Richard Clarke, former Director CyberSecurity at the White House is interviewed. Clarke clearly holds Microsoft responsible for the Cyber Vulnerabilities which may be the cause for the power outage a couple months ago. The program discusses the power grid SCADA systems. During submitting his complaints to the white house and senate, Bill Gates responded by announcing Microsofts bug-hunting period in februari 2002. However Clarke's department was removed from the White House to Homeland Security in feb 2003. Subsequently Clarke was fired, but formally he resigned.
Robert
I know you love to bash the moron in the White House, BUT Bush wasn't in office when deregulation took place. It was the President with his pants around his ankles: Clinton.
Believe it or not, Bush had nothing to do with it.
"A plan fiendishly clever in its intricacies"- Homer Simpson
Your analysis appears to be dead on. It also matches this earlier analysis:
4 ,84510,00.html
... apparently [migrated] through the corporate networks until it finally reached the critical SCADA network via a remote computer through a VPN connection," the report states. As a result, "the worm propagated, blocking SCADA traffic."
http://www.computerworld.com/printthis/2003/0,481
> Blaster worm linked to severity of blackout
> The W32.Blaster worm may have contributed to the cascading effect of the Aug. 14 blackout, government and industry experts revealed this week.
> On the day of the blackout, Blaster degraded the performance of several communications lines linking key data centers used by utility companies to manage the power grid, the sources confirmed.
> "It didn't affect the [control] systems internally, but it most certainly affected the timeliness of the data they were receiving from other networks," said Gary Seifert, a researcher at the U.S. Department of Energy's Idaho National Engineering and Environmental Laboratory in Idaho Falls, referring to flow-control and load-balancing data that's transmitted over public telecommunications networks. "It certainly compounded the problems" relating to the congestion of key communications links used by utilities to coordinate contingency efforts, Seifert added.
Also note, in the same article, these references to earlier cases:
> A spokesperson for the North American Electric Reliability Council (NERC), which is helping to spearhead a task force to study the causes of last month's blackout, declined to comment on the role the Blaster worm may have played. However, a NERC report dated June 20, 2003, shows that the Slammer worm had a significant impact on some utilities.
> In one case, a server on a control center LAN running Microsoft's SQL Server wasn't patched, according to the report. "The worm
> In a second case documented by Princeton, N.J.-based NERC, a frame-relay-based control network using Asynchronous Transfer Mode "became overwhelmed by the worm, blocking SCADA traffic."
Note the similarity of those cases (network traffic due to Windows worm blocks SCADA traffic) to the current report.
While a careful reader can pick out the information, the report seems to have gone out of its way to avoid naming names. Likewise, the posted introduction to the report seems to suggest that it was a Unix system that failed, which is false. Also note how this forum has been buried in posts that back up the false explanation. Why do I get the impression that Microsoft is spending a lot of money in order to manage this story?
Maybe someone will write an article with a clear analysis, bearing the very headline that Microsoft is working so hard to avoid:
BLACKOUT CAUSED BY WINDOWS SECURITY FAILURE
Well, I do not post much, but I thought this was worth it.
The day of the black out, I was heading home to my house in Cleveland. Naturally, we were hit first. It was beautiful that night because my neighbor had his 12" telescope out in his yard.
However, what is really intriguing is what happened over the next week. Everyday, driving to and from work I would notice all of these FirstEnergy trucks out on the street. No, they were not restoring power. THEY WERE TRIMMING TREES.
The NERC report states that FirstEnergy failed to trim trees to protect the grid. FirstEnergy knew that they had caused the problem and they tried to cover it up.
Also keep in mind that FirstEnergy owns the troubled Davis-Bessie Nuclear plant in Sandusky, Ohio. The nuke plant was shut down almost 2 years ago because boric acid had eaten a football sized hole in the reactor lid. It has been repaired but not re-started due to "safety culture concerns".
Finally, my G/F is a field biologist. She is constantly doing field surveys for FirstEnergy. She always speaks of their poor land use and inability to maintain their lines. She has even seen over-heated lines drooping almost to the ground.
peptidbond I was crazy once....
QNX baby!
Facts do not cease to exist because they are ignored. - Aldous Huxley
Posting something only to undo some bad moderation choices.
"but, but, but, 9/11 changed everything!"
And whoosh! All of the good points Mr. Palast raised no longer apply.
[o]_O
I notice that your post has come down from +5, to where it now stands at +3.
That's interesting, since your post is almost the only one that gets to the truth, and provides the necessary information to explain it.
Microsoft really wants to hide the truth in this case. I guess, aside from the negative publicity, even Microsoft is frightened by the thought of being a defendent in a $billion lawsuit for negligence.
Simple. I just spent $700 on a Honda generator. So it's pretty well guaranteed that the grid will stay 100% reliable from now on.
Unless of course I forget to maintain it, so it rusts up. Then the power will take a dive on the coldest night of the year for sure.
http://www.corpwatch.org/issues/PID.jsp?articleid= 8131
Firstenergy is going to walk, deregulation is going to continue. Bush isn't going to bite the hand that feeds.
The US's biggest problem with maintaining a reliable grid is its size. The US has several times the surface area of France, and, it is simply exponentially more difficult to manage a national grid. So, like Europe, the country's power grid is divided into regions, and some are good and some are bad. MISO and NYPP are anemic, southern grids are terrible, and the northeastern PJM interconnect is incredibly good.
In fact, I would take the US PJM interconnect up against France's power grid at any time. That covers about the same area as France, and has withstood several recent blackouts that occurred in other areas.
PJM also has a burgeoning marketplace for congestion management products, dispatches real time prices at any point on its grid for anyone who cares to listen, and a number of other features.
This is my sig.
After looking at the report, it looks more like the GE XA21 SCADA network failure was not the primary cause of the cascading failure but more an effect of the failure. The key failure seems to be a software system callled the "State Estimator" (SE) that is used by the Midwest System Operator (MISO), a NERC reliability coordinator, to develop optimal solutions of for the planned operating level of all of the power generation and transmission equipment in the MISO area covering about 10 midwest states and 1 million square miles. It is not described in much detail but the SE seems to be an optimization tool using a linear programming model that gathers availability data for all of the major system components and load demand every five minutes and then calculates the 'optimal' use of those system components to maintain system reliability at the required level. The 'solution' of the model is then used to plan the operation of the overall system by sending the target operating levels to each facility in the system. So why did it fail? Two reasons. First, the model depends on having accurate availability information from each major system component. Status information is sent to MISO in Indiana by the "ECAR" data netork or by direct links. On the day of the failure, the direct link to a key transmission line was not working and the analyst had turned off the estimator to troubleshoot it. After fixing the problem, he went to lunch and forgot to put the system back in automatic mode where it would develop updated solutions. This situation existed for 2 hours from 12:15 to 14:40. When the estimator was switched back to automatic, it was unable to develop a solution because another key transmission line had overloaded and tripped and *its* new non-operational status was unknown to the model, apparently because the status of that line is assumed to be 'on' until told otherwise. This problem was not corrected until 16:04. The bottom line is that a critical major planning tool was not available for 4 hours for a regional generation and distribution system that absolutely required it's use to be operated successfully when the system power supply was very close to the demand.
The SCADA system itself did not fail, but its alarm function did, which provides alarms to control room operators about system operational problems. The problem with the alarm function seems to be a case of too many alarms for the system to handle as the problems multiplied. The new alarm inputs built up and then overflowed the process input buffers. The alarm system just stalled while processing an alarm event and the alarm function stopped. Then, at 14:41 the primary server hosting the alarm processing application failed due to some combination of the stalling of the alarm application and the queueing to the remote terminals. The hapless backup server then was automatically activated and everything was was transferred to it, even the functional non-alarm stuff. The backup server failed after 13 minutes. Basically, the SCADA alarm system seems to have been massively overloaded (which shouldn't ever happen, of course) beyond the capability of the system design to cope with.
The immediate cause of the failure seems to be the forgetfulness of the analyst who was operating the planning model. The significant underlying contributory cause seems to be a very poor regional operational design in which a critical centralized system planning tool was being used with insufficient backup and oversight. It looks as though both Unix and Windows escape blame. The SCADA system probably was doing far more than it's designers intended and probably performed heroically until it died. 'Aye Captain...I canna do no more.'
Another is how much influence the coal/oil industry still holds in this country. Coal power is SUCH a bad option, its amazing any modern society is still using it as much as America is. Its bad for the environment, its highly inefficient, and its bad for the environment. Its also really bad for the environment!
Oil is likewise a poor option when compared to nuke plants, both in terms of efficiency and polltion. Modern nuke plants are amazingly safe, and produce comparitively little waste. But, that leads into another NIMBY problem of where to dispose of that nuke waste!
Manipulate the moderator system! Mod someone as "overrated" today.
Don't quit your day job... given that you obviously know so little about how these type of high-performance networks work, I suspect its time for you to come up from the basement and go have some supper.
Although this blackout occured in the US, it has not been the only one. Italy, Scandinavia and the UK have also suffered similar problems.
This has been sending political shockwaves all over the place as the realisation that the pressure on costs on the network owners has meant that contigency margins have been reduced. In the UK (where I live) the public face is one of calm, but behind the scenes there is considerable concern.
What it shouldn't mean though, is a step back to the old days when networks were considerably over engineered. Care should be taken that those who advocate this don't put us back 15 to 20 years and hike the price of energy as a result.
What I think is needed is much more sophisticated control systems that recognise the domino effect built in as a safety measure and actually proactively shutdown parts of the networks in much the same way as firemen make firebreaks to stop a rampant fire. This would keep control of a situation that currently just runs out of hand, and would prevent both the widespread loss of power, but also a much faster return to normality.
You are talking about a natural monopoly - nobody is going to build a second set of power lines to deliver power to your house. Also, the networks are all interconnected - as actually happened, a failing network can (and will) pull the others down. So, even if you are willing to pay more for higher reliability, you don't have that option (short of moving/emigrating).
That assertion has been questioned, I think reasonably.
The story told here is a familiar one, resembling many similar stories, giving it the ring of truth.
How much do you really know about power supply? Are you a God of power supply, or are you trusting the assertions of people who you trust know what they're talking about? I can't, myself, claim to know enough about power supply to say definitively whether this or that aspect is a "natural monopoly", but my general experience with assertions that this or that is a natural monopoly has led me to be skeptical of such assertions.
What were the effects of MS Blaster BLASTING its way through the networks between power sites, disrupting the communication which would have stopped the cascading effect? I'm still betting on that being the root cause.