Slashdot Mirror
Tracking the Blackout Bug
Alien54 writes "This earlier Slash story cited a CNN news report on how the August blackout was preventable. But, as seen in this Security Focus article, things are not so simple. 'In the initial stages, nobody really knew what the root cause was,' says Mike Unum, manager of commercial solutions at GE Energy. 'We test exhaustively, we test with third parties, and we had in excess of three million online operational hours in which nothing had ever exercised that bug,' says Unum. 'I'm not sure that more testing would have revealed that. Unfortunately, that's kind of the nature of software... you may never find the problem. I don't think that's unique to control systems or any particular vendor software.' Which leads to a number of other questions."
According to the SecurtyFocus article, the operators had no way of knowing, because the data wasn't "live." This is a common problem with SCADA systems--the systems will display the "last known-good value" if something goes offline. However, the system should also visibly identify the data as "out of service" or "offline," and this didn't seem to happen. That could be an issue at the server, or it could be something blamed on the people commissioning the XA/21 system (assuming the display is configurable enough to allow you to program it at this level).
Even so, there should have been sufficient watchdog messages between the client, the server, and the field hardware for the XA/21 to broadcast a general alarm along the lines of "I can't talk to the stinking field, so we're all flying blind here, you morons!" This is exactly the same as software in my industry (HVAC fire/security systems for large buildings), where if you lose communication to a subsystem or the field, you have to raise alarms all over the place.
The real question is how you could lose such comm and the operators had no visible indication that they were relying on old data. This sounds like a missed requirement, if not insufficient testing.
Tim
The software bug was just one piece of a much bigger problem; I wouldn't want to overstate its' role. There were many other factors; here are just a few:
Poor vegetation management probably played an even bigger role as overloaded power lines warmed up, expanded and sagged into trees and bushes that were supposed to have been cut back.
Poor communications between utilities played a major role.
This whole section of the transmission system was known to be unstable.
An inadequate regulatory structure lacked teeth to deal with known problems.
Lack of adequate transmission line capacity
If all these other problems hadn't been in place, the software bug might never have surfaced. And certainly, the rpoblems would have been contained within a much smaller area -- maybe just First Energy's service area.
An article featured on Slashdot last year lays out the underlying complexity of the power grid very well: "The World's Largest Machine"
Al Bonnyman
Community Broadband Networks
Ths is really strange, since GE is one of those companies that is really high on Quality. Their products are absolutly trustable. The Six sigma focus at GE is famous. GE's jet engines apparently are 12 sigma.
how can you respond to an incident? It just goes to show the need for multiple monitoring systems in mission critical systems.
Wireless News www.DailyWireless
I agree that there's more to this than just one line of code, as some folks seem to believe- I think referring to it as 'one bug' is rather misleading.
As well refer to the things leading up to WWII as 'one problem'.
If a bug exists in the code, but it's never triggered, is it really a bug?
Not everything is pheasibly preventable, with something happening first. Honestly people are people and they might overlook something.
"the bug was unmasked as a particularly subtle incarnation of a common programming error called a "race condition," triggered on August 14th by a perfect storm of events and alarm conditions on the equipment being monitored. The bug had a window of opportunity measured in milliseconds. "
Isn't this the type of problem the B Method (and maybe the Z language too) are designed to address? Use proof logic initially - once you have decided on a behavior you want, design the system in such a way that it is provable it executes this design.
That doesn't mean the DESIGN is flawless, of course. But if we start engineering software on as many levels as we can, mightn't things improve? Normal software development and testing would never have found a critical bug with rare trigger conditions and a millisecond window. If you need precision on that level, you need to (for starters) to KNOW your implimentation of your design is sound, and preferably the code you are running exactly impliments the proven logic. Isn't this what the B Method was created for?
"I object to doing things that computers can do." -- Olin Shivers, lispers.org
...I don't know him from a hole in the wall. But his cousin, E. Pluribus Unum.... that guy, I trust. :)
Honey, I shrunk the Cygwin
Did anyone ever retract their statements? I know the NY Mayor was pretty quick to blame us Canucks.
We all know it was Microsofts fault, this is Slashdot remember? The Blaster Worm?
Oh wait..
If they did all this testing and this bug didn't show up, it makes me wonder how many killer bugs are still in there.
This is the sig that says NI (again)
From the perspective of New York, they saw a surge race through their system East to West, through the choke point into Canada at Niagra station. NY constantly has problems with IMO not following schedules, and from their perspective, this was yet another incident of bad reliability control across the border.
What they didnt know is that the energy was routed through the southern bit of Canada along the lake area, back into the USA in Michigan, to feed all of the communities along the southern shores of the great lakes. The reason this happened is that the coastal towns became electrically isolated from southern ohio because of failures in FirstEnergy territory. I don't think to this day FE has accepted full responsibility for their roles in the failures, something I think should be done with a good house-clearing in their company...
You can't expect just testing to reveal all bugs in a program. Even a simple program would have to be fed completely random data constantly, in every different order and circumstance concievable, for a very long time, to reveal all bugs. That's just not a real option.
The only way to have bug-free software is to write it properly. You have to modularize and simplify everything down to the point that each one is easilly understandable, and it is easy to detect when one is providing a sensless answer (in other words, cross-checking every result). Then, you have to tie them all together in a robust but simple way.
I know it's far easier to say it than do it, but it seems like nobody even tries to do it these days. Even mission-critical systems are commonly built as a single monolithic program, and when you have a lot of things going on within a single program, with no checks of the sanity of the data going into or comming out of each component, there is no way to be 100% certain that the program is theoretically and genuinely perfect. Meanwhile, by modularizing everything, you can PROVE that it is actually perfect.
But this is really just the old Macrokernel vs. Microkernel arguement all over again. A Microkernel can be perfect, while a macrokernel can never be completely bug-free, but people just find the latter to be easier to write, and then spend hundreds times more man-hours finding and removing bugs, rather than spending (less, overall) time doing it correctly in the first place.
Oh yes, almost forgot, IMHO...
Slashdot gets worse every day... Pipedot: News for nerds, without the corporate slant
I've been reading several papers on this for a grad class I'm taking. One of the several problems is no government control. If a power outage might be prevented by shedding some load (turning out power to some people), no company wants to step up to the plate and be the one to turn out the power to their customers. So they luck out, or they have a massive power outage.
This paper (click on the PDF link) has a good summary of the problems in keeping power outages from happening again.
Realistically none of these problems had to happen and wouldn't have happened if the people in charge were doing their jobs. Maybe they were working on a way to make cold fusion feasible, I don't know but if they were negligent then they need to be removed from their position. If they were just too busy with other aspects of the system then they need to bring more people in so the system can be properly maintained. A power outage is a big deal. Of course, one outage is hardly a trend so probably the whole thing is just blown out of proportion.
OK, it's nitpicking, but the largest machine is arguably the telephone system. Among other things, it maintains a synchronized clock (8 kHz base), even across oceans and continents.
Human being (n.): A genetically human, genetically distinct, functioning organism.
If I want to build a large structure (bridge or building) where it is possible that public safety is at issue, I had better have an engineer's signature on the drawings.
This case seems like a real good argument for having the same requirement for software.
Good engineering practice would probably have prevented this. A simple example of such a system would be a burglar/fire alarm panel. The system is self-checking. If any part of the system isn't working (ie. someone cuts a wire), then that causes an alarm.
I realize that there will be strange undetectable bugs in software but if the system as a whole is properly engineered, the system will fail gracefully and safely.
int main()
{
return 0;
}
Because I have shown you bug free software, does that invalidate the rest of your argument?
Matt Fahrenbacher
James Tiberius Kirk: "Spock, the women on your planet are logical. No other planet in the galaxy can make that claim."
Oddly enough, while writing a comment to another user's message, I threw some info in google to learn about FirstEnergy's EMS system, and found this other SecurityFocus story in Feburary 2004, which gives more raw facts than this newer story.
"DiNicola said Thursday that the company, working with GE and energy consultants from Kema Inc., had pinned the trouble on a software glitch by late October and completed its fix by Nov. 19..."
"With the software not functioning properly at that point, data that should have been deleted were instead retained, slowing performance, he said. Similar troubles affected the backup systems. " This dovetails well with why the testers had to "slow" their testing to make the race condition appear.
So, as far as I can figure, there are 24 hours in a day, and 365 days in a year, which equals about 8760 hours in a year (give or take).
Now then, 3 million hours divided by 8760 hours per year equals approximately 342 years, modulo 4070 hours (i.e. approximately 169 days...).
Now then... how the hell do they get the idea that they've been up-and-running for 342 years? Are they counting things in parallel? Even if they were counting end-user operational hours, the number should at least be a couple orders-of-magnitude higher, no?
3M online operational hours sounds like fuddy-duddy accounting to me... although, obviously I haven't looked over the books. I would be interested to see how they came up with this number.
Comment removed based on user account deletion
In the case of HVAC fire systems, there are probably over 500,000 installations of HVAC systems, and these are tested under real fire conditions several times a year (where the type of feedback seen in this blackout investigation is made, each time).
I think this should support Raindance's pointKinetic stupidity has a new brand leader: Allen Zadr.
if(int(rand()*1e20)==31337){
blow_up();
} else {
do_your_work();
}
Now I can't imagine amount of testing in proprietary software that could reveal this example of malicious code. In open source one look at the code will reveal it. Of course not all cases are so obvious, but always reading the code should be used together with "testing the software". How do you know lots of proprietary software that IS close-source isn't i.e. a gatweway for terrorists? How do you know biggest companies' stuff isn't all trojans? It wouldn't be hard to hide it. Say your software is kind of server. It does its job okay unless it receives TCP packets starting with certain string. Then it just executes commands contained after that string. Boom. No amount of -testing- will reveal this.
And there are bugs that can be triggered once in several billion cases. Only looking at the code could fix them and explaining "we did a lot of tests" is bullshit.
I put a lot of iron, gum, different materials, C4, glass and some more together and it goes, I call it "a car" and I rode 1000's of kilometers okay. Now no amount of testing in all road conditions will reveal it contains the C4 explosives. Looking under the hood will reveal it really fast.
45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
From the Slashdot story: "Unfortunately, that's kind of the nature of software... you may never find the problem."
What the parent poster said sounds right. The GE spokesperson is just trying to fix with bullshit what should be fixed with engineering.
I'm not saying it's always feasible to test exhaustively, but don't say you did when you clearly didn't.
Also: "we had in excess of three million online operational hours in which nothing had ever exercised that bug"
Taken with the "exhaustively" statement, I'm thinking that whoever said these things doesn't understand QA very well. It's easy to write code that works well when everything's good, and it's often just as easy to test that. It's another thing entirely to write code that works well (or fails gracefully) when everything's wrong. And again, it's harder to test that.
-- Fratz, human
I'm also a big fan of watchdog timers. The process that periodically resets the timer can make all sorts of health and sanity checks.
Mea navis aericumbens anguillis abundat
I think the nation/region would be served better if we stepped back a bit and took another look at more decentralised power generation as a full bore government encouraged option. Not as a complete replacement, but frankly, I see no reason we can't have millions more solar panels and wind generators out there. Economy of scale in manufacturing, spurring on even more R&D, etc, works for everything else it appears. And having a lot more points of production, spread out, would help to mitigate cascading failures, especially if islanding was more precise and easier to implement in smaller areas. Wind, were the average wind is adequate enough, is especially cost effective now, approaching coal burning costs per watt. Solar is nice where applicable by climate because it can make use of dead space going to waste, millions of roofs already there.
I like the "not all your eggs in one basket" approach to problems, and I believe in backups for everything.
They were doing their job, cutting budgets and payroll costs. Oh, you wanted the system to operate reliably too?
Mea navis aericumbens anguillis abundat
Once you hit a level of professionalism, then you are PAID to think outside the box and anticipate unlikely problems.
In a disater, this becomes difference between companies that take a financial loss and those that file Chapter 11.
Why don't we point out the real problem that likely caused this to happen. Energy deregulation in the first place.
I know I'll be jumped on by the free market types for daring to suggest this, but I'd rather have a regulated monopoly then a free-market for my life essential services anyday of the week. That article you linked is very interesting reading. Some quotes:
Of course it's the first quote that rings true with me. If deregulation is so friggen great then where is the cheap electric? Why can my Village sell me electric for $0.04/kWh with their regulated municipal power authority (while paying their workers Government rates and with Government benefits) when my girlfriend (who lives a whole two miles away) pays $0.14/kWh for electric supplied by a company that is supposedly part of the free market (a company that pays their employees crap and outsources their call center/billing functions to India). What's the problem with that picture?
Before energy deregulation the price of our electric was regulated by the PSC (Public Service Commission) and was fairly stable. The company that had the monopoly in this area made a set amount of profit (it wasn't a bad stock to pick up either -- you knew what you were getting), treated their employees well and charged a fair rate. Nowadays they treat their employees like crap, the stock has tanked because they are eating the price difference from their suppliers (otherwise we'd be paying about $0.20 kWh) and they are being raped by out of state suppliers that bought all of their generation capacity.
In another slightly related story the out of state company that bought one of their power plants sued the local township because they wanted the tax levy on the power plant reduced. They claimed that it wasn't worth what it used to be because they didn't plan on operating it (it was to be backup generation). After a three-year legal battle the township lost (ran out of money to pay the lawyers) and the tax levy was reduced by some 60%. Property and school taxes on
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
That's assuming the faults get fixed. I've seen buildings with the new fancy computerized fire alarm systems where alarms for sensor and wiring faults get ignored for months.
Mea navis aericumbens anguillis abundat
It is possible for some problems to construct a formal
description of the code. There are many,
many tools (e.g., SPIN, ACL2) that take this
formal description and produce a rigorous
proof of some property, e.g., that some state is
never reached, that a safety or liveness property
is upheld, etc.
http://spinroot.com/spin/whatispin.html
AMD uses this to test the floating point unit
in their chips, to make sure the algorithm they
use will not result in an Intel-style half
billion dollar mistake.
The question is: does your application warrant
the time and cost needed to create the formal
description of the problem, needed to drive these
tools.
This probably would've been prevented if they had compiled using -O3 and -march=athlon-xp.
Someone said "always go with package installs" and that person had more seniority.
Unum. 'I'm not sure that more testing would have revealed that. Unfortunately, that's kind of the nature of software... you may never find the problem. I don't think that's unique to control systems or any particular vendor software.'
+++ATHZ 99:5:80
Ok, so they found the trigger ... poor maintenance left cables hanging down on trees, and a bug in software failed to set an alarm off when those cables tripped off.
.. the cascade failure.
But They need to deal with the REAL PROBLEM.
The surrounding electrical utilities, when they measured the power fluctuations hours before the cascade, acted soley to protect themselves instead of protecting the system and the customers. They acted to trip off their own systems and shunt the power drain to other utilities.
By doing so, loads too big to fill were thrown on down the line, forcing more and more utilities to trip off. .
The utilities are required by law to act for the system first, before ducking their heads in the sand the way they all did. They could have isolated a small area and left the outage as a minor event never making the news.
Instead in typical dumbass ignorant american fashion, everyone ignored what was happening including Dubya and tried to blame it on a utility in Canada.
The truth is out now, but with their rude american ways and short attention spans, it will never occur to them to even apologize to the Canadian people and systems that they in fact disrupted.
George Bush + Linux = "I will not let information get in the way of the fight against Windows"
It's called "code coverage analysis". You run tests, with the code profiled to track which instructions have run. Then you generate a report, and go look at all the code which never got run, and try to figure out how to change your tests to make it run.
/., so I wouldn't expect very many of the posters to actually have a clue...
And then you add "fault injection", which is a technique to force "errors" to happen (which in this case would cause a particular return value from rand()) - and Ta Da! You have found the "bug".
But then, this is
You need programmers with a good background in real-time and concurrent programming, who understand the hazards and how to avoid them.
Mea navis aericumbens anguillis abundat
Exhaustive testing, however you wish to define that, can reduce the number of defects in the code, but it isn't going to eliminate them in a complex system. The number of defects found per unit of test time follows a predictable curve, where each new defect found requires more test time. It's like accelerating to the speed of light, the closer you get to 0 defects, the more test time is needed.
Mea navis aericumbens anguillis abundat
There were many other factors; here are just a few:
Yeah, and don't forget the biggest cause: Canada! We all knew immediately that it was their fault. They probably wrote this software too.
"I have never let my schooling interfere with my education." - Mark Twain
Power over IP would prevent blackouts like this from happening in the future. The internet is the solution to everything... even power.
I feel obliged to point out 2 things about this statement:
Systems designed to operate for 3 million unbroken hours without failure should have been tested both before release, and after release using information gathered during operation.
I would expect (were this my system, for instance) to have to periodically redesign software and upgrade hardware. Accountants might hate it, but it has to be done if you want to garuntee your uptime.
And finally (and all you EE's out there please consider careful before you decide I need my clock cleaned for this remark): this is the kind of thing you can expect to happen when you send electrical engineers to a software engineering job. (And don't all you kiddies who consider yourselves Software Engineers get too excited patting yourselves on the back, either; you haven't heard my thoughts on the current state of software engineering programs, yet; there are good reasons the EE's still dominate the embedded and real-time fields)
The testing required and at least some part of the requirements analysis should have involved software engineers. The fact that there probably were no software engineers when this system was designed and implemented just highlights the fact that the corps invovled were too busy trying to prevent distributed power technology from catching hold to maintain and upgrade their existing systems as new technology became available -- esp in the software side. Imo.
"The Internet is made of cats."
[100% ISO 646 Compliant]
SVM, ERGO MONSTRO.
Why don't we point out the real problem that likely caused this to happen. Energy deregulation in the first place.
I think it is more accurate to say that deregulation enabled, not caused, the problem. Certainly First Energy used deregulation to put in place much of the pieces of the problem. You just don't hear about all the well run deregulated power systems.
If you open your mind too wide, people will throw trash in it.
No matter how fancy your testing system, the real world has more connections, more diiots with fingers on keyboards, more feet tripping over cables, more weather knocking out transformers and lines, more everything.
I'm not sure why that is even remotely hard to understand.
Infuriate left and right
For an obscure race condition, this is undoubtedly true.
Unfortunately, that's kind of the nature of software... you may never find the problem.
This is sorta true, sorta false, and definitely misleading.
I don't think that's unique to control systems or any particular vendor software.
No, it's not unique; bugs that may never be found are rampant in most varieties of software. What's false -- tragically, crushingly false -- is the presumption that these unfindable bugs are therefore inevitable. They are not.
If there's a class of bugs that's hard to test for -- and of course there are many such classes -- the prudent thing to do is to find development methodologies that skirt those bugs entirely. If you don't put in so many bugs in the first place, you obviously don't have to work so hard trying to find and fix them.
It's been said there's a monitoring business just North of here (Indianapolis) which is responsible for tracking power issues and taking care of these types of situations and helping to balance the power grid when it happens.
The local media has investigated it pretty thoroughly and determined they were a major cause in the blackout becoming as widespread as it was.
Apollo landed on 40,000 lines of beautiful, bug free code. Yes, Mission critical can be done perfectly, it just takes half the GDP of the USA to do.
Also brings up the joke if they can land a guy on the moon in 40,000 lines...What the heck is going on with Windows 95 that it needs 16 million?
Couldn't find a HTML link fast, Word doc:
Word reference
Yes, we do not hear about them, because they do not exist.
Sure, it was First Energy's lines that failed initially, but if it wasn't First Energy, some other utility would have failed eventually. The engineering and the legal descriptions of the current electrical generation and distriubtion system in North America are at odds with one another.
There's a good technical discussion on the failings of the power grid that may interest you.
Naturally, deregulation, the method by which government supports monopolies by restricting competition in bizarre ways (as opposed to the less-fashionable old tactic, regulation, by which government supports monopolies directly), is a major contributor to the problem. But it is critical that one distinguish between the shortcomings of deregulation and the shortcomings of unregulated utilities. They are not at all the same thing.
Not to flog a dead issue: but why is it that it's ok for California to have Months of Rolling blackouts, affecting more people (total) over a longer period of time than the entire single blackout on the east coast?
So if I enable a problem that wasn't enabled before that means I didn't cause it? Explain that one to me. You just don't hear about all the well run deregulated power systems.
Generally speaking if you don't hear about something then it probably doesn't exist.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Perhaps because the municipal power authorities don't pay any attention to the future, take new lines the the non-municipal paid to install without paying for it, has many more customers per mile, and does minimal maintenance.
At least in my area it is like that. I'm a member of an electric co-op. We have 16 customers per mile of line on average, the nearest investor owned utility has ~45, and the municipal ~115. The municipal takes the high profit lines, and leaves the rest to someone else. Both the company, and the co-op are paying attention to future needs, making sure generators are getting upgraded before there is a need. The Municipals know nothing about running a utility, so they do only what is required to get by.
I find it hard to feel sorry for that one township you sited, since there are many townships without that high taxed power plant around. I'll admit a bias, the nearest city to me is facing a budget crunch because they counted an a power plant to pay for everything, and those taxes are going away, now they want to annex me to pay their dept on a beautiful (but too large) town hall, and other boondoggles.
Wholesale rates have gone up everywhere. I live in an area where there never was regulation, and we face exactly the same higher rates. Coal prices are higher. Haven't you noticed that gas is nearly twice what it was 5 years ago? (Was $1.15/gallon, $1.78 now) It all connects.
I work for a large electrical utility in Texas. I've heard that vegetation management
(as you put it) is the first thing to go when budgets get tight; they'd rather pay out
bonuses than proactively trim trees to prevent undue outages.
While I can understand that one does not necessarily want every Tom, Dick, and Henrietta checking changes into the current CVS branch, software which is created to reliably serve the General Public's need on a 24/7 basis, should be available for the said General Public to at least examine and critique. This would create not only the much needed conduit between Industry and Academia, but also the background 'body of literature' which is so essential to all learning. It would also vastly improve the code quality as the coders would know that they were doing their job in the public gaze.
Comment removed based on user account deletion
I'm going to call bullshit on that. My Village has been running municipal electric since the 1910s. It is self-sustaining (i.e: takes in enough money to operate without using tax dollars) and geared towards the future. They didn't annex any lines or equipment from private companies -- it was built from the ground up. They don't own their own generator plants anymore (last one went offline in the 50s) -- they buy it from the wholesale grid just like everybody else. And yet somehow they are able to provide it at $0.04 kWh without screwing over their employees or customers. This municipal grid feeds everybody in town from houses to streetlights to factories. I'll grant you they don't have to serve a rural area but rural areas aren't automatically four to five times as expensive -- if that was the case then why do my parents, girlfriend, grandparents and friends all pay the same high rate even though they all live in suburban or urban areas with the exception of Grandma?
I find it hard to feel sorry for that one township you sited, since there are many townships without that high taxed power plant around.
Perhaps you'd feel for them more if it was your friends and family that lived there. Perhaps you'd feel for them if half of them had previously worked there before being laid off by the company that bought the plant and screwed the town over. Then the company fired the plant back up and brought in it's own people from out of state to run things. But that's ok -- last I heard New York state was going to go after them. Care to place bets on who will run out of money first in that legal battle?
Wholesale rates have gone up everywhere. I live in an area where there never was regulation, and we face exactly the same higher rates. Coal prices are higher.
So coal prices are the reason why Enron and it's buddies were slashing power production at their plants in California to drive up the wholesale prices? I wish my state would just tell the Feds to fuck off and regulate our own power industry. Everybody was better served when it was regulated -- from the power company itself (I never heard them complaining when they were posting a 15-20% profit margin) to the consumers. The product was more reliable. Those are facts. I just don't think it's a good idea to allow essential services that you really can't get elsewhere to be run by unregulated industries. What's your option if your power company is screwing you (and they are screwing you in all likelihood because if they don't they won't survive because they are being screwed by the wholesalers)? Not using electric? Try that one in a New England winter.
While I'm on this rant I might also point out the phone and cable companies. Before the deregulation of the phone companies my standard phone service was $15-$20 (before long distance charges). Now it's $35 -- or would be if I still had a landline phone. Before the deregulation of the cable industry (forced on my state by the Feds) we had tons of local cable companies and basic cable (50-70 channels depending on where you lived) was $15 a month. Now with Time Warner it's $45 a month. The only thing you can count on Time Warner for is to raise their rates once a year. You can set your watch by it.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Amen, brother.
Not only that, but 20 years ago, our phone bill came on a postcard.
Once you figure out how to simulate the electric power grid for a good section of the country, you will be set. You seem to have a good handle on the approach, just scale up and scale up.
One outage in recent years is hardly a trend and is more likely just blown out of proportion.
Quite. A quarter of the country losing power is surely blowing things out of proportion, after all, that's how you scale things up, eh?
Infuriate left and right
Generally speaking if you don't hear about something then it probably doesn't exist.
Yeah, like those fictional women who aren't mugged and gang-raped, and those non-existent cars that manage never to crash into one another.
There are many, many, many things that do exist, but you don't hear about them, because they work just fine and therefore remain unnoticed.
Ask a competent network administrator how many people know his name...
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
The writer makes some excellent points, and I certainly agree with the statement that the power grids are literally the World's biggest machines. And being such, they are among the World's most complex systems. Therefore, they are subject to the laws of Chaos Mathematics. In other words, no matter how well we test such as sytem, and no matter how many safequards we build into it, it will occasionally behave chaotically. This is true for Shuttle crashes, airplane crashes, train crashes, any really large system (crashing). Anytime something of this sort happens, and you read or hear that "a string of very improbable events all happened at one", and the investigator goes on to say that, "if one of these events hadn't happened, the plane wouldn't have crashed.", you just heard about "Chaos in Action". The long list of improbable events the writer mentioned just illustrate my point.
The first thing they teach you in a software testing course is that testing cannot guarantee the absence of bugs. The only way you can guarantee, through testing alone, that your program is error-free is to exhaustively test every possible "input" (combination of external inputs and internal state) and check them. When was the last time you wrote a program with a finite (and tractable) input space?
If you need 100% reliable programs, you'll need to prove them correct, and that's enormously difficult to do, and doesn't help if the bug is the result of a flaw in the program's requirement specification rather than an incorrect implementation of that specification.
What testing *can* do is provide estimates of a system's reliability, and in the real world that's all you're going to get.
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
Blackout slashfic? Is the internet broken, can I get my quarter back?
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
He means for all intents and purposes.
Another victim of whole language reading. He did as he was taught, he guessed at what would convey his thoughts.
Phonics rules.
An analogy would be Canadian drug prices. It's easy for those of us in the US to marvel at the lower prices of drugs in Canada without first considering the fact that it's only cheaper because Canadians pay a bulk of their taxes towards their health care expenses (29% of total tax revenue in 2002).
If it's not taxes, then the municipal funds itself by offering bonds, which then pushes the higher costs onto future subscribers. This isn't an effective solution, as it depends on future growth to give current subscribers a lower rate. You're effectively mortgaging your children's future so you can leave the lights on now.
Further more, if municipals were truly better, then it should have been the Canadian Power authorities or ConEdison that recognized the problem and cut Ohio off of the grid, thereby preventing this whole problem. But they didn't. Instead it was a private company outside of Philadelphia that helps maintain the grid that recognized the issue as it was happening and isolated it further. In fact, they said in a news story that they practice the very type of blackout event twice a year. They do this because they have a responsibility to their shareholders, and their customers and know that screwing either of them is not good business.
Yes, a company like Enron can game the system, and screw a lot of people, but I think we can honestly recognize:
btw, nice strawman, mentioning outsourcing while talking about a deregulated power company. sure to get a raise, but can we keep the logical fallacies to a minimum please? thanks
I know none of the big unregulated power companies are saints - all of them put profit before safety and reliablility these days.
But First Energy is spectacularly unsafe. The nation's second and third worst nuclear accidents happened on their watch, at their David-Besse plant in Ohio. For six years they just wiped off the leaking coolant from the reactor head. It was inspected several times by the Nuclear Regulatory Commission. It wasn't until there was a hole eaten in the head big enough to stick a gallon milk jug into, with only a thin (1/16 inch) veneer of stainless steel between the US and a Three Mile Island to Chernobyl accident, that it was even discovered and shut down! They are trying, with NRC supervision (like that helps), to restart it with a new head, and last I heard they couldn't go 24 hours without another failure of some sort or another!
This is what I would do:
First, I would re-regulate the industry and get tough. Make it clear that power companies are expected to be safe and reliable as being a power company is a public trust.
Second, regulatory bodies should not be responsible for industry promotion. They should be regulatory bodies only. This would toughen inspections, as there would be no more "nudge, nudge, wink, wink" at their industry "pals".
Third, look at what worked during the blackout. Coal and nuclear, dirty power, when down. Clean energy, such as the Niagara Falls hydroelectric plant, kept right on chugging. That shows us a good direction to go for the future.
Fourth, repair or replace the grid. It is held together badly by a bunch of companies who don't want to spend money on it. Either repair it, or localize the power sources (even per building) so no more huge blackouts can occur.
Shinoda: "Is Godzilla showing his hatred toward man-made energy?"
Godzilla: "Human! Impertinent! I rule the Atom!"
"Godzilla 2000 Millennium" (Japanese version)
... you can DO IT YOURSELF and not wait on the government or the energy monopolies. And it's scalable from 10$ on up. At the ten buck level you get get dedicated small solar powered devices, I have a radio I wrote about that has a crank genny on the side and a solar panel on the top. A friend gave it to me, he sold them, and I know his wholesale cost was around 10 clams, retail is around 30. We have a small solar rig here for grid juice backup, 3 panels, charge controller, batteries, small inverter, and we have a small wind genny. The wind genny I keep non mounted as a backup now in case of nasty storm damage, but I got all the stuff needed to quick install it within an afternnoon should there be a severe emergency. I believe in backups for backups.. I tell you on the wind genny, there is an industry going begging from potential customers just not realising how well they work and how cheap they are. We could put entire laid off out of work US rust belt guys back to work making them in mass quantities, all they are is a freaking vacuum cleaner motor (more or less, casually speaking now, they are DC not AC mostly) with some propeller doo dads on them. I mean, easy to make, cheap too. They go up in size from there of course. We also have a couple of smallish fuel gennys for backup to that here, and backup to THAT we got firewood and kerosene..
I built a small scale demonstration model methane digester before, man o man there's another major *thing* being under utilized inside the US, you get burnable gas easy. Took me less than 1/2 hour to build a working model out of scrap junk I had kicking around.
We DON'T have an energy crisis, we have a MONOPOLY energy supplier & governmental & media -> to the people education crisis. The fatcats who make trillions off "energy" DON'T want people to find out how easy and affordable it is to be your own micro energy producer. They want you to keep sending them a check,month after month, forever. Produce your own you can pay it off and own that sucker. Grid only is rent your juice from them, zero price guarantees down the road. I issued a challenge several times, I'd like to see ONE example where joe paycheck can go to any local elelctrico monopoly and get a carved in stone price guarantee good for ten to 20 years down the road. No one has even bothering replying, because it don't exist. so you can't say what it'sgoing to cost you even next year, let alone 20 years down the road. folks looking at retirement and a more restricted income might want to think on that some. With home produced, you got that guarantee, at least you'll always have "some" power that can't be fugged over by government/industry/politics. Just like with a nice garden you can always produce a lot of your chow. Just makes sense to me.. You know up front what it costs.
I always chime in on any energy related topics here at slasherdott, and on other forums, with anecdotals to help counteract the industry FUD out there. To produce at least some of your power-for any random regular joepaycheck, is QUITE doable and affordable now, especially when there are any number of big lenders out there that will let you tie in your start up costs into your 20 year note. Costs no more for a real decent home primary or backup system than an additional bathroom in your house. And it's doable and scaleable from hardly zip, I started with one small panel and one battery, worked up from there.
I have seen people who will gladly drop more on a big screen TV than it would have cost them for a good starter rig, then complain that "it isn't affordable". Geeks especially, home power production and storage has a GREAT application in the SOHO, you get a killer good UPS system out of it and you get controllable, tweakable clean double emphasis clean day to day power. win/win there, you're gonna have/want a UPS system for your boxes anyway, might as well do the logical next step and make it *nice*.
I am not a software engineer but I think the real problem with software in practice is this:
With civil engineering stuff, people draw the blueprints etc, make the nice plastic/clay models, and once everything seems fine they build the real thing.
With software the plastic/clay models are actually _fully_functional_, and too many people think that's the real thing, especially since the plastic/clay models are as costly to make as the real thing if not _more_costly_.
But hey, I'm probably not as smart as those "Real Programmers" talking about P, NP and so on...
I just work in the IT security line, so what would I know...
Problem with hydroelectric is there hasn't been a new dam built in this country for a few decades. I'd take a dam (granted it harms the local fish population -- but there are way's to avoid or limit this) over a coal powerplant spewing radioactive dust into the atomosphere any day. I also wouldn't mind seeing more investment in nuclear power -- though that seems to be a taboo subject these days. Gas is also an option -- it pollutes but at least it's not as bad as coal and we (combined with Canada) have large reserves of the stuff.
Fourth, repair or replace the grid. It is held together badly by a bunch of companies who don't want to spend money on it. Either repair it, or localize the power sources (even per building) so no more huge blackouts can occur.
I think the grid is in better shape then people give it credit for. It just wasn't designed with this deregulated system (transporting power over hundreds of miles) in mind. If you re-regulated the power industry the grid would probably be in pretty good shape.
I'd like to see my state implement regulation again. People bitch about the bureaucracy and high taxes of New York but at least when you need it (be it the Public Service Commission, the Insurance Department, legal aid, etc etc) it's going to be there for you. We should buy back all the power plants from the out of state bastards that are trying to hold us hostage (the only reason it isn't working as well as it did in California is we have access to cheap hydroelectric power from Canada -- so they can't blackmail us as effectively as they did California) and let our local utility companies run things again.
Of course it probably won't happen until there is another disaster. That's just the way we seem to work these days (be it with blackouts, 9/11, or what have you).
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Bzzzt wrong answer. My municipal power agency has been self-sustaining since 1920. They don't take in any tax dollars -- they run it all on the money they take in. Sure it's a Government run Agency so it can't make a profit (though they do take in extra cash for a rainy day fund) -- but for the sake of the argument if they increased prices 50% (to make a profit) they'd still be cheaper then the non-municipal options.
If it's not taxes, then the municipal funds itself by offering bonds, which then pushes the higher costs onto future subscribers.
Wrong again. The last bond they issued was back in the 1950s to build a new substation. The Agency started in the 1900s off tax dollars with a charter to provide street lighting. Over time they hooked up private customers (the infrastructure was already in place) and became self-sustaining. Perhaps that's the exception rather then the rule but you shouldn't go painting all municipal power with a broad brush of "You are just being screwed on your taxes" or what not.
Enron is the exception, and not the norm. Not many companies operate like Enron did, or was as unethical they were.
Really? Did you bother to read the story about the power plant in a local township near me? After they won their petty tax battle by exhausting the town's financial resources they fired the plant back up with out of state employees that they brought in. Sure we could rehire the local people that used to work there but they actually fought us on our tax levy so fuck em! I hope NYS shoves it up their ass -- they are going after them last I heard and something tells me that NYS won't run out of money like the township did.
I think we can all agree that unethical behavior, ignorance, and incompetence are not limited to private corporations, but government agencies, municipal authorities also exhibit those human qualities.
Your point?
btw, nice strawman, mentioning outsourcing while talking about a deregulated power company. sure to get a raise, but can we keep the logical fallacies to a minimum please? thanks
Why not? It's a valid point. Our power company (which was always a publicly held company) used to make enough profit that they could hire local people and pay them a decent (some would say too high but that's another story) wage. Now that they were forced to sell off their generation capacity they are being raked over the coals by the out of state suppliers and profits are a thing of the past.
So how did they respond? By laying off as many workers as possible and outsourcing whatever they could. And they still aren't back in the black. The PSC isn't going to let them charge the $0.20 kWh it would cost to put them in the black (why should they? All the money would just be leaving NYS) so it's a lose-lose battle for all involved. The customers get screwed, the employees get screwed, the townships get screwed and the shareholders (of the power company) get screwed. The only people who are winning are the shareholders of the out of state energy company that's screwing us over. The only reason it's not as bad as it was in California is because NYS has access to cheap hydroelectric power from Canada. That's the only thing keeping them from screwing us completely -- and it's the only thing keeping our power companies solvent. Thank god the Canadian companies at least have some ethics and responsibility.
So keep advocating your deregulated industry. I'm waiting for individual states to just start regulating it on their own. It wouldn't be the first time.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Okay, so you've had a bad experience in your area, with a poorly run company, and you've got a stellar municipality. Kudos to the PU, and it's shame that the company can't find their ass with both hands. But why judge EVERYONE'S experiences based on what's happening locally? Why return everyone to a single government-run monopoly based on your township? Deregulation doesn't work for you, and you've found a town that agrees with you. Cool. I don't want that and I'm happy where I am. That's what I love about this country: the ability for us to disagree politely and go our different ways. If, in the future, deregulation doesn't work, we'll fix it. If it does work, it will spread to those who want it.
It's not my area -- it's my entire state that is having this problem. The power company isn't being run poorly -- they are simply trying to survive while being raked over the coals by their suppliers. They built a functional system from the ground up and were forced to sell parts of it off (the power plants) to out of state suppliers and become nothing more then a energy deliveryman because some dolt in Washington figured it would be a good idea.
What's the solution? Let them pass the charges on to the customers? Hint: Our (already shitty) economy won't survive the electric rates going up by a factor of two or three. We are being raped and nobody seems to give a damn.
Why return everyone to a single government-run monopoly based on your township
I wasn't advocating that. I was advocating a return to a regulated power system -- perhaps a regulated monopoly but not a Government-run monopoly (though that is an idea).
Deregulation doesn't work for you, and you've found a town that agrees with you. Cool. I don't want that and I'm happy where I am.
Well good for you. But deregulation has been a disaster for my entire state. If it works for you then great -- but we will probably regulate our power industry again (with or without the permission of Washington). If that means our Attorney General needs to file suit against the Federal regulations and laws then so be it. We won't be the next California. We watched all the help they got from the Feds when the shit hit the fan -- the cynical part of me wonders if they didn't allow that to happen because California (like New York) is a Democratic bastion.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
Poor vegetation management probably played an even bigger role as overloaded power lines warmed up, expanded and sagged into trees and bushes that were supposed to have been cut back. This is a red herring. The alarm system's purpose was to alert the system operators if a transmission line went down. These things happen, and that is why they have an alarm system. A failure in an alarm system will never lead to a serious problem if the events it is supposed to detect never happen. This does not absolve the the XA/21 developers in any way.