Tracking the Blackout Bug

Alien54 writes "This earlier Slash story cited a CNN news report on how the August blackout was preventable. But, as seen in this Security Focus article, things are not so simple. 'In the initial stages, nobody really knew what the root cause was,' says Mike Unum, manager of commercial solutions at GE Energy. 'We test exhaustively, we test with third parties, and we had in excess of three million online operational hours in which nothing had ever exercised that bug,' says Unum. 'I'm not sure that more testing would have revealed that. Unfortunately, that's kind of the nature of software... you may never find the problem. I don't think that's unique to control systems or any particular vendor software.' Which leads to a number of other questions."

Comments

Software bug was just one part of bigger problem

[bonnyman]

The software bug was just one piece of a much bigger problem; I wouldn't want to overstate its' role. There were many other factors; here are just a few:

Poor vegetation management probably played an even bigger role as overloaded power lines warmed up, expanded and sagged into trees and bushes that were supposed to have been cut back.

Poor communications between utilities played a major role.

This whole section of the transmission system was known to be unstable.

An inadequate regulatory structure lacked teeth to deal with known problems.

Lack of adequate transmission line capacity

If all these other problems hadn't been in place, the software bug might never have surfaced. And certainly, the rpoblems would have been contained within a much smaller area -- maybe just First Energy's service area.

An article featured on Slashdot last year lays out the underlying complexity of the power grid very well: "The World's Largest Machine"

Well if you've got no warning...

[mindless4210]

how can you respond to an incident? It just goes to show the need for multiple monitoring systems in mission critical systems.

Re:Software bug was just one part of bigger proble

[Raindance]

I agree that there's more to this than just one line of code, as some folks seem to believe- I think referring to it as 'one bug' is rather misleading.

As well refer to the things leading up to WWII as 'one problem'.

For the 21st century...

[Anonymous Coward]

If a bug exists in the code, but it's never triggered, is it really a bug?

Re:For the 21st century...

[Raven42rac]

Yes, yes it is. If a mime gets hit by a tree in the forest, does anyone care? Sometimes, no matter how much testing you do, shit just happens. It is a fact of life. Show me one perfect, bug free, piece of software. Stuff breaks all the time, we only notice it when it affects us. We take for granted sometimes how good we have it. Power in this country is extremely reliable. We act as if a bomb dropped when the power goes out. Some parts of the world do not have power, clean water, etc. We should think of that before we start whining about having to actually talk to each other, use candles, read books, etc.

Re:For the 21st century...

[timeOday]

I suppose one silver lining in having an outage once a year or so is that it forces us to keep backup systems for hospitals etc in place. If we only lost power once every 10 years, probably nobody at the hospital would even know what to do when power was lost, and people could die. It's just so hard to keep a backup system maintained and working if you are never forced to really use it once in a while. Like planning ahead for a weeklong camping trip, if you don't work up to it by taking shorter trips your chances of being fully prepared are nigh on 0%.

Re:For the 21st century...

[evilviper]

Power in this country is extremely reliable.

Actually, that's statistically untrue. We have, perhaps, the least reliable power system in all the countries of the first-world. Sure, 3rd-world countries have worse-off power systems, but the comparison isn't valid at all.

Some parts of the world do not have power, clean water, etc. We should think of that before we start whining about having to actually talk to each other, use candles, read books, etc.

Since when does the hardship of others make an unreliable power system a plus? Some places may be worse, but so what? We pay a lot for power, and expect our money is being spent on making sure we DO NOT have many outages.

Meanwhile, in California, prices are high, and power was VERY unreliable. "Rolling Blackouts" anyone?

My point is this. If something is broken, we want to fix it. We don't want to sit around saying "Well, it isn't as broke as that one". If we do, pretty soon it will get worse, and worse, and worse, until we have no other countries to point at.

How about our medical system, and water utility? Should we accept thousands of deaths due to malpractice, or contaminated water, by just saying "Well, it's not as bad as country XYZ"? No, I don't think anyone would believe that, but it's really the same thing. Power outages do mean deaths, and do mean losses of lots of money. Businesses can't run, food can't be properly preserved, or even delivered. People die of heat-stroke, or hypothermia due to power loss. Ambulances can't get through dense traffic caused by traffic signals loosing power, etc.

A power outage is a lot more serious than people "whining" about not being able to watch TV... And yet you get moderated up anyhow... Amazing.

Re:For the 21st century...

[Mark_in_Brazil]

Meanwhile, in California, prices are high, and power was VERY unreliable. "Rolling Blackouts" anyone?

Good point. I live in Brazil, and there's a real sick tendency among people here to kiss American ass and fantasize that the United States are a place where everything works perfectly and nobody has to pay for anything. When they do that, I chuckle and point out things like the difference in the electrical power systems in the two countries.
NOTE: I AM NOT SAYING BRAZIL IS BETTER THAN THE USA... JUST THAT IT'S NOT WORSE EITHER.
Brazil's electrical power, as of 2001, was about 97% hydroelectric. Because of years of below-average rainfall, this system was threatened, and in 2001, we were told there might be "rolling blackouts" here (except that the Brazilian government, unlike the US government, was honest enough to call it what it was: power rationing). We ended up not getting any "rolling blackouts," and a regression toward the mean in rainfall has left us sufficiently well off that we don't even have to use the new polluting thermo plants that were built around the time of the crisis. Electrical power here is cheap and reliable, especially compared to places like California, where a lot of my friends had to endure "rolling blackouts" because the folks at the deregulated power companies decided to put more money on their bottom line by not investing in infrastructure upgrades and maintenance. So the execs who made those decisions increased profits in the short term, increasing their bonuses and the value of their stock. When the $#!+ hit the fan, guess who had to pay, both in damages from "rolling blackouts" and in higher rates? The consumers, of course!
The only power problems I've had here in São Paulo were a neighborhood issue, not a city-wide, state-wide, or nation-wide problem. Basically, the new condo across the street overloaded the local grid 3 times in a 2-week span. The worst thing is that the new condo has its own generator, so the newcomers would knock out the neighborhood power and then not even notice, because their generator kicked in. Meanwhile, those of us who had already been in the neighborhood were screwed. Even those problems have been resolved, though. With even more people moving into the new condo, it's been about 6 weeks since we had a problem. The power companies here are pretty efficient. Yeah, I'd have liked for somebody to stop people from moving into the new condo until the local power grid was adequately updated, but they responded pretty quickly once the problem did present itself in an inconvenient way.

--Mark

B Method?

[starseeker]

"the bug was unmasked as a particularly subtle incarnation of a common programming error called a "race condition," triggered on August 14th by a perfect storm of events and alarm conditions on the equipment being monitored. The bug had a window of opportunity measured in milliseconds. "

Isn't this the type of problem the B Method (and maybe the Z language too) are designed to address? Use proof logic initially - once you have decided on a behavior you want, design the system in such a way that it is provable it executes this design.

That doesn't mean the DESIGN is flawless, of course. But if we start engineering software on as many levels as we can, mightn't things improve? Normal software development and testing would never have found a critical bug with rare trigger conditions and a millisecond window. If you need precision on that level, you need to (for starters) to KNOW your implimentation of your design is sound, and preferably the code you are running exactly impliments the proven logic. Isn't this what the B Method was created for?

Re:B Method?

[mccalli]

Isn't this the type of problem the B Method (and maybe the Z language too) are designed to address? Use proof logic initially - once you have decided on a behavior you want, design the system in such a way that it is provable it executes this design.

Ye gods, you've frightened the hell out of me with reference to Z. I'd almost entirely forgotten it, and had hoped its cold corpse would lie in the ground undisturbed, undiscovered and most importantly of all unreferenced until the end of time. Still, "That is not dead which may eternal lie"...

Z is a beautiful way to mathematically prove that you have design bugs at the highest level possible. You can then design your unit tests around those bugs, and confirm that they're valid.

That's it. It provides nothing else that unit testing on its own couldn't do, with the exception of a few salaries and a research grant here and there. Whilst you can mathematically prove implementations of certain designs, the vast majority of designs have more complex interactions. Try using Z for a multithreaded real-time environment for example - my Software Engineering tutor at the time, Iain Sommerville (well known in the field due to his books, oh and 'at the time' would ~1993), basically said that Z just breaks down in those circumstances. I wouldn't know - I personally had no clue how to even make it begin in those circumstances, let alone break down.

Please confine Z to camp-fire ghost stories used to scare new programmers. It always was a living hell, and it really shouldn't be resurrected now.

Cheers,
Ian

Re:B Method?

[Orne]

SCADA systems transport data samples. My company's system collects from several hundred thousands of meters, about half of which are expected to send in a sample about once every 10 seconds, some as fast as once every two seconds. The concept is that you have a communications buffer that collects the data, the link writes to the memory while the other EMS applications (about a dozen) read from the memory.

Now admittedly, FirstEnergy's system is a little smaller in territory, but I wonder if their mergers over the recent years (Cleveland Electric and Ohio Edison became FE, and then proceeded to take Toledo Edison and GPU of PA) have outpaced the collection capabilities of their mainframe (which was already at the end of its life and was scheduled to be replaced). That could account for some of the "slowing" that the G.E. testers said they had to do to make the race condition appear.

Re:B Method?

[Mr. Slippery]

Use proof logic initially - once you have decided on a behavior you want, design the system in such a way that it is provable it executes this design.

Problem is, doing and verifying proofs is just as subject to error as creating and reviewing code. All you've really done is change your symbol set.

Re:B Method?

[bruthasj]

design the system in such a way that it is provable it executes this design

Unfortunately, if you actually come out of the library and the computer labs, software has to be done -- yesterday. Flawless, provable code would cause most software houses to go bankrupt. It's a fact of life...

Re:The problem with SCADA systems

[Vancorps]

This all reminds me of the movie Resident Evil where they shut down power and all the doors unlock when power is restored.

You bring up a great point about failure states. I work for several large hotels and the fire control systems are the ones that alert whenever there is any problem of any kind largely because any problem of any kind needs to be addressed immediately so it makes sense.

I would think power systems would think along the same lines since the odds are, ANY failure whatsoever needs immediate attention of engineers that maintain the system. This is not a requirement for all software but when it comes to such critical services why doesn't everybody do the same practice? It seems so blatently obvious that alarms should have been raised.

Also, in situation's where you don't work on a live environment you can always create a test environment that is for all intensive purposes "live" For web development work I do I have a testing domain which is used to test sites to ensure that because they work here in my lab they will work when I hand them off to the client. Its 100% accurate, I've seen it done with countless other systems, so why wasn't it done here?

The American jackasses who blamed Canada

[Kevin Mitnick]

Did anyone ever retract their statements? I know the NY Mayor was pretty quick to blame us Canucks.

Re:The American jackasses who blamed Canada

[spinkham]

We blame you, you blame the Newfies. It's the pecking order around here, deal with it ;-)

Canada has a history of bad grid control

[Orne]

From the perspective of New York, they saw a surge race through their system East to West, through the choke point into Canada at Niagra station. NY constantly has problems with IMO not following schedules, and from their perspective, this was yet another incident of bad reliability control across the border.

What they didnt know is that the energy was routed through the southern bit of Canada along the lake area, back into the USA in Michigan, to feed all of the communities along the southern shores of the great lakes. The reason this happened is that the coastal towns became electrically isolated from southern ohio because of failures in FirstEnergy territory. I don't think to this day FE has accepted full responsibility for their roles in the failures, something I think should be done with a good house-clearing in their company...

Testing isn't the answer...

[evilviper]

You can't expect just testing to reveal all bugs in a program. Even a simple program would have to be fed completely random data constantly, in every different order and circumstance concievable, for a very long time, to reveal all bugs. That's just not a real option.

The only way to have bug-free software is to write it properly. You have to modularize and simplify everything down to the point that each one is easilly understandable, and it is easy to detect when one is providing a sensless answer (in other words, cross-checking every result). Then, you have to tie them all together in a robust but simple way.

I know it's far easier to say it than do it, but it seems like nobody even tries to do it these days. Even mission-critical systems are commonly built as a single monolithic program, and when you have a lot of things going on within a single program, with no checks of the sanity of the data going into or comming out of each component, there is no way to be 100% certain that the program is theoretically and genuinely perfect. Meanwhile, by modularizing everything, you can PROVE that it is actually perfect.

But this is really just the old Macrokernel vs. Microkernel arguement all over again. A Microkernel can be perfect, while a macrokernel can never be completely bug-free, but people just find the latter to be easier to write, and then spend hundreds times more man-hours finding and removing bugs, rather than spending (less, overall) time doing it correctly in the first place.

Oh yes, almost forgot, IMHO...

Re:Testing isn't the answer...

[Grimmtooth]

Your comments remind me of an old QA maxim: "We can only prove the existance of bugs - not the absense of them."

You invoke the magic buzzwords of "modular design" as if it were a new thing. It isn't. That concept is older - in practice, even - than the median user on /.. Edsger W. Dijkstra was one of the earliest proponents of such coding practices - you can find archives of his papers HERE and see for yourself.

Magic buzzwords can't prevent defects from occurring. QA can't find them all, no matter the budget or amount of time they spend on it. You can only minimize the effects of bugs and put procedures in place to deal with them, programatically and non-programatically.

"Our software contains no known undetected bugs."

Re:Testing isn't the answer...

[Kirill Lokshin]

Meanwhile, by modularizing everything, you can PROVE that it is actually perfect.

Umm, no. Modular design is great for theoretical process correctness, i.e. if a certain input is made to the running program, will it provably produce a certain output. The main problem with this, of course, is it assumes that the program is physically running the whole time.

The systems (I assume) are being used here have to deal with more ephemeral and unpredictable conditions: failing hardware, CPUs going offline in the middle of instructions, random electric interference, etc. The main issue is that the program may not be able to run in its original state, and attempting to recover usually deals with potential data loss, which prevents good theoretical proofs.

Reasons for power blackouts

[pcraven]

I've been reading several papers on this for a grad class I'm taking. One of the several problems is no government control. If a power outage might be prevented by shedding some load (turning out power to some people), no company wants to step up to the plate and be the one to turn out the power to their customers. So they luck out, or they have a massive power outage.

This paper (click on the PDF link) has a good summary of the problems in keeping power outages from happening again.

World's largest machine

[stefanb]

An article featured on Slashdot last year lays out the underlying complexity of the power grid very well: "The World's Largest Machine"

OK, it's nitpicking, but the largest machine is arguably the telephone system. Among other things, it maintains a synchronized clock (8 kHz base), even across oceans and continents.

Re:World's largest machine

[IncohereD]

Among other things, it maintains a synchronized clock (8 kHz base), even across oceans and continents.

It's actually plesiochronus, and only synchronized within certain (relatively large) regions. And I don't know where you're getting that 8 kHz figure from.

Basic relativity (not to mention propogation) will tell you that what you're describing is impossible.

Race conditions are nasty ...

[cagle_.25]

As you programmers all know, avoiding race conditions is really difficult. The fellow Neumann quoted in the article who said

But Peter Neumann, principal scientist at SRI International and moderator of the Risks Digest, says that the root problem is that makers of critical systems aren't availing themselves of a large body of academic research into how to make software bulletproof.

is overly optimistic; it's theoretically impossible to write a general test to find all race conditions in code. This is a variant of the Halting Problem.

Re:Race conditions are nasty ...

[Animats]

it's theoretically impossible to write a general test to find all race conditions in code.

Baloney. It is possible to write programs for which race conditions are undecideable. Such programs are broken. It is possible to write programs for which race condition detection is NP-hard. Such programs are broken if N is large. It is also possible to write programs for which race conditions can be proven to be absent. That's what you want to do.

Actually, it's straightforward to design software to be free of race conditions on a single machine. You then have a deadlock avoidance problem, but deadlocks are easily detected when they occur.

Hardware is routinely designed to be free of race conditions, after all.

Re:Race conditions are nasty ...

[Mr. Slippery]

it's theoretically impossible to write a general test to find all race conditions in code. This is a variant of the Halting Problem.

I doubt PGN was refering to software to test for race conditions; I expect he was alluding to methods for writing code that does not contain them. People have, after all, been thinking about Dining Philosophers for quite a while now, yet coders still do amazingly stupid things with threads.

Re:Race conditions are nasty ...

[platipusrc]

how do you have a large nondeterministic?

hint: NP-hard is a problem that is NP-complete, or worse. An NP-hard problem does not have to be solvable. NP in this context stands for nondeterministic polynomial (with reference to time bounds). NP means that a problem can be solved in polynomial time with an infinitely parallel system. NP-complete problems are at least as hard as all other NP problems.

Sorry, it just bugs me whenever people try to talk about theory of CS and use "non-polynomial" or something else for NP.

Software ENGINEERING

[Anonymous Coward]

If I want to build a large structure (bridge or building) where it is possible that public safety is at issue, I had better have an engineer's signature on the drawings.

This case seems like a real good argument for having the same requirement for software.

Good engineering practice would probably have prevented this. A simple example of such a system would be a burglar/fire alarm panel. The system is self-checking. If any part of the system isn't working (ie. someone cuts a wire), then that causes an alarm.

I realize that there will be strange undetectable bugs in software but if the system as a whole is properly engineered, the system will fail gracefully and safely.

Re:Software ENGINEERING

[Orne]

The two systems you describe are fundamentally different from the design of this alarming system. In fire or safety, the "reading" is the voltage of the closed loop wire itself; 12 volts connected, 0 volts open.

Now imagine if you have a layer in between; you want to monitor the fire status of a complex of warehouses from a single room several miles away. Analog/Digial the signals to all of the individual buildings, transport the data to a common computer, and view the data there. Figure you have several hundred buildings you're watching at once, and now you're getting closer in scale to how the grid dispatchers get their data.

Now imagine that the computer's software back at the main station reads all these meters, and if a line's open (say you're tracking window openings for security), it writes an alarm to a text log on the screen; on a good day, you don't get any alarms. Now suppose the driver that writes the alarms to the screen hangs; since you werent expecting any alarms, you're not that concerned that you aren't seeing anything. That's pretty much what caught FirstEnergy for those 3 hours that afternoon, while the system was failing and they didn't realize they needed to act.

Re:Software ENGINEERING

[Kirill Lokshin]

the system will fail gracefully and safely.

A mission-critical system, by definition, cannot fail "safely", since it must not fail at all.

Re:Software ENGINEERING

[sjames]

At first glance, that can seem like a good idea, but are you prepared to pay for that signoff from each engineer whenever you install a piece of software?

A PE signs off on each particular instance of a design taking intended use, site and other construction into account. If you then build elsewhere, you need a new signoff. If you make any significant change (including adding other structural elements to the design (that is, installing more software), you'll need a new signoff. Add a new network driver, another signoff. Upgrade the CPU? You guessed it!

Some software is poorly designed and crash prone. Other software is well designed but cannot be signed off on because it might be installed on nearly anything that pretends to be a compatible platform.

The one justification for that sort of signoff is in situations where a bug will kill someone. Even then, the system should be divided into critical and auxillary parts to limit what must be signed off on.

Autopilots work that way. You have a small and reletivly simple part that assures safe conditions, is extensively tested, and rarely changed. Another portion is more frequently updated, attempts to optimize the flight and provides a nicer interface. The latter can fail completely and the plane will continue to fly (possibly with poor fuel economy and the pilot navigating manually, but it won't fall out of the sky).

There are many tradeoffs. In some sense, many small distributed systems are more robust than centralized control. However, it's a lot easier to create a chaotic system that way. If you do, you won't know until the system falls into a weird state without warning.

Re:Software ENGINEERING

[iabervon]

One issue is that there is no safe state for the system to go to if the control system breaks down. Bringing the power grid in an area down safely is as hard as bringing it up safely (which, if you remember, took a while) and is harder than just keeping the system running.

The system is full of inductors, whose voltage drop is determined by the change in current through them. If you disconnect a transmission line, suddenly you're trying to change the current to 0, which puts all of the inductors at whatever voltage is necessary to make the current change more slowly. Generally, the way of making the current change more slowly is either to shoot a bolt of lightning across the gap you're creating or to melt your equipment into a conductive lump of metal, but this is only a temporary solution. Instead, the inductors (inside transformers and such) can melt down so that they aren't inductors any more and the current can change more quickly. Of course, when this happens, the next segment of transmission line is now not getting current, so it has the same problem.

The only safe way to bring down the grid is by coordinating with the adjacent grids to carefully remove the load on the line you want to disable; but that's not really an option when the problem is that communication is out.

Bug free!

[Ghoser777]

int main()
{
return 0;
}

Because I have shown you bug free software, does that invalidate the rest of your argument?

Matt Fahrenbacher

Additional Information

[Orne]

Oddly enough, while writing a comment to another user's message, I threw some info in google to learn about FirstEnergy's EMS system, and found this other SecurityFocus story in Feburary 2004, which gives more raw facts than this newer story.

"DiNicola said Thursday that the company, working with GE and energy consultants from Kema Inc., had pinned the trouble on a software glitch by late October and completed its fix by Nov. 19..."

"With the software not functioning properly at that point, data that should have been deleted were instead retained, slowing performance, he said. Similar troubles affected the backup systems. " This dovetails well with why the testers had to "slow" their testing to make the race condition appear.

342 years of online operational hours?

[VoidEngineer]

So, as far as I can figure, there are 24 hours in a day, and 365 days in a year, which equals about 8760 hours in a year (give or take).

Now then, 3 million hours divided by 8760 hours per year equals approximately 342 years, modulo 4070 hours (i.e. approximately 169 days...).

Now then... how the hell do they get the idea that they've been up-and-running for 342 years? Are they counting things in parallel? Even if they were counting end-user operational hours, the number should at least be a couple orders-of-magnitude higher, no?

3M online operational hours sounds like fuddy-duddy accounting to me... although, obviously I haven't looked over the books. I would be interested to see how they came up with this number.

Re:342 years of online operational hours?

[Creepy Crawler]

342/x

x = "how many reactors they have in operation"

Testing vs RTFS. Proprietary vs open.

[SharpFang]

if(int(rand()*1e20)==31337){
blow_up();
} else {
do_your_work();
}

Now I can't imagine amount of testing in proprietary software that could reveal this example of malicious code. In open source one look at the code will reveal it. Of course not all cases are so obvious, but always reading the code should be used together with "testing the software". How do you know lots of proprietary software that IS close-source isn't i.e. a gatweway for terrorists? How do you know biggest companies' stuff isn't all trojans? It wouldn't be hard to hide it. Say your software is kind of server. It does its job okay unless it receives TCP packets starting with certain string. Then it just executes commands contained after that string. Boom. No amount of -testing- will reveal this.
And there are bugs that can be triggered once in several billion cases. Only looking at the code could fix them and explaining "we did a lot of tests" is bullshit.
I put a lot of iron, gum, different materials, C4, glass and some more together and it goes, I call it "a car" and I rode 1000's of kilometers okay. Now no amount of testing in all road conditions will reveal it contains the C4 explosives. Looking under the hood will reveal it really fast.

"We test exhaustively..."

[Fratz]

Um, no you don't. By definition, if you tested exhaustively, you'd have found everything that could possibly go wrong with whatever you tested.

I'm not saying it's always feasible to test exhaustively, but don't say you did when you clearly didn't.

Also: "we had in excess of three million online operational hours in which nothing had ever exercised that bug"

Taken with the "exhaustively" statement, I'm thinking that whoever said these things doesn't understand QA very well. It's easy to write code that works well when everything's good, and it's often just as easy to test that. It's another thing entirely to write code that works well (or fails gracefully) when everything's wrong. And again, it's harder to test that.

Clocks

[Detritus]

That's one reason that I like to put UTC clocks on displays. A quick glance at the clock will tell you if the display subsystem has crashed.

I'm also a big fan of watchdog timers. The process that periodically resets the timer can make all sorts of health and sanity checks.

Re:Clocks

[corngrower]

A watchdog timer on the alarm system (that was deadlocked) would probably have prevented this scenareo. And I also agree that displying clocks on the screen is a good way for the operator so see if the display system is functioning properly.

Not having the display system give a visual indication of stale data was also a deficiency.

There also seems to have been a problem in that the data collection and monitoring portion of the system was held up by a malfunctioning alarm system.

de centralised power

[zogger]

I think the nation/region would be served better if we stepped back a bit and took another look at more decentralised power generation as a full bore government encouraged option. Not as a complete replacement, but frankly, I see no reason we can't have millions more solar panels and wind generators out there. Economy of scale in manufacturing, spurring on even more R&D, etc, works for everything else it appears. And having a lot more points of production, spread out, would help to mitigate cascading failures, especially if islanding was more precise and easier to implement in smaller areas. Wind, were the average wind is adequate enough, is especially cost effective now, approaching coal burning costs per watt. Solar is nice where applicable by climate because it can make use of dead space going to waste, millions of roofs already there.

I like the "not all your eggs in one basket" approach to problems, and I believe in backups for everything.

Re:Software bug was just one part of bigger proble

[Detritus]

They were doing their job, cutting budgets and payroll costs. Oh, you wanted the system to operate reliably too?

Re:Software bug was just one part of bigger proble

[Shakrai]

The software bug was just one piece of a much bigger problem; I wouldn't want to overstate its' role. There were many other factors; here are just a few:

Why don't we point out the real problem that likely caused this to happen. Energy deregulation in the first place.

I know I'll be jumped on by the free market types for daring to suggest this, but I'd rather have a regulated monopoly then a free-market for my life essential services anyday of the week. That article you linked is very interesting reading. Some quotes:

Prior to the implementation of Federal Energy Regulatory Commission Order 888, which greatly expanded electricity trading, the cost of electricity, excluding fuel costs, was gradually falling. However, after Order 888, and some retail deregulation, prices increased by about 10%, costing consumers $20 billion a year.

"Under the new system, the financial incentive was to run things up to the limit of capacity," explains Carreras. In fact, energy companies did more: they gamed the system. Federal investigations later showed that employees of Enron and other energy traders "knowingly and intentionally" filed transmission schedules designed to block competitors' access to the grid and to drive up prices by creating artificial shortages. In California, this behavior resulted in widespread blackouts, the doubling and tripling of retail rates, and eventual costs to ratepayers and taxpayers of more than $30 billion. In the more tightly regulated Eastern Interconnect, retail prices rose less dramatically.

In the four years between the issuance of Order 888 and its full implementation, engineers began to warn that the new rules ignored the physics of the grid. The new policies " do not recognize the single-machine characteristics of the electric-power network," Casazza wrote in 1998. "The new rule balkanized control over the single machine," he explains. "It is like having every player in an orchestra use their own tunes."

Equally important, the frequency stability of the grid rapidly deteriorated, with average hourly frequency deviations from 60 Hz leaping from 1.3 mHz in May 1999, to 4.9 mHz in May 2000, to 7.6 mHz by January 2001. As predicted, the new trading had the effect of overstressing and destabilizing the grid.

Of course it's the first quote that rings true with me. If deregulation is so friggen great then where is the cheap electric? Why can my Village sell me electric for $0.04/kWh with their regulated municipal power authority (while paying their workers Government rates and with Government benefits) when my girlfriend (who lives a whole two miles away) pays $0.14/kWh for electric supplied by a company that is supposedly part of the free market (a company that pays their employees crap and outsources their call center/billing functions to India). What's the problem with that picture?

Before energy deregulation the price of our electric was regulated by the PSC (Public Service Commission) and was fairly stable. The company that had the monopoly in this area made a set amount of profit (it wasn't a bad stock to pick up either -- you knew what you were getting), treated their employees well and charged a fair rate. Nowadays they treat their employees like crap, the stock has tanked because they are eating the price difference from their suppliers (otherwise we'd be paying about $0.20 kWh) and they are being raped by out of state suppliers that bought all of their generation capacity.

In another slightly related story the out of state company that bought one of their power plants sued the local township because they wanted the tax levy on the power plant reduced. They claimed that it wasn't worth what it used to be because they didn't plan on operating it (it was to be backup generation). After a three-year legal battle the township lost (ran out of money to pay the lawyers) and the tax levy was reduced by some 60%. Property and school taxes on

Re:The problem with SCADA systems

[miu]

For web development work I do I have a testing domain which is used to test sites to ensure that because they work here in my lab they will work when I hand them off to the client. Its 100% accurate, I've seen it done with countless other systems, so why wasn't it done here?

Mostly because web systems are still toys compared to real systems.

These systems get real and very intensive testing in labs as close to live as they can get. Even once they knew the conditions and affected subsystems it took the dev and testing teams months to recreate this bug in the lab. The lab is never just like real life, it cannot be - because even real life now is not always the real life of 10 seconds ago.

Re:The problem with SCADA systems

[Kirill Lokshin]

This is exactly the same as software in my industry (HVAC fire/security systems for large buildings), where if you lose communication to a subsystem or the field, you have to raise alarms all over the place.

And perhaps the software in question also tries to do that. However, there are any number of reasons it could still fail.

Consider the following scenario: one software component (a proccess, if you will) is responsible for synchronizing the data between the remote testing station and the local data storage. Another pulls the locally stored data and displays it to the user. The natural place to check for lost comm is in the first component; but if, for some reason, the lost comm causes that component to fail, the second one may not be aware that the locally cached data is not being refreshed (a silly mistake, but I've seen it happen). Furthermore, the user will be unaware that the link failed because the process responsible for generating the notification will no longer be running.

Re:The problem with SCADA systems

[spurdy]

You make a good point, but in my company, we have hundreds of data points reporting continuously. When the communications (telephone company) fails, which it does multiple times every day, you end up with wrong data temporarily. If the operator had to investigate every comm failure, he'd never get anything else done. So, there has to be a threshold somewhere of when does a problem reach a level that it needs to generate an alarm.

Re:Software bug was just one part of bigger proble

[Grayswan]

Why don't we point out the real problem that likely caused this to happen. Energy deregulation in the first place.

I think it is more accurate to say that deregulation enabled, not caused, the problem. Certainly First Energy used deregulation to put in place much of the pieces of the problem. You just don't hear about all the well run deregulated power systems.

Re:The problem with SCADA systems

[fermion]

It kind of depends on how often the out of data conditions occur and how long they occur. My understanding is that the design of proper alarms is actually a complicated security issue, and improper alarms leads to less effective security.

For example, I once worked at a place with many many Window web servers. Every time a server failed, an alarm would sound. But the reason we used Window servers is that they were dirt cheap so we could buy enough to compensate for the expected frequent failures. The result were near constant alarms that were uniformly ignored. Therefore, the alarms resulted in no security benefits. This place had many other example of impressive front door security with nonexistent backdoor security.

It could be that the data was often "not live". Such 'failures' might be due to perfectly legitimate and expected condition. As such, these would not be exception in the sense that it was not unexpected. It is quite possible that the system was designed to have a human check some board on a periodic basis to confirm the age of the data. It may be that as long as an operator did this job once an hour there would be no problem. Some group decided that additional indication would not do any good because the data was so often "not live" that the operators would suffer blindness to the alarm.

Of course we do not know this for sure, but it could happen. But it is a consideration. As another example my check engine light has been on for a long time, and yet the mechanic says that nothing is significantly wrong with the engine. How will I ever trust the light again?

Re:The real problem

[sjames]

From the reports I have seen, other than FE, the various companies did take appropriate action and shed load where necessary, it's just that the situation developed too quickly (from their perspective) and was too large to save by the time they could see it.

The problem was that the grid was running too close to capacity in general. Since the electricity is traveling as fast as any control signal could, it is necessary for the system to be able to tolerate whatever condition may exist long enough for systems to react and get a command transmitted. To make matters worse, you can't just switch off that much current, it takes several seconds for a switch to trip and the arc to be extinguished.

At 50% of design limits, a sudden doubling of demand due to a failure is no big problem (but needs to be dealt with before something else goes wrong), if you're at 90% though, you have a problem.

The real problem is that peak capacity simply isn't there. Our grid does run around 90% during peak load. The question is, are we willing to pay for the extra peak capacity.

California's problems were quite different since it was basically an effort to wring out more profit than existed in the system. THAT is a good reason for regulation. It may be that a more limited form of that is why we don't have more peak capacity, and that needs to be addressed.

bugs are not inevitable

[ummit]

We test exhaustively... I'm not sure that more testing would have revealed that.

For an obscure race condition, this is undoubtedly true.

Unfortunately, that's kind of the nature of software... you may never find the problem.

This is sorta true, sorta false, and definitely misleading.

I don't think that's unique to control systems or any particular vendor software.

No, it's not unique; bugs that may never be found are rampant in most varieties of software. What's false -- tragically, crushingly false -- is the presumption that these unfindable bugs are therefore inevitable. They are not.

If there's a class of bugs that's hard to test for -- and of course there are many such classes -- the prudent thing to do is to find development methodologies that skirt those bugs entirely. If you don't put in so many bugs in the first place, you obviously don't have to work so hard trying to find and fix them.

Re:Statistics

[chgros]

Exhaustive testing, however you wish to define that
Exhaustive \Ex*haust"ive\, a.
Serving or tending to exhaust; exhibiting all the facts or arguments; as, an exhaustive method. Ex*haust"ive*ly, adv.

Basically, it should mean you've tested everything (which is of course impossible in most cases).
The term usually used (and rightfully so) is extensive testing.

Re:The problem with SCADA systems

[Fishead]

Wasn't Chernobyl taken out by a test gone bad?

Testing is all fine and good, but there are always going to be instances where something will remain undetectable for years until circumstances are just right (wrong?)

I am a technician at a plant that makes batteries and we see this all the time.

I remember one time where an operator was cleaning a conveyor with a cloth soaked in Methanol (standard procedure) but forgot about the rag he had left on the underside of the running conveyor. Once the Meth had all evaporated, the dry rag got caught on the conveyor and jammed in the sprocket. At the same instance a valve had opened to fill the electrolyte tank. The jammed sprocket blew a breaker which stopped the machine. The PLC (Programmable Logic Controller) is programmed to keep valves in their current state in the case of an emergency (you kill less operators this way), but in this case it should have closed the valve. The result was a large puddle of nasty smelling, toxic, expensive electrolyte underneath the machine. Much fun.

My point is that as much as we try to make our machines foolproof, there is always at least one fool out there that will one day outsmart you.

Re:The problem with SCADA systems

[wintermute1974]

I agree. These SCADA systems can become quite complex. If you are interested, you can even read General Electric's brochures for the XA/21 system.

Re:Software bug was just one part of bigger proble

[wintermute1974]

You just don't hear about all the well run deregulated power systems.

Yes, we do not hear about them, because they do not exist.

Sure, it was First Energy's lines that failed initially, but if it wasn't First Energy, some other utility would have failed eventually. The engineering and the legal descriptions of the current electrical generation and distriubtion system in North America are at odds with one another.

There's a good technical discussion on the failings of the power grid that may interest you.

Re:Statistics

[aardvarkjoe]

Maybe it just means that they got very tired while testing the software?

Re:Software bug was just one part of bigger proble

[Shakrai]

The Municipals know nothing about running a utility, so they do only what is required to get by

I'm going to call bullshit on that. My Village has been running municipal electric since the 1910s. It is self-sustaining (i.e: takes in enough money to operate without using tax dollars) and geared towards the future. They didn't annex any lines or equipment from private companies -- it was built from the ground up. They don't own their own generator plants anymore (last one went offline in the 50s) -- they buy it from the wholesale grid just like everybody else. And yet somehow they are able to provide it at $0.04 kWh without screwing over their employees or customers. This municipal grid feeds everybody in town from houses to streetlights to factories. I'll grant you they don't have to serve a rural area but rural areas aren't automatically four to five times as expensive -- if that was the case then why do my parents, girlfriend, grandparents and friends all pay the same high rate even though they all live in suburban or urban areas with the exception of Grandma?

I find it hard to feel sorry for that one township you sited, since there are many townships without that high taxed power plant around.

Perhaps you'd feel for them more if it was your friends and family that lived there. Perhaps you'd feel for them if half of them had previously worked there before being laid off by the company that bought the plant and screwed the town over. Then the company fired the plant back up and brought in it's own people from out of state to run things. But that's ok -- last I heard New York state was going to go after them. Care to place bets on who will run out of money first in that legal battle?

Wholesale rates have gone up everywhere. I live in an area where there never was regulation, and we face exactly the same higher rates. Coal prices are higher.

So coal prices are the reason why Enron and it's buddies were slashing power production at their plants in California to drive up the wholesale prices? I wish my state would just tell the Feds to fuck off and regulate our own power industry. Everybody was better served when it was regulated -- from the power company itself (I never heard them complaining when they were posting a 15-20% profit margin) to the consumers. The product was more reliable. Those are facts. I just don't think it's a good idea to allow essential services that you really can't get elsewhere to be run by unregulated industries. What's your option if your power company is screwing you (and they are screwing you in all likelihood because if they don't they won't survive because they are being screwed by the wholesalers)? Not using electric? Try that one in a New England winter.

While I'm on this rant I might also point out the phone and cable companies. Before the deregulation of the phone companies my standard phone service was $15-$20 (before long distance charges). Now it's $35 -- or would be if I still had a landline phone. Before the deregulation of the cable industry (forced on my state by the Feds) we had tons of local cable companies and basic cable (50-70 channels depending on where you lived) was $15 a month. Now with Time Warner it's $45 a month. The only thing you can count on Time Warner for is to raise their rates once a year. You can set your watch by it.

Re:The problem with SCADA systems

[Vancorps]

Web systems were but one example. I'll through another much more complex example. Take DNA from bacteria and splice it with stem cells to produce nerves much more resistent to damage. You are talking thousands about thousands of long protein strands most of which you have no idea what perform what task. Do this without destroying the cell. When you are done with that test you move on to a more complex test until ultimately you are ready to do it with humans, at which time you can accurately predict exactly what it will do. Yes there are occasions when that doesn't happen that way but that is usually because something was missed in the testing procedure.

The elitism seen here is incredible, just because a system in and of itself isn't complex doesn't mean you can take stock of how they manage. Although personally I'm about to design a call center application for Mercedes that will be used by hundreds of thousands of people. This system can get quite complex albeit, not as important as a power system.

When it comes to troubleshooting systems you always have the option of making an exact scale model. You scale it up for more precision. This is a simple concept and apparently a lot of people think just because a system is complex and antiquated the same ideas can't apply.

Re:You can't simulate the real world

[Vancorps]

Forgot rats, for some reason they likes to chew cables.

Now, for an example. I stress tested a database I am in the process of building for Mercedes, I made the machine come to crawl. I did it to a dual cpu server, a quad cpu server, and a 16 cpu server. Guess what? They all behaved exactly the same as the system grew. Now scale it up to the DB/2 cluster that it will actually be working on. I do the same thing and guess what? Yep, the exact same result.

If testing fails to produce an outcome that brings a fault then there is a flaw in the testing procedure. The real world can have more connections, but I don't care, software can be 100% bug free. I constantly hear programmers saying that every complex piece of software will have bugs. Its poor planning, and management, even if the programmer is incredibly gifted they will make mistakes without proper planning.

Yes, in the real world there are unforeseen variables but as systems because critical they should be undergoing more testing to ensure such things don't occur. But as I said in my original post. One outage in recent years is hardly a trend and is more likely just blown out of proportion.

Re:The problem with SCADA systems

[miu]

When it comes to troubleshooting systems you always have the option of making an exact scale model. You scale it up for more precision. This is a simple concept and apparently a lot of people think just because a system is complex and antiquated the same ideas can't apply.

Even if you could create a model to test with that is identical to the live system you cannot test every possible situation which can occur in the real world. Integration testing can only test those things which can be envisioned by those responsible for testing.

You absolutely do the best testing you can, unit test every piece of functionality, test subsystems and whole systems in integration testing, but you will never test every single possibility. The more complex (and antiquated) the system, the greater the number of interactions, and the greater the potential for bugs. I'm convinced that there are bugs lurking in every piece of hardware and software I use, the conditions under which those bugs manifest may have never occurred, but they are there.

I'm not fatalistic about software quality, and I don't disagree that we need to test better, but complexity to testing difficulty is not linear and I dislike seeing it trivialized. People who underestimate the difference between a system with 100 parts and 1000 parts are in for a rough time.

Testing cannot guarantee systems

[Goonie]

If testing fails to produce an outcome that brings a fault then there is a flaw in the testing procedure. The real world can have more connections, but I don't care, software can be 100% bug free.

The first thing they teach you in a software testing course is that testing cannot guarantee the absence of bugs. The only way you can guarantee, through testing alone, that your program is error-free is to exhaustively test every possible "input" (combination of external inputs and internal state) and check them. When was the last time you wrote a program with a finite (and tractable) input space?

If you need 100% reliable programs, you'll need to prove them correct, and that's enormously difficult to do, and doesn't help if the bug is the result of a flaw in the program's requirement specification rather than an incorrect implementation of that specification.

What testing *can* do is provide estimates of a system's reliability, and in the real world that's all you're going to get.

Re:Software bug was just one part of bigger proble

[Shakrai]

Well, let's handle this first. Your municipal can offer lower rates because you're paying more in taxes to subsidize them. You pay local, state, and federal taxes which then go to artificially lower the up-front costs you pay for electricity. But it is not necessarily cheaper.

Bzzzt wrong answer. My municipal power agency has been self-sustaining since 1920. They don't take in any tax dollars -- they run it all on the money they take in. Sure it's a Government run Agency so it can't make a profit (though they do take in extra cash for a rainy day fund) -- but for the sake of the argument if they increased prices 50% (to make a profit) they'd still be cheaper then the non-municipal options.

If it's not taxes, then the municipal funds itself by offering bonds, which then pushes the higher costs onto future subscribers.

Wrong again. The last bond they issued was back in the 1950s to build a new substation. The Agency started in the 1900s off tax dollars with a charter to provide street lighting. Over time they hooked up private customers (the infrastructure was already in place) and became self-sustaining. Perhaps that's the exception rather then the rule but you shouldn't go painting all municipal power with a broad brush of "You are just being screwed on your taxes" or what not.

Enron is the exception, and not the norm. Not many companies operate like Enron did, or was as unethical they were.

Really? Did you bother to read the story about the power plant in a local township near me? After they won their petty tax battle by exhausting the town's financial resources they fired the plant back up with out of state employees that they brought in. Sure we could rehire the local people that used to work there but they actually fought us on our tax levy so fuck em! I hope NYS shoves it up their ass -- they are going after them last I heard and something tells me that NYS won't run out of money like the township did.

I think we can all agree that unethical behavior, ignorance, and incompetence are not limited to private corporations, but government agencies, municipal authorities also exhibit those human qualities.

Your point?

btw, nice strawman, mentioning outsourcing while talking about a deregulated power company. sure to get a raise, but can we keep the logical fallacies to a minimum please? thanks

Why not? It's a valid point. Our power company (which was always a publicly held company) used to make enough profit that they could hire local people and pay them a decent (some would say too high but that's another story) wage. Now that they were forced to sell off their generation capacity they are being raked over the coals by the out of state suppliers and profits are a thing of the past.

So how did they respond? By laying off as many workers as possible and outsourcing whatever they could. And they still aren't back in the black. The PSC isn't going to let them charge the $0.20 kWh it would cost to put them in the black (why should they? All the money would just be leaving NYS) so it's a lose-lose battle for all involved. The customers get screwed, the employees get screwed, the townships get screwed and the shareholders (of the power company) get screwed. The only people who are winning are the shareholders of the out of state energy company that's screwing us over. The only reason it's not as bad as it was in California is because NYS has access to cheap hydroelectric power from Canada. That's the only thing keeping them from screwing us completely -- and it's the only thing keeping our power companies solvent. Thank god the Canadian companies at least have some ethics and responsibility.

So keep advocating your deregulated industry. I'm waiting for individual states to just start regulating it on their own. It wouldn't be the first time.