Blackout Cause: Buggy Code

blanca writes "The big northeast blackout from last summer was caused in part by a software bug in an energy managment system sold by General Electic, according to a story on SecurityFocus. The bug meant that a computerized alarm that should have been triggered never went off, hindering FirstEnergy's response to the train of events that lead to the cascading blackout. Investigators found the bug in a intensive code audit following the outage, and a patch is now available."

Comments

Uh...

[Short Circuit]

Didn't the story used to be that after a tech maintenenced the machine, he forgot to re-enable an alarm?

Re:Development vs Engineering

[Jeff DeMaagd]

I'd sort of tend to agree, although under your standards, the stuff I do as an EE really would fit under development, we don't have the budget to send out for external certification and external testing. No biggie, I guess I can live with being a hardware developer.

Is it true that some states have prohibited Microsoft from issuing MSCEs? I heard this somewhere but I can't remember. Something about Microsoft not having the authority to certify engineers.

Re:Hmm

[A55M0NKEY]

Once upon a time, there was a power grid without any software. This is true because electricity predates computers. What did they do then?

I bet they had much wider safety margins built into the system which prevented blackouts. But these safety margins probably cost money ( I say this without knowing a thing about the electrical system ) they probably mean a less efficient use of resources. So power companies buy GE's software. They don't buy it so that they can have an added measure of blackout prevention, they buy it because it enables them to cut out expensive/inefficient safety margins without (supposedly) sacrificing reliability. They do this to lower their cost of providing electricity to you.

What about the actual Engineers involved?

[GoofyBoy]

The software handled one part of the electrical system involved.

What about a good Electrical/Mechanical/Civil Engineering solution that would have prevented it from cascading through different systems / electrical companies / countries?

One piece of software which didn't raise an alarm is shocking. The fact that it cascaded over such a wide area is simply mind blowing.

Before we talk about "software engineers" how about talking about "traditional engineers" and their role in this massive failure?

Re:Development vs Engineering

[Anonymous Coward]

In Canada, "Engineer" is a protected term, like "Doctor."

Doctor is not a protected term. Perhaps you mean "Medical Doctor"? There are lots of non-medical doctors.

I was arguing once with a MD friend of mine who thought that PhDs (like myself) don't have the right to call themselves Doctor. I explained that while medicince has been around for a very long time, the degree of MD has not. PhDs degrees have a much longer history than MD degrees.

It gets very funny when another friend of mine (who has a PhD in nursing) is called "Dr" in her hospital.

Re:50MV arc'd to a tree

[plover]

My property abuts a set of high voltage transmission lines. (I'm about three miles from a coal plant.) The lines cut a long, skinny park through my city. The plat for the site shows a 200 foot wide easement, which is about 30 meters to the property on either edge of the park. I've never measured the height of the towers, but my rough guess is that the line itself is perhaps 25 meters above ground. That puts the line itself about 39 meters from the edge of my property.

The land beneath the lines was clear-cut about 12 years ago. But there are now trees under this line that are about 10 meters high.

Years ago when my wife was concerned about "power line emissions" the power company loaned her a meter that showed "electrical fields." I don't remember the scale, or even what it was supposed to measure, but I do remember that we had to actually get about 200 feet from the wire before the field from the line stopped affecting the meter. (Yes, on a humid summer day I once stood in my back yard with a neon bulb and caused it to illuminate by simply dangling a three foot wire from one lead and touching the other.) I had always assumed it was a 750kV line, and that the 100 foot easement was more than sufficient. Now, I wonder. Hey, maybe this is enough of an excuse to go out and get one of those IKE toys!

Re:Development vs Engineering

[Tassach]

I like to think I'm an engineer, not a developer. The problem is not that I don't know how to do good SW engineering, it that I'm usually not allowed to do good SW engineering. Good engineering is expensive in terms of time and money. The people who sign the checks aren't usually willing to pay for it and aren't willing to wait. The sad part is that they're often right: if you can't afford to wait, and you can't afford to pay the price, you have to settle for what you can get and hope that it's good enough to keep you moving forward.

You have 4 main variables in the software development equasion: Time, Quality, Functionality, and Efficiency. Notice that we only measure time, not man-hours or monetarycost. As we know from reading The Mythical Man-Month , we cannot reduce time by adding more people or by spending more money. While we list efficiency as a variable, we really have to treat it as a constant within the scope of a single release cycle. Improvements in efficency are generally very gradual and incremental, and for the most part cannot be effectively implemented in the middle of a release cycle.

I postulate that Time is directly proportional to the product of Quality, Functionality, and Efficiency [T = EQF]. Since E is constant within the scope of a single release, we can't use process improvements or similar techniques to improve quality in the short term. Assuming our goal is to improve quality, we either have to decrease functionality or increase time. Since monetary cost is directly proportional to time (time is money!), managers are very reluctant to give you more time. Furthermore, we are frequently under hard time constraints due to contractual obligations or market pressure. If we can't change time, we either have to sacrifice quality or functionality. Missing functionality is very obvious, whereas low quality isn't necessarily noticable in the short term, so it should be no suprise that quality is almost always takes the back seat to functionality.