Air Traffic Snafu: FAA System Runs Out of Memory

minstrelmike writes: Over the weekend, hundreds of flights were delayed or canceled in the Washington, D.C. area after air traffic systems malfunctioned. Now, the FAA says the problem was related to a recent software upgrade at a local radar facility. The software had been upgraded to display customized windows of reference data that were supposed to disappear once deleted. Unfortunately, the systems ended up running out of memory. The FAA's report is vague about whether it was operator error or software error: "... as controllers adjusted their unique settings, those changes remained in memory until the storage limit was filled." Wonder what programming language they used?

Comments

Software error ...

[gstoddart]

You can make the argument that if the software allowed the operators to crash the system, it's a software fault.

You can also make the argument that stuff like this should have been tested in parallel with the live system so this wasn't a possibility.

I mean, my god, what are the change management and testing practices which allowed this to only be discovered in your real system?

I've been around a few systems which had to do with aircraft ... and the rules and practices surrounding them are pretty paranoid and rigorous, because the stakes are so high. For an actual air traffic system I'm stunned this happened.

I guess I'm not surprised, but I am stunned.

Re:Software error ...

[SpeedBump0619]

Professionals make mistakes. Garbage collection is a useful tool to make it more difficult to screw up.

I get this. And as a software engineer I fully agree. However, in practical terms, there shouldn't be any dynamic memory management happening at all.

It's a real-time system. It *must* interact, on time, with all the planes that are in it's domain. That should be a bounded, predictable load, or there's no way to guarantee responsiveness. Given that, an analysis should have been done on the maximum number of elements the system supported. Those elements should have been preallocated (into a pool if you want to treat them "dynamically") before actual operation began. If/When the pool allocator ran out of items it should do two things: allocate (dynamically) more, and scream bloody murder to everyone who would listen regarding the unexpected allocation.

This is (one of) the reason(s) I generally haven't liked garbage collected languages for real time systems. There's rarely ever a way to guard against unexpected allocations, because *every* allocation is blind.

Re:Software error ...

[phantomfive]

You are trying to be sarcastic, but the MISRA standard for embedded systems includes these rules:

1) absolutely no recursion. it could lead to stack overflows.
2) absolutely no local variables. it could lead to stack overflows.
3) absolutely no use of of malloc or free. it could lead to stack overflows.

So yeah, that has been an accepted approach for many years.

Re:Software error ...

[phantomfive]

Garbage collection is a useful tool to make it more difficult to screw up.

Recently I've seen a lot of memory leaks in Java and Javascript. People stick things in a hash table or a queue, then forget to remove them (angular.js also has gotchas to watch for avoiding memory leaks). Because programmers in those languages don't think about memory, they end up with more memory leaks than programmers in C.

For a system that needs high reliability, garbage collection is not the answer, and can make things worse.

Language: ADA

[JumboMessiah]

While everyone speculates on GC vs heap vs what flavor is my coffee, ERAM approach systems use ADA as the language of choice.

reference

And the language is......

[jeremyp]

Ada and Java apparently

http://dl.acm.org/citation.cfm...