Slashdot Mirror

← Back to Stories (view on slashdot.org)

Air Traffic Snafu: FAA System Runs Out of Memory

Posted by Soulskill on from the must-run-in-chrome dept.

minstrelmike writes: Over the weekend, hundreds of flights were delayed or canceled in the Washington, D.C. area after air traffic systems malfunctioned. Now, the FAA says the problem was related to a recent software upgrade at a local radar facility. The software had been upgraded to display customized windows of reference data that were supposed to disappear once deleted. Unfortunately, the systems ended up running out of memory. The FAA's report is vague about whether it was operator error or software error: "... as controllers adjusted their unique settings, those changes remained in memory until the storage limit was filled." Wonder what programming language they used?

10 of 234 comments (clear)

  1. Software error ... by gstoddart · · Score: 5, Informative

    You can make the argument that if the software allowed the operators to crash the system, it's a software fault.

    You can also make the argument that stuff like this should have been tested in parallel with the live system so this wasn't a possibility.

    I mean, my god, what are the change management and testing practices which allowed this to only be discovered in your real system?

    I've been around a few systems which had to do with aircraft ... and the rules and practices surrounding them are pretty paranoid and rigorous, because the stakes are so high. For an actual air traffic system I'm stunned this happened.

    I guess I'm not surprised, but I am stunned.

    --
    Lost at C:>. Found at C.
    1. Re:Software error ... by SpeedBump0619 · · Score: 4, Informative

      Professionals make mistakes. Garbage collection is a useful tool to make it more difficult to screw up.

      I get this. And as a software engineer I fully agree. However, in practical terms, there shouldn't be any dynamic memory management happening at all.

      It's a real-time system. It *must* interact, on time, with all the planes that are in it's domain. That should be a bounded, predictable load, or there's no way to guarantee responsiveness. Given that, an analysis should have been done on the maximum number of elements the system supported. Those elements should have been preallocated (into a pool if you want to treat them "dynamically") before actual operation began. If/When the pool allocator ran out of items it should do two things: allocate (dynamically) more, and scream bloody murder to everyone who would listen regarding the unexpected allocation.

      This is (one of) the reason(s) I generally haven't liked garbage collected languages for real time systems. There's rarely ever a way to guard against unexpected allocations, because *every* allocation is blind.

    2. Re:Software error ... by tomhath · · Score: 2, Informative

      The civilian aircraft control system has been chronically underfunded for decades, since Reagan fired PATCO.

      Reagan initiated and appropriately funded a complete overhaul of the control system.

      The illegal strike by the air traffic controllers is irrelevant.

    3. Re:Software error ... by phantomfive · · Score: 4, Informative

      You are trying to be sarcastic, but the MISRA standard for embedded systems includes these rules:

      1) absolutely no recursion. it could lead to stack overflows.
      2) absolutely no local variables. it could lead to stack overflows.
      3) absolutely no use of of malloc or free. it could lead to stack overflows.

      So yeah, that has been an accepted approach for many years.

      --
      "First they came for the slanderers and i said nothing."
    4. Re:Software error ... by phantomfive · · Score: 4, Informative

      Garbage collection is a useful tool to make it more difficult to screw up.

      Recently I've seen a lot of memory leaks in Java and Javascript. People stick things in a hash table or a queue, then forget to remove them (angular.js also has gotchas to watch for avoiding memory leaks). Because programmers in those languages don't think about memory, they end up with more memory leaks than programmers in C.

      For a system that needs high reliability, garbage collection is not the answer, and can make things worse.

      --
      "First they came for the slanderers and i said nothing."
  2. Language: ADA by JumboMessiah · · Score: 5, Informative

    While everyone speculates on GC vs heap vs what flavor is my coffee, ERAM approach systems use ADA as the language of choice.

    reference

  3. You Said by dcw3 · · Score: 2, Informative

    But, you said that 8G was enough!

    --
    Just another day in Paradise
  4. Re:Implemented in Ada 2005 by Anonymous Coward · · Score: 2, Informative

    The backend code is implemented in Ada but all of the display code is implemented in a mix of C and C++

  5. And the language is...... by jeremyp · · Score: 4, Informative

    Ada and Java apparently

    http://dl.acm.org/citation.cfm...

    --
    All I want is a secure system where it's easy to do anything I want. Is that too much to ask ~~ Randall Munroe
  6. Re:But, but, but... by Richard+Steiner · · Score: 2, Informative

    One advantage of many airline online transaction systems: An applications programmer cannot do a malloc equivalent.

    Programs are created with a fixed memory size, and complex applications are simply a series of program modules which pass data between each other via common memory areas or memory-mapped files.

    Memory leaks in such an environment are quite rare.

    --
    Mainframe/UNIX Bit Twiddler and long time Windows/Linux Hobbyist.
    The Theorem Theorem: If If, Then Then.