Slashdot Mirror


Missing Files Blamed For Deadly A400M Crash

An anonymous reader writes: Think you had a bad day when your software drivers go missing? Rejoice, you get to live! A fatal A400M crash was linked to data-wipe mistake during an engine software update. A military plane crash in Spain was probably caused by computer files being accidentally wiped from three of its engines, according to investigators. Plane-maker Airbus discovered anomalies in the A400M's data logs after the crash, suggesting a software fault. And it has now emerged that Spanish investigators suspect files needed to interpret its engine readings had been deleted by mistake.This would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.

18 of 253 comments (clear)

  1. Good god. by Anonymous Coward · · Score: 5, Insightful

    Is it so hard to have a integrity check and diagnostic set run as part of the preflight checks? If you can place hundreds of miles of wire and know what's what, surely they have computer engineers competent enough to make something like this to catch such glaring errors.

    1. Re:Good god. by Anonymous Coward · · Score: 5, Insightful

      We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have. In this fast-paced road of cutting costs and letting the marketing group run the show, the pressure to get product out the door as quickly as possible no matter what is unstoppable for software in particular, but really almost anything that is able to be 'patched' later. Making consumers into your beta testers is douche-y enough, but doing it when lives are at stake should be punished as criminal and in an extremely harsh and public way.

    2. Re:Good god. by fuzzyfuzzyfungus · · Score: 4, Insightful

      What's surprising to me is not merely that; but if the calibration data are so important that the engine shuts down without them, how did the aircraft take off?

      If the calibration data are nice, good for fuel economy, improve reliability, etc. you'd expect things to continue working without them, albeit possibly not as well as the manual specifies.

      If the calibration data are Absolutely Vital Lest The Engine Throw A Propellor Right Through The Cockpit, or something of that nature, how did the aircraft allow you to take off with 75% of the data missing? An actual error handling arrrangement would, of course, be in good taste; but even without one I would have (naively, apparently) expected the situation to take one of two courses: if the data are semi-optional, things would work, if perhaps not well. If they are Vital, attempting to get off the ground would have failed. Successful takeoff, followed by shutdown and fiery death, though, seems weird.

    3. Re:Good god. by ChumpusRex2003 · · Score: 3, Insightful

      My guess is that rather than "files" per se, these are look-up tables which were statically linked into the binary.

      On this type of safety critical application, it's a key design aim to avoid code which might fail or throw an exception at runtime. So, rather than load data from a file, which could fail due to a memory allocation failure, a file system failure, etc. the relevant data is static linked, so if the executable successfully launches, it cannot fail to have the data available.

      I don't know what these tables might have been mapping, but conceivably if they torque tuning parameters, the engine might still have run if the data was all NULLs, but delivered the incorrect torque in response to control inputs. Of course, if the missing data was things like fueling data, then the engine may have failed to start.

    4. Re:Good god. by TWX · · Score: 4, Insightful

      Case in point, the Toyota vehicle acceleration problem.

      --
      Do not look into laser with remaining eye.
    5. Re:Good god. by Anonymous Coward · · Score: 5, Insightful

      Read the article... the warning was not designed to kick in until the aircraft was at an altitude of 400ft (120m).

      Not only do you not know you have a problem until are in the air. You don't know you have a catastrophic problem until you are at an unsurvivable altitude. Too low to effectively use a parachute. Too high to just 'jump out' or belly-land it.

      The worst thing is... a committee signed off that this was an 'acceptable risk'. Members of that committee should be brought of up on criminal negligence and manslaughter charges.

      Not a Luddite, but give me my bicycle back...

    6. Re:Good god. by joh · · Score: 4, Insightful

      We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have. In this fast-paced road of cutting costs and letting the marketing group run the show, the pressure to get product out the door as quickly as possible no matter what is unstoppable for software in particular, but really almost anything that is able to be 'patched' later. Making consumers into your beta testers is douche-y enough, but doing it when lives are at stake should be punished as criminal and in an extremely harsh and public way.

      As far as I know aerospace software is far away from what you describe. Of course you're right if you say that these things are a reason for problems, but THIS is very well understood and usually software for planes is nothing like a consumer product.

      They screwed up, yes, but if they would be "punished as criminal and in an extremely harsh and public way" nobody would ever do anything useful anymore. The problems leading to this crash have to be analyzed and understood and then they have to make sure that the same thing can't happen again.

      But of course: If this was due to someone not following procedures or messing around with maintenance this can (and will) have consequences. I'm also pretty sure that one or more people will lose their job over that.

      But if you really think you can make shit never happen and things working 100% all the time by "hard punishment" you're just wrong.

    7. Re:Good god. by TWX · · Score: 5, Insightful

      If I remember right, Toyota got into trouble in court when the firmware provided to the investigators did not match the firmware in the vehicles. Toyota never did provide the real code if memory serves.

      --
      Do not look into laser with remaining eye.
    8. Re:Good god. by Anonymous Coward · · Score: 2, Insightful

      >(yeah, I know, that means no left-foot braking, but if you're doing that in an SUV, you're probably doing it wrong).

      Sooooooo... no offroading for your SUV?

    9. Re:Good god. by tlhIngan · · Score: 5, Insightful

      They screwed up, yes, but if they would be "punished as criminal and in an extremely harsh and public way" nobody would ever do anything useful anymore. The problems leading to this crash have to be analyzed and understood and then they have to make sure that the same thing can't happen again.

      You know, when an accident happens, the safety board (NTSB, TSB, BEA, etc) interviews are actually privileged information. As in, if you're being interviewed by the safety board, anything you say cannot be used as evidence against you.

      It's a privilege that the safety boards all fight for.

      The reason for this is the safety board's goal is to not find fault, but to find solutions to preventing it from happening again. Doesn't matter if someone hit a button that said "Crash this plane" and pushed it on purpose. They know that if the interviews were not privileged communications, no one would speak to them for fear of self-incrimination. And when that happens, everyone clams up, and you can't figure out why an accident happened or make recommendations to prevent the issue the next time it happens.

      This is especially more so when most complex accidents are a chain of events - this happened, then that happened, then this next thing, plus X, Y and Z and if any of them didn't occur, the accident wouldn't have happened. Almost never is it the result of one definitive action.

    10. Re:Good god. by Anonymous Coward · · Score: 3, Insightful

      You clearly have not done the SAAB slide to get around a corner faster. You do not press the clutch or change gears during the maneuver as the whole point is to apply power during the turn. Although heel-toe, or fat-footing works to apply both throttle and brake, it is not as controllable.

      Sadly many use the left foot for braking under normal circumstances as they are unfamiliar with a manual transmission.

  2. BIST - Built In Self Test by presidenteloco · · Score: 4, Insightful

    My printer at home does it every time it starts up.

    Too bad the airplane doesn't.

    I guess production delays are more expensive than debugging-by-crash. Sad.

    --

    Where are we going and why are we in a handbasket?
  3. Return codes? by cfalcon · · Score: 5, Insightful

    This is a tragedy, but since we're on a tech site, lets talk tech.

    Return values are handled oddly in pretty much every major language. Many API calls want to return something simple- int or bool- and if anything is more complex than that, generally require an actual data structure to be returned, often as a reference. This means that the "I didn't do this" action has a variety of ways to be be passed back- none of them even close to standard.

    If something returns a distance, magnitude, or size, "0" normally means "Error, nothing happened" which is often the same as "Sure, I wrote 0 bytes. Really."
    If something needs to distinguish between success ("I did the thing 0 times as requested" and failure "I couldn't do the thing because of an error condition"), then sometimes a -1 is returned, or an exception thrown, or something else.

    In this plane, something was, at some point, responsible for getting data about the engines. Likely, this happened in layers, each one having access to the results of the lower pieces. One of those pieces had the task of parsing those files.

    So EITHER someone (process, program, whatever) meant to say "This is a problem" and instead said "Here's some default data", OR someone ELSE in that chain of commands (process, program, whatever) has a default for a "This is a problem" result to use as a failsafe, and it was never tested or never communicated up.

    We probably won't get the technical details that go from "files missing" to "engines don't work". Certainly, several level of software or hardware could allow for any number of workarounds in this case, and I'm sure they have a complex system and this was some eventuality that was hard to test for.

    Still, interesting to think about the error return methodology, and how it's so different everywhere in CS.

  4. The last word... by Unknown74 · · Score: 4, Insightful

    " The more they overthink the plumbing, the easier it is to stop up the drain. " - Montgomery Scott, Star Trek III

  5. Re:This is what happens when you use Luddite softw by schlachter · · Score: 4, Insightful

    WTF? No automated system check to determine if all needed files are present before flying??!

    --
    My God can beat up your God. Just kidding...don't take offense. I know there's no God.
  6. Re:It's time to regulate software by Anonymous Coward · · Score: 2, Insightful

    Impractical.

    Suppose a developer uses GCC to compile code. Does the developer need to prove the correctness of GCC?

    What about the Windows OS? Or the Linux OS?

    And software moves so readily from place to place and is so easily incorporated into other projects that it's difficult to imagine a project in a safety critical environment being written completely de novo.

    Had the software (and hardware, for that matter) industry been held accountable from the beginning, we wouldn't have these problems today.

    It's time computer engineering grew up. Sure, regulation will slow the pace of progress and make things difficult for entrepreneurs, but programmers shouldn't make their money off the backs of those injured by their incompetence. Other industries labor under very strict regulation. There are literally thousands of regulations just for banking, and we saw what happened when things were simplified.

    Only strict government oversight of the software industry can prevent injuries to the public.

  7. Re:It's time to regulate software by Anonymous Coward · · Score: 2, Insightful

    Re: Banking, saying "simplified" is over-simplifying. Oversight by the CFTC was effectively removed by the Bush administration. That's what gave the banks free reign to give million dollar mortgages to Walmart greeters. Then they sliced and diced that crap with good loans, packaged it up as investable instruments on Wall St. and then sold it to the nice folks in Iceland (among many others).

    Of course Iceland didn't know that the ratings agencies were also in on the scam. And not many knew about the $500T in notional value swaps that leveraged the crap at 100:1 ratios. Much of which still sits as off-balance sheet assets that will never be unwound.

    That's why you regulate the piss out of the banks and Wall St. It's really not about politics, it's about keeping the cobra in a closed box and never trust him outside of it.

  8. Re:This is what happens when you use Luddite softw by x0ra · · Score: 3, Insightful

    So it would probably have worked, and not crash because someone was using tr(1) to parse some output in an overly complicated shell startup system...