Slashdot Mirror
Missing Files Blamed For Deadly A400M Crash
An anonymous reader writes: Think you had a bad day when your software drivers go missing? Rejoice, you get to live! A fatal A400M crash was linked to data-wipe mistake during an engine software update. A military plane crash in Spain was probably caused by computer files being accidentally wiped from three of its engines, according to investigators. Plane-maker Airbus discovered anomalies in the A400M's data logs after the crash, suggesting a software fault. And it has now emerged that Spanish investigators suspect files needed to interpret its engine readings had been deleted by mistake.This would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.
Done did it.
Is it so hard to have a integrity check and diagnostic set run as part of the preflight checks? If you can place hundreds of miles of wire and know what's what, surely they have computer engineers competent enough to make something like this to catch such glaring errors.
Tuesday is crash-day, oops I meant patch-day.
The engineers probably did one but obviously it wasn't good enough. Software fails, but you can design it to fail in safe ways.
I think Facebook and cellphones have caused our attention span to fall so short that people can't even finish a line of code ;)
Let's all sue facebook and cellphone companies for the reduction in IQ they are responsible for and the related disasters they are now causing do to this ever worsening trend of brain lapses. This can only get worse as more students with lower IQs and more aptitude to be distracted causes them to be unable to comprehend the consequences of their coding/actions/thoughts/etc.
Squirrel!
Come on, folks. Turn the power on to the engine controllers at the flight line and the status display should have been flashing warnings. Nobody should have even started this thing.
Have gnu, will travel.
Remember when you could start your car with the turn of a key and not have to worry about a software update refuckulating your engine timing system? Adding computers to EVERYTHING is a really bad idea, especially because PEOPLE are required to write the code that runs the software that controls X Y and Z systems on the engine, chassis, and security systems.
Depressingly, that might actually be true.
Not because of 'apps' of course; but because no self-respecting consumer OS would fail to cryptographically verify the execution environment(lest some precious 'premium content' be absconded with by pirates) and an entire missing file probably would have caused the aircraft to refuse to move until taken back to Airbus HQ for re-blessing by the vender.
They don't succeed against motivated pirates, of course; but this is one area where consumer software vendors do actually give a fuck. If people believed that a sabotaged voting machine or a defective ECU could pirate Blu-rays, we'd live in a safer world.
You'd think there would be some kind of checks in place that wouldn't allow the plane to operate when critical files are missing. Or that the files couldn't be deleted.
Stories like these are the reason I can't believe auto manufacturers are even considering being able to push updates to cars. The checks in place for aircraft hardware is extremely rigorous. Pretty much every nut and bolt has a complete history log. If this kind of thing can happen on an aircraft, what happens when some weird conditions occur that cause brakes to fail in an automotive update? That's a a rhetorical question. The answer is the manufacturer will deny everything and make counter accusations, and hope they can fix the problem before anyone finds out.
I'm all for moving forward with technology, but sometimes it seems we're creating more dangerous problems with our solutions.
The most advanced crashing in history.
My printer at home does it every time it starts up.
Too bad the airplane doesn't.
I guess production delays are more expensive than debugging-by-crash. Sad.
Where are we going and why are we in a handbasket?
Don't feed the trolls.
"Reduced control of an airplane" is such an extreme euphemism for "Engines quit abruptly after take-off and kill most of the crew."
Rather than try to run in some fail-safe manner, the control system was DESIGNED to shut down the misconfigured engines. The article suggests that while this could have been reasonable for shutting down one engine, it's not to automatically shut down three of four engines. What's crazy is that the engines start and run at high speed for take-off, but then shut down at 400ft - that's not the usual behavior from a "safety system."
Manual shutdown of failing engines can lead to problems too, as in the Kegworth Air Disaster. http://en.wikipedia.org/wiki/K...
Separated by cause: Software bug vs Hardware bug.
Looking for people to chat about multicopters, coding, music. skype: gtsiros
Just my take as a software engineer and current DoD employee that works with C17...
There should have been some process on firing up the jet / avionics / computers that ran checks to see that even if software was not latest, was it CONSISTENT?
Big fail from the software engineering standpoint.
If you want news from today, you have to come back tomorrow.
This is a tragedy, but since we're on a tech site, lets talk tech.
Return values are handled oddly in pretty much every major language. Many API calls want to return something simple- int or bool- and if anything is more complex than that, generally require an actual data structure to be returned, often as a reference. This means that the "I didn't do this" action has a variety of ways to be be passed back- none of them even close to standard.
If something returns a distance, magnitude, or size, "0" normally means "Error, nothing happened" which is often the same as "Sure, I wrote 0 bytes. Really."
If something needs to distinguish between success ("I did the thing 0 times as requested" and failure "I couldn't do the thing because of an error condition"), then sometimes a -1 is returned, or an exception thrown, or something else.
In this plane, something was, at some point, responsible for getting data about the engines. Likely, this happened in layers, each one having access to the results of the lower pieces. One of those pieces had the task of parsing those files.
So EITHER someone (process, program, whatever) meant to say "This is a problem" and instead said "Here's some default data", OR someone ELSE in that chain of commands (process, program, whatever) has a default for a "This is a problem" result to use as a failsafe, and it was never tested or never communicated up.
We probably won't get the technical details that go from "files missing" to "engines don't work". Certainly, several level of software or hardware could allow for any number of workarounds in this case, and I'm sure they have a complex system and this was some eventuality that was hard to test for.
Still, interesting to think about the error return methodology, and how it's so different everywhere in CS.
The summary, as usual, is terrible. The missing files were calibration data for the engine controllers, not executables of any kind.
However, the article says some astonishingly stupid things, like: "'Nobody imagined a problem like this could happen to three engines,' a person familiar with the 12-year-old project said."
Well, duh.
Since the human imagination is known to be almost completely useless as a tool for understanding reality or predicting the future, this has to be the most obvious observation since the dawn of time.
Anything that can happen, will. Since we have finite resources, we have to guess what is most likely to happen. If we have data, we can run predictive models to inform our guesses. The one thing we know with near-certainty is that what we imagine might happen is completely irrelevant to what will actually happen.
The human imagination is no better at understanding or predicting today than it was when people were imagining bloodletting balanced the humours. It makes as much sense mentioning it in this context as saying, "Our astrologers and scriers never saw this coming!"
Blasphemy is a human right. Blasphemophobia kills.
The checks in place for aircraft hardware is extremely rigorous.
Yes, but how many of those regulations and checks trace back to accidents versus an engineer's foresight? I'd expect that most items in a pilot's pre-flight checklist do trace back to accidents. And it seems the computer's pre-flight checklist will too.
I once heard that the expression "Navy regulations are written in blood" was used to explain to new sailors why so many tasks are to be performed exactly the way the regs say and in no other manner. The phrase was then elaborated on explaining to the sailors that when things were done otherwise sailors sometimes died, for small things like failing to properly secure a hatch (door).
There was a warning. From the article:" the first warning pilots would receive of the engine data problem would be when the plane was 400 feet (120 meters) in the air". Probably a messages along the lines "you're f@#$-ed! have a nice day!"
+ http://en.wikipedia.org/wiki/T...
The first computer controlled X-ray machine.... which accidentally irradiated some people to death...
due to *gasp* software faults! (say it ain't so!)
I first heard about the Therac-25 during my "Ethics in Computer Science" class many years ago - it made an excellent case study... about problems just like this one.
Once the textbooks get updated, Therac-25 will be replaced with a case study about the a400m roll out. ^_^
EPROMS are no more immune to bad data than flash memory.
Besides, being well into the era of malware I'm surprised that files aren't delivered as a complete image. Complete with a manifest of files and version numbers and each file being digitally signed.
Or maybe some developer did have such a manifest, his/her code detected the error, reported the error, but the error/exception was handled in a way that didn't rise to the pilot's attention nor prevent engine startup.
FTFA: "...Without the vital data parameters, information from the engines is effectively meaningless to the computers controlling them. The automatic response is to hunker down and prevent what would usually be a single engine problem causing more damage. This is what the computers apparently did on the doomed flight, just as they were designed to do."
So, in other words, each engine did exactly what is was designed to do, which is to act independently and shut itself down. There's no executive override function that says "hmmm, maybe we shouldn't shut down 3 engines at the same time!" The crew had no chance against an obviously buggy software implementation. Pilots need more control to override complex software like this in emergencies.
Dilbert explains:
http://dilbert.com/strip/2000-...
Some drink at the fountain of knowledge. Others just gargle.
They were spinning too slowly? Isn't this why the pilot has a throttle? And if they are supposed to 'correct' and 'adjust' the input from the pilot, as one article explains, then how did it ever take off in the first place? Shouldn't there be a basic check like 'if altitude != 0 { allow_engine_off("NO!") } I'm sure there are all sorts of reasons why it's better this way, but it seems like when the plane is able to just ignore the pilot, then you are simply waiting for a catastrophe to occur.
I will shred my adversaries. Pull their eyes out just enough to turn them towards their mewing, mutilated faces. Illyria
" The more they overthink the plumbing, the easier it is to stop up the drain. " - Montgomery Scott, Star Trek III
If the plane had used apps, it would have systemD!
Get free satoshi (Bitcoin) and Dogecoins
The high tech word of aviation is at least 30 years old. There is a reason for that, it works and it rarely fails. All the fancy stuff is bolted on top of the bombproof legacy gear, which usually will keep working even after a complete loss of power.
This is a fishing trip to try to dump the blame on the manufacturer. Pilot error is in the high 90's when it comes to crashed planes. Props spinning to slowly? When all else fails look out the window and fly the goddamn plane!
For too long a wild West attitude has prevailed over software engineering.
Only strict government regulation can prevent programmers from cutting corners.
Programming should be like any other profession. No one should be allowed to practice programming unless they've been certified by the government as capable. They should also be bonded, so that any damage they do can be paid for.
There's the EPA. There's OSHA. There should be a regulator that can oversee all software engineering. Programmers should have to justify their code and prove its correctness before it's allowed out in the wild.
Of course some programmers will complain that such regulations will hurt the industry. What they really mean is that they can't exploit their customers.
Engineer 1: "Hey, I know, I'll build in a function that wipes the entire control system when it starts a firmware update so that no old software gets left behind after the update."
Engineer 2: "It'll save a ton of time on this firmware update if I leave out the engine control functions, since those aren't being updated. My bosses will love me!"
Nobodies Prefect
Tidbits for Techs Technology Blog
(icon of plane) + (icon of computer) = (icon of computer)
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
WTF? No automated system check to determine if all needed files are present before flying??!
My God can beat up your God. Just kidding...don't take offense. I know there's no God.
We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have.
Oh, they slowed down alright, but the attitude was not right.
this would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.
My God can beat up your God. Just kidding...don't take offense. I know there's no God.
WTF? No automated system check to determine if all needed files are present before flying??!
Sure there is.
We call it 'gravity'.
Don't starve the trolls.
More like (after 400 feet of ascent):
Device driver file power.drv missing for engine OS.
Device driver file control.drv missing for engine OS.
Device driver file fan_sensor.drv missing for engine OS.
Engine 1 will shut down.
Engine 2 will shut down.
You may want to concider a religion with afterlife: now
Those error messages are important, y'know.
But also important to keep a sense of humour in all the seriousness..
Boeing has an equally unimpressive architecture track record... http://www.cnn.com/2015/05/17/... So you shouldn't fly much...
So it would probably have worked, and not crash because someone was using tr(1) to parse some output in an overly complicated shell startup system...
>caused the affected propellers to spin too slowly
Propellers? On a jet?
Low credibility.
And what exactly make of plane would you fly on that is safer? Boeing has an equally bad track record. Sounds like you must walk everywhere
I like reminiscing about the rope-and-pulley days but i've been stranded with a broken clutch steel-rope cable, I've had another one snap on a bike, and points-and-condenser ignitions are inhumane and intolerant of lapses in maintenance. That peculiar smell that old cars and old planes had? incomplete combustion.
I like this computer-controlled world. Things work much better.
The rope-and-pulley analog here would be "Hey Bertie, did you put the cotter pin on that rod?" "Ya ya, sure sure!"
Meanwhile, as the plane reaches 400 ft:
*clink* "Hey.. what was that?" "Hey man the thrott*BLAM* (impact on ground)
The "Civilized World" jumped the shark ca. 1973.
The pilots should have reasoned: "Engines not responding to control. Since the engines are still at least giving us high power, we should climb to a height that gives us options, then try some things to fix the problem, or figure out how to cut the engines completely and glide in, having enough height to get the setup of the difficult approach just right."
Of course the maintenance program manager for the aircraft manufaturer should have reasoned: "All maintenance procedures should be performed by checking off, in an app, a detailed automated checklist of steps, such as restoring custom-data files. The maintenance software app should not permit maintenance to be signed off as complete until the automated checklist is all checkmarked. and it goes without saying that all such step-by-step procedures should be verified as complete and working before being included in allowed maintenance procedures of operational aircraft."
Where are we going and why are we in a handbasket?
It was simply a rush job, and extremely likely the managers were responsible. I am sure no developer worth her / his salt (avionics engineers generally dont have the temperament of 19year kids) would have liked to proceed in this manner that led to deletion of the said integral files.
So, one unsubstantiated claim versus hundreds of proven examples of intentional incompetence by Airbus? I'm flying Boeing.
The engines should have never have started without the missing files. Crazy crazy.
This is why Dr. McCoy didn't trust the transporter.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
In other news, Microsoft announced today the release of Microsoft Windows Blue. This is a version of Windows to run all kinds of aerial vehicles, ....
from the Airbus A400M to the smallest 4-rotor drone. Contrary to rumors, the "Blue" has to do with the color of the sky, not the color of the
computer screen when
Boeing doesn't have anywhere near the horrible record that Airbus does.
I'd bet a million dollars that this isn't the first crash caused by this, or something similar (damaged configuration settings file, mis-matched file versions, etc). Great- another thing to worry about when flying. :(
Just cruising through this digital world at 33 1/3 rpm...
You act like it is hard to have the ECU during bootup tell itself it's at 400ft and then see if everything is OK. This was half assed shit programming and there's really no excuse.
And how many episodes were we treated with transporter transposition accidents that the ol' Doc then had to deal with, hmmm?
having a computer between the pilot of any system and the mechanical components is just bending over and begging for it. Humans are mechanical. Engines are mechanical. Keep the fucking interfaces mechanical and the transport later the same way, the only thing that's coming of all this so-called automation and computer controlled engine management is butthurt and dead people. I come off as a bit of a luddite? Good. I'd sooner fucking walk anyway, the only thing I have to worry about is blisters. You go fly, the only thing you have to worry about is:
"NOT READY READING DRIVE A. (A)BORT, (R)ETRY, (F)AIL?>
six vertical miles away from your nearest Apple Genius.
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
well, good for you. The A400m isn't generally available for civilian use anyway (unless you have 200 million Dollars cash just lying around), it's a ramp heavylifter.
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
The standard way to return an error is to return a negative value, as zero can very often be a normal value but a negative value is more often than not never valid (distance, weight, volume, etc...). Unfortunately that means that the program can mistake a negative value for an incredibly non signed int value...
THIS IS BAD SOFTWARE DESIGN !!!
It is simply INCREDIBLY STUPID to create a "workflow" by which CATASTROPHIC SYSTEM FAILURE "happens" for a stupid reason like this .... as evidently there was no backup "safety" ....
this warrants the REMOVAL of the management structure of Airbus and the FULL STOP UNTIL A COMPLETE FIX AND REPURPOSE is done to further development of a plane which was originally designed to replace the C130 but MARKETED AS A STRATEGIC AIRLIFTER like the C17 ( BUT WHICH CANNOT EVEN CARRY A MID SIZE MILITARY ASSAUT TRACK VEHICLE , AND IMPOSSIBLE TO CARRY A MBT (MEDIUM BATTLE TANK like the C17...)
This is a COLOSSAL WASTE of public money and it should be stopped JUST AS the incredibly stupid F35 flying chicken....probably THE largest MILITARY PLANNING FAILURE of the century....
Crashing in [30] seconds. Press OK, to crash now.
A military project with multi-nation politics. Need I say more?
My cousing worked with airbus as an engineer, prepping the A380 for release, after the cableing debacle. No single responsible project lead with competence and a mandate, subcontractors 6 levels deep with the suits drawing out money at every level, nationalistic policing, etc.. A burocratic nightmare barely imaginable by the human mind.
Think Berlin Airport but with a bunch of EU nations thrown into the mix involving complex new machinery and avantgarde technology. Yeah, right.
We suffer more in our imagination than in reality. - Seneca
The new breed of developers do not care about boring details. If you doubt, a few more years and we will have a joker wanting to program avionics using javascript.
Religion: The greatest weapon of mass destruction of all time
2 if I remember correctly and 1 where it was used to fix the aging issue.
Yea, standard rule.
Maybe the plane was in test? So not production. So no need to follow that rule?
Well ok, a test plane is still running in production as far as avionics goes, even if it's still in testing!
And if you have 4 systems in your test environment, there's a reason. You *don't* update all of them, or 3/4 of them. But half!
And I guess, with 2 engines, the plane could have been saved.
I'm a sysadmin, and I blame the sysadmin as the cause of the catastrophe.
Atari rules... ermm... ruled.
Then the engine should either report a shitload of warnings to the cockpit or it should refuse to turn on.
This "software" thing has gone horribly horribly wrong.
The engine should refuse to start when some critical files are missing.
On the other hand, there are so many ways to mess up, that the quest for perfection is elusive. Especially when a team is inexperienced, politically appointed instead of by real skills or when the relationshipt between management and employees is bad. This often comes from underfunding ANY YOU BET this program is underfunded.
"Europe" is ruled by a bunch of socialist posers who are basically clueless about modern industry, modern technology and modern military. And even the captains of industry are mostly simpletons here. We have no such thing as Google, Intel, Huawei, Samsung here.
They have huge ambitions and want a super-duper aircraft, but then proceed to piss away the necessary money for people who never worked and will never work. The typical socialist disease.
Jäger 90 was the same thing. Lots of ambition, little funding. The amazing thing is it actually does the job somehow. 15 years later than planned, of course.
The boss of Airbus already said he will never again do an underfunded program like this. And I know why, because I know the Euro Collectivists. People who want somebody else to bring them the Paradise of Laziness and Irresponsibility. They are both dumb and malicious, otherwise they would do some real, hard work. Because THAT is what generates WEALTH. Printed paper DOES NOT bring wealth is everybody is playing lazyman.
Me ? A swabian.
We'll round up a few Jews, make them confess and then throw a big show of executing them and their families, as always. It's not possible a plane made by European Aryan ubermensch could fail unless sabotaged by Jews.
Just google for "reset button Tiltrotor".
Aviation software MUST WORK. When you are in the air and errors pop up it is very often TOO LATE.
And of course there are some well-funded, well-executed projects like the A320 which absolutely depend on correct software. They had a major problem which was actually solved by "overriding", but that approach is not a panacea, rather a last-ditch measure.
You apparently know nothing about Spain. General Franco was a highly capable and intelligent guy. He steered his country around both Materialism (a London Bankster invention) and Fascism. After the war he was best friend with NATO. That didnt stop liars like you to call him a "fascist", I know that.
https://en.wikipedia.org/wiki/Spain_and_World_War_II#Jews_and_other_refugees
So he GAVE JEWS ASYLUM. But yeah, the Cultural Marxists label him a "fascist" until today. What a load of wicked, devil-worshipping scumbags. I bet they have an icon of Feliks Dzhershinsky which they pray to every night.
Also, how much does Boeing pay you for this shite ?
And, why dont you drop yourself off a bridge. It's all easier then.
I don't believe it!
Did anyone ever hear of distributed systems? One simple computer to run evey critical system all networked together. Each and everyone can keep it's device functioning with manual command if everything else fails. The only way to upgrade the system is to physically remove it an plug in a new one.
It's embedded programming 101!!
WTF? No automated system check to determine if all needed files are present before flying??!
Ironically, I would call this "preflight checks"...
What nonsense!