Slashdot Mirror
Missing Files Blamed For Deadly A400M Crash
An anonymous reader writes: Think you had a bad day when your software drivers go missing? Rejoice, you get to live! A fatal A400M crash was linked to data-wipe mistake during an engine software update. A military plane crash in Spain was probably caused by computer files being accidentally wiped from three of its engines, according to investigators. Plane-maker Airbus discovered anomalies in the A400M's data logs after the crash, suggesting a software fault. And it has now emerged that Spanish investigators suspect files needed to interpret its engine readings had been deleted by mistake.This would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.
Is it so hard to have a integrity check and diagnostic set run as part of the preflight checks? If you can place hundreds of miles of wire and know what's what, surely they have computer engineers competent enough to make something like this to catch such glaring errors.
Tuesday is crash-day, oops I meant patch-day.
The engineers probably did one but obviously it wasn't good enough. Software fails, but you can design it to fail in safe ways.
Come on, folks. Turn the power on to the engine controllers at the flight line and the status display should have been flashing warnings. Nobody should have even started this thing.
Have gnu, will travel.
Remember when you could start your car with the turn of a key and not have to worry about a software update refuckulating your engine timing system? Adding computers to EVERYTHING is a really bad idea, especially because PEOPLE are required to write the code that runs the software that controls X Y and Z systems on the engine, chassis, and security systems.
Depressingly, that might actually be true.
Not because of 'apps' of course; but because no self-respecting consumer OS would fail to cryptographically verify the execution environment(lest some precious 'premium content' be absconded with by pirates) and an entire missing file probably would have caused the aircraft to refuse to move until taken back to Airbus HQ for re-blessing by the vender.
They don't succeed against motivated pirates, of course; but this is one area where consumer software vendors do actually give a fuck. If people believed that a sabotaged voting machine or a defective ECU could pirate Blu-rays, we'd live in a safer world.
You'd think there would be some kind of checks in place that wouldn't allow the plane to operate when critical files are missing. Or that the files couldn't be deleted.
Stories like these are the reason I can't believe auto manufacturers are even considering being able to push updates to cars. The checks in place for aircraft hardware is extremely rigorous. Pretty much every nut and bolt has a complete history log. If this kind of thing can happen on an aircraft, what happens when some weird conditions occur that cause brakes to fail in an automotive update? That's a a rhetorical question. The answer is the manufacturer will deny everything and make counter accusations, and hope they can fix the problem before anyone finds out.
I'm all for moving forward with technology, but sometimes it seems we're creating more dangerous problems with our solutions.
I hate squirrels!
The most advanced crashing in history.
My printer at home does it every time it starts up.
Too bad the airplane doesn't.
I guess production delays are more expensive than debugging-by-crash. Sad.
Where are we going and why are we in a handbasket?
Separated by cause: Software bug vs Hardware bug.
Looking for people to chat about multicopters, coding, music. skype: gtsiros
Just my take as a software engineer and current DoD employee that works with C17...
There should have been some process on firing up the jet / avionics / computers that ran checks to see that even if software was not latest, was it CONSISTENT?
Big fail from the software engineering standpoint.
If you want news from today, you have to come back tomorrow.
This is a tragedy, but since we're on a tech site, lets talk tech.
Return values are handled oddly in pretty much every major language. Many API calls want to return something simple- int or bool- and if anything is more complex than that, generally require an actual data structure to be returned, often as a reference. This means that the "I didn't do this" action has a variety of ways to be be passed back- none of them even close to standard.
If something returns a distance, magnitude, or size, "0" normally means "Error, nothing happened" which is often the same as "Sure, I wrote 0 bytes. Really."
If something needs to distinguish between success ("I did the thing 0 times as requested" and failure "I couldn't do the thing because of an error condition"), then sometimes a -1 is returned, or an exception thrown, or something else.
In this plane, something was, at some point, responsible for getting data about the engines. Likely, this happened in layers, each one having access to the results of the lower pieces. One of those pieces had the task of parsing those files.
So EITHER someone (process, program, whatever) meant to say "This is a problem" and instead said "Here's some default data", OR someone ELSE in that chain of commands (process, program, whatever) has a default for a "This is a problem" result to use as a failsafe, and it was never tested or never communicated up.
We probably won't get the technical details that go from "files missing" to "engines don't work". Certainly, several level of software or hardware could allow for any number of workarounds in this case, and I'm sure they have a complex system and this was some eventuality that was hard to test for.
Still, interesting to think about the error return methodology, and how it's so different everywhere in CS.
The summary, as usual, is terrible. The missing files were calibration data for the engine controllers, not executables of any kind.
However, the article says some astonishingly stupid things, like: "'Nobody imagined a problem like this could happen to three engines,' a person familiar with the 12-year-old project said."
Well, duh.
Since the human imagination is known to be almost completely useless as a tool for understanding reality or predicting the future, this has to be the most obvious observation since the dawn of time.
Anything that can happen, will. Since we have finite resources, we have to guess what is most likely to happen. If we have data, we can run predictive models to inform our guesses. The one thing we know with near-certainty is that what we imagine might happen is completely irrelevant to what will actually happen.
The human imagination is no better at understanding or predicting today than it was when people were imagining bloodletting balanced the humours. It makes as much sense mentioning it in this context as saying, "Our astrologers and scriers never saw this coming!"
Blasphemy is a human right. Blasphemophobia kills.
The checks in place for aircraft hardware is extremely rigorous.
Yes, but how many of those regulations and checks trace back to accidents versus an engineer's foresight? I'd expect that most items in a pilot's pre-flight checklist do trace back to accidents. And it seems the computer's pre-flight checklist will too.
I once heard that the expression "Navy regulations are written in blood" was used to explain to new sailors why so many tasks are to be performed exactly the way the regs say and in no other manner. The phrase was then elaborated on explaining to the sailors that when things were done otherwise sailors sometimes died, for small things like failing to properly secure a hatch (door).
+ http://en.wikipedia.org/wiki/T...
The first computer controlled X-ray machine.... which accidentally irradiated some people to death...
due to *gasp* software faults! (say it ain't so!)
I first heard about the Therac-25 during my "Ethics in Computer Science" class many years ago - it made an excellent case study... about problems just like this one.
Once the textbooks get updated, Therac-25 will be replaced with a case study about the a400m roll out. ^_^
EPROMS are no more immune to bad data than flash memory.
Besides, being well into the era of malware I'm surprised that files aren't delivered as a complete image. Complete with a manifest of files and version numbers and each file being digitally signed.
Or maybe some developer did have such a manifest, his/her code detected the error, reported the error, but the error/exception was handled in a way that didn't rise to the pilot's attention nor prevent engine startup.
FTFA: "...Without the vital data parameters, information from the engines is effectively meaningless to the computers controlling them. The automatic response is to hunker down and prevent what would usually be a single engine problem causing more damage. This is what the computers apparently did on the doomed flight, just as they were designed to do."
So, in other words, each engine did exactly what is was designed to do, which is to act independently and shut itself down. There's no executive override function that says "hmmm, maybe we shouldn't shut down 3 engines at the same time!" The crew had no chance against an obviously buggy software implementation. Pilots need more control to override complex software like this in emergencies.
Dilbert explains:
http://dilbert.com/strip/2000-...
Some drink at the fountain of knowledge. Others just gargle.
They were spinning too slowly? Isn't this why the pilot has a throttle? And if they are supposed to 'correct' and 'adjust' the input from the pilot, as one article explains, then how did it ever take off in the first place? Shouldn't there be a basic check like 'if altitude != 0 { allow_engine_off("NO!") } I'm sure there are all sorts of reasons why it's better this way, but it seems like when the plane is able to just ignore the pilot, then you are simply waiting for a catastrophe to occur.
I will shred my adversaries. Pull their eyes out just enough to turn them towards their mewing, mutilated faces. Illyria
" The more they overthink the plumbing, the easier it is to stop up the drain. " - Montgomery Scott, Star Trek III
If the plane had used apps, it would have systemD!
Get free satoshi (Bitcoin) and Dogecoins
The high tech word of aviation is at least 30 years old. There is a reason for that, it works and it rarely fails. All the fancy stuff is bolted on top of the bombproof legacy gear, which usually will keep working even after a complete loss of power.
This is a fishing trip to try to dump the blame on the manufacturer. Pilot error is in the high 90's when it comes to crashed planes. Props spinning to slowly? When all else fails look out the window and fly the goddamn plane!
For too long a wild West attitude has prevailed over software engineering.
Only strict government regulation can prevent programmers from cutting corners.
Programming should be like any other profession. No one should be allowed to practice programming unless they've been certified by the government as capable. They should also be bonded, so that any damage they do can be paid for.
There's the EPA. There's OSHA. There should be a regulator that can oversee all software engineering. Programmers should have to justify their code and prove its correctness before it's allowed out in the wild.
Of course some programmers will complain that such regulations will hurt the industry. What they really mean is that they can't exploit their customers.
Engineer 1: "Hey, I know, I'll build in a function that wipes the entire control system when it starts a firmware update so that no old software gets left behind after the update."
Engineer 2: "It'll save a ton of time on this firmware update if I leave out the engine control functions, since those aren't being updated. My bosses will love me!"
Nobodies Prefect
Tidbits for Techs Technology Blog
(icon of plane) + (icon of computer) = (icon of computer)
(-1: Post disagrees with my already-settled worldview) is not a valid mod option.
WTF? No automated system check to determine if all needed files are present before flying??!
My God can beat up your God. Just kidding...don't take offense. I know there's no God.
Reminds me of the term that airlines' use when they collect the insurance on a crashed plane - "Involuntary Conversion of an Asset"
We've lost that kind of 'slow down and make sure it's right' attitude that engineers really need to have.
Oh, they slowed down alright, but the attitude was not right.
this would have caused the affected propellers to spin too slowly causing loss of power and eventually, a crash.
My God can beat up your God. Just kidding...don't take offense. I know there's no God.
WTF? No automated system check to determine if all needed files are present before flying??!
Sure there is.
We call it 'gravity'.
Don't starve the trolls.
Engines did not quit abruptly or kill any crew.
3 out of 4 engines stopped responding to changes in power level and continued to run as they were.
The pilot noticed this, talked to air traffic control and requested to make an emergency landing.
They plan was turned around and headed back to the airport.
The plane crashed into a pylon while attempting the emergency landing.
Boeing has an equally unimpressive architecture track record... http://www.cnn.com/2015/05/17/... So you shouldn't fly much...
So it would probably have worked, and not crash because someone was using tr(1) to parse some output in an overly complicated shell startup system...
I like reminiscing about the rope-and-pulley days but i've been stranded with a broken clutch steel-rope cable, I've had another one snap on a bike, and points-and-condenser ignitions are inhumane and intolerant of lapses in maintenance. That peculiar smell that old cars and old planes had? incomplete combustion.
I like this computer-controlled world. Things work much better.
The rope-and-pulley analog here would be "Hey Bertie, did you put the cotter pin on that rod?" "Ya ya, sure sure!"
Meanwhile, as the plane reaches 400 ft:
*clink* "Hey.. what was that?" "Hey man the thrott*BLAM* (impact on ground)
The "Civilized World" jumped the shark ca. 1973.
The pilots should have reasoned: "Engines not responding to control. Since the engines are still at least giving us high power, we should climb to a height that gives us options, then try some things to fix the problem, or figure out how to cut the engines completely and glide in, having enough height to get the setup of the difficult approach just right."
Of course the maintenance program manager for the aircraft manufaturer should have reasoned: "All maintenance procedures should be performed by checking off, in an app, a detailed automated checklist of steps, such as restoring custom-data files. The maintenance software app should not permit maintenance to be signed off as complete until the automated checklist is all checkmarked. and it goes without saying that all such step-by-step procedures should be verified as complete and working before being included in allowed maintenance procedures of operational aircraft."
Where are we going and why are we in a handbasket?
This is why Dr. McCoy didn't trust the transporter.
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
In other news, Microsoft announced today the release of Microsoft Windows Blue. This is a version of Windows to run all kinds of aerial vehicles, ....
from the Airbus A400M to the smallest 4-rotor drone. Contrary to rumors, the "Blue" has to do with the color of the sky, not the color of the
computer screen when
It's not a jet moron.
I'd bet a million dollars that this isn't the first crash caused by this, or something similar (damaged configuration settings file, mis-matched file versions, etc). Great- another thing to worry about when flying. :(
Just cruising through this digital world at 33 1/3 rpm...
having a computer between the pilot of any system and the mechanical components is just bending over and begging for it. Humans are mechanical. Engines are mechanical. Keep the fucking interfaces mechanical and the transport later the same way, the only thing that's coming of all this so-called automation and computer controlled engine management is butthurt and dead people. I come off as a bit of a luddite? Good. I'd sooner fucking walk anyway, the only thing I have to worry about is blisters. You go fly, the only thing you have to worry about is:
"NOT READY READING DRIVE A. (A)BORT, (R)ETRY, (F)AIL?>
six vertical miles away from your nearest Apple Genius.
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
it's a quad turboprop, fool.
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
well, good for you. The A400m isn't generally available for civilian use anyway (unless you have 200 million Dollars cash just lying around), it's a ramp heavylifter.
Political debates have me rolling my eyes so much I think I got optical whiplash. I should sue. - Foamy The Squirrel
The A400M is a turboprop.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
THIS IS BAD SOFTWARE DESIGN !!!
It is simply INCREDIBLY STUPID to create a "workflow" by which CATASTROPHIC SYSTEM FAILURE "happens" for a stupid reason like this .... as evidently there was no backup "safety" ....
this warrants the REMOVAL of the management structure of Airbus and the FULL STOP UNTIL A COMPLETE FIX AND REPURPOSE is done to further development of a plane which was originally designed to replace the C130 but MARKETED AS A STRATEGIC AIRLIFTER like the C17 ( BUT WHICH CANNOT EVEN CARRY A MID SIZE MILITARY ASSAUT TRACK VEHICLE , AND IMPOSSIBLE TO CARRY A MBT (MEDIUM BATTLE TANK like the C17...)
This is a COLOSSAL WASTE of public money and it should be stopped JUST AS the incredibly stupid F35 flying chicken....probably THE largest MILITARY PLANNING FAILURE of the century....
Crashing in [30] seconds. Press OK, to crash now.
A military project with multi-nation politics. Need I say more?
My cousing worked with airbus as an engineer, prepping the A380 for release, after the cableing debacle. No single responsible project lead with competence and a mandate, subcontractors 6 levels deep with the suits drawing out money at every level, nationalistic policing, etc.. A burocratic nightmare barely imaginable by the human mind.
Think Berlin Airport but with a bunch of EU nations thrown into the mix involving complex new machinery and avantgarde technology. Yeah, right.
We suffer more in our imagination than in reality. - Seneca
The new breed of developers do not care about boring details. If you doubt, a few more years and we will have a joker wanting to program avionics using javascript.
Religion: The greatest weapon of mass destruction of all time
Yea, standard rule.
Maybe the plane was in test? So not production. So no need to follow that rule?
Well ok, a test plane is still running in production as far as avionics goes, even if it's still in testing!
And if you have 4 systems in your test environment, there's a reason. You *don't* update all of them, or 3/4 of them. But half!
And I guess, with 2 engines, the plane could have been saved.
I'm a sysadmin, and I blame the sysadmin as the cause of the catastrophe.
Atari rules... ermm... ruled.
I think the term is "hull loss" by insurance companies
Just for fun ships are have been termed "bottoms" before and after they sink.
https://en.wikipedia.org/wiki/...
I don't believe it!
Did anyone ever hear of distributed systems? One simple computer to run evey critical system all networked together. Each and everyone can keep it's device functioning with manual command if everything else fails. The only way to upgrade the system is to physically remove it an plug in a new one.
It's embedded programming 101!!
WTF? No automated system check to determine if all needed files are present before flying??!
Ironically, I would call this "preflight checks"...