Slashdot Mirror
ESA: European Mars Lander Crash Caused By 1-Second Glitch (space.com)
An anonymous reader quotes a report from Space.com: The European Space Agency (ESA) on Nov. 23 said its Schiaparelli lander's crash landing on Mars on Oct. 19 followed an unexplained saturation of its inertial measurement unit (IMU), which delivered bad data to the lander's computer and forced a premature release of its parachute. Polluted by the IMU data, the lander's computer apparently thought it had either already landed or was just about to land. The parachute system was released, the braking thrusters were fired only briefly and the on-ground systems were activated. Instead of being on the ground, Schiaparelli was still 2.3 miles (3.7 kilometers) above the Mars surface. It crashed, but not before delivering what ESA officials say is a wealth of data on entry into the Mars atmosphere, the functioning and release of the heat shield and the deployment of the parachute -- all of which went according to plan. In its Nov. 23 statement, ESA said the saturation reading from Schiaparelli's inertial measurement unit lasted only a second but was enough to play havoc with the navigation system. ESA said the sequence of events "has been clearly reproduced in computer simulations of the control system's response to the erroneous information." ESA's director of human spaceflight and robotic exploration, David Parker, said in a statement that ExoMars teams are still sifting through the voluminous data harvest from the Schiaparelli mission, and that an external, independent board of inquiry, now being created, would release a final report in early 2017.
Man, if I had a nickel for every time some kind of sensory saturation forced a premature release...
I've calculated my velocity with such exquisite precision that I have no idea where I am.
They're blaiming lag?
https://en.wikipedia.org/wiki/...
How in hell did they test their Kalman filter to allow such bad data to reach the decision logic? (I assume they used one.)
Government cannot make man richer, but it can make him poorer. - Ludwig von Mises
Overflows and bad data problems happened to ESA before.
"Obligatory" Dark Star reference.
Brings to mind the failure of the first Arianne 5 launcher because control software spat an Ada stack trace over a line which was supposed to only contain kinematic data.
http://michaelsmith.id.au
When the altitude stops changing for a whole second the filter is going to have to be a long one! And that ain't desirable for responsive control.
The real question is how could the sensory processor have overloaded in the first place? My money is on simple [b]code bloat[/b]. Ie: They used a bunch of generic libraries that use further libraries that use further libraries that use further libraries that use further libraries that use further libraries ...
So they didn't correlate the IMU data with ranging radar or even barometric altitude information so as to avoid this?
I know weight and volume are at a premium on such craft but a barometric sensor (even one capable of operating in Mars's rarefied atmosphere, is the size of a thumbnail and weighs just a fraction of a gram.
Sigh!
Should've used metric seconds.
systemd is Roko's Basilisk.
What kind of IMUs are normally used in these craft? The same kind used in aircraft and weapons?
... $1000 quadcopters back here on Earth ship with multiple IMUs for redundancy, since the bloody things are about as trustworthy as your average politician.
Having made that glib remark, I'm sure it either did have redundancy, or if it didn't that was for a good reason (e.g. risk of failure deemed too low to warrant the weight penalty in adding redundancy). I would also like to think that they're using somewhat more reliable IMUs than those found in quads.
Yes, you have to wonder why on a mission of this expense and complexity the height about the ground is essentially done by mathematical dead reckoning. Would adding a ranging radar really have added so much to the weight and/or required package size that it was infeasible to include it? Obviously they must have considered it and I'd be interested to know why in the end it was not seen as a viable part of the solution.
Hidden malloc()'s is a good example of the bloat problem I'm referring to.
Why wasn't the IMU sensor doubled by other ways of detection? There was no fallback in case it malfunctioned.
No basic sanity checks? As in "This phase must last at least X seconds", or "No switching to landing behavior if altitude measurement from 1 second ago still said '2 miles above surface'"?
If the landing struts are subject to a compressive force, you've probably landed. If not, you haven't. Why wouldn't the computer make use of this?
Am I missing something, or is this a stupid design?
"[T]he erroneous information generated an estimated altitude that was negative," ESA said.
Which resulted in an actual altitude that was negative.
A brief burst was enough.
I wonder whether making the source code of these probes available to the public, for vetting would help spot bugs like these? I am also curious whether releasing the code would be problematic for any reason?
Jumpstart the tartan drive.
Lots of just plain old ignorant comments here. I say this in a nonperjorative sense - if you've not worked on flight software, there's no way you could know.
1) Space is unforgiving, hardware designs change very, very slowly. Project schedules move fast and have limited budgets. Just because you can buy a MEMS based IMU for your quadcopter does not mean that you can get one for a spacecraft that will work reliably from -40 to +80C, withstand the vibe tests, the pyroshock, etc. Oh, yeah, and it (and the surrounding electronics) has to not suffer any ill effects from a stray high energy particle: Single Event Effects, generally, but Latch-up, gate rupture, etc. are the issues one would worry about - bit flips and SEFI are something software can potentially handle, but destructive latchup is, well, destructive. Parts made for consumer or automotive high-rel applications just don't worry about this kind of thing. Sure, you could test your brand new whiz-bang IMU in an accelerator, but that costs money and schedule.
2) Yes, not range checking or reasonableness checking the data from the IMU was a definite design problem. Or, maybe, the entry descent and landing algorithm was such that if you DID get a hiccup and detect it, the missing data means all is lost anyway, so why bother. There's a basic principle in spacecraft design that you don't add hardware or software (which might fail, and costs time, money, and maybe mass or power) to give you information which you cannot use.
3) With respect to the landing radar - it's very possible that it has poor accuracy up high and is only used in the final descent stages.
4) Fault handling is tricky - you can easily go down a rat maze of low probability events generating code (and hardware) to handle obscure corner cases, thereby increasing your test costs and time, and potentially introducing other faults. For a lot of plausible error scenarios, it's likely you're going to fail for other reasons, so there's no point in trying to do things like estimate state from other sensors.
Summary - Entry, Descent, and Landing on another planet is really, really hard. It's even harder when you have time and budget constraints.
The Schiaparelli EDM lander is an example of the typical one-off missions that humanity does these days. It's worth noting that they could have had built and launched two or more of these vehicles for much less than the first and already be correcting the erroneous code on a second spacecraft. Then they wouldn't have to wait years for a replacement mission and have a much better chance of mission success.
Heh. When I heard about the crash landing I literally said to a friend of mine, "I bet the subroutine that cuts the parachute loose so it doesn't land on top of the payload detected the thump of the parachute strings going-taught, determined that meant it was on the ground, and cut the parachute." Didn't something like this happen to an American probe sent to Mars around twenty years ago?
Do not look into laser with remaining eye.
Software to land a probe on mars is quite similar, if not identical, to software to put a (nuclear) warhead on a target. That's an important strategic capability for "first world" nations - otherwise you're in the category of Saddam firing Scuds, which are basically V2s with newer parts, and quite literally cannot hit the broad side of a barn (albeit from 100 km away).
So, the hard parts of solving the problem (after you've done the basic college physics part) are likely to not be open source. Things like handling the rapidly changing aerodynamic effects at hypersonic speeds are a long way from "considering air as an incompressible fluid": Your state estimator has inputs with wildly varying inputs (dynamic range is huge compared to, say, a quadcopter), with equally wildly varying uncertainties. The transitions as you go through various deployments are also something that does not have a lot of commonality with other applications: airplanes do have deployments (landing gear, flaps, stores), quads and hexes do not; and the deployment of flaps on a plane occurs in a fairly narrow, and well understood, set of dynamic conditions.
And then, what? Should it take an "instantenous" decision about what to do next?
This is not a bugginess issue.
Point is the layers create bloat. Any hidden dynamic memory allocations that occur, by whatever system call, is just one more part of the bloat.
...detected the thump of the parachute strings going-taught, determined that meant it was on the ground, and cut the parachute...
Just think about that again for a moment.
It's "taut" by the way, but apart from that I see a bright new career in space exploration in your future.
If God forks the Universe every time you roll a die, he'd better have a damned good memory.
So say they're doing some kind of weighted average of an altitude computation from the inertial navigation unit and an altitude computation from the doppler radar altimeter.
They should have some code in there saying: If these two values that we're averaging are wildly off from each other, let's not take the average. Instead, let's go into some exception handling code which uses some kind of heuristic (and a little time perhaps) to determine which of the two instruments should become the solely trusted source of the altitude value.
Sounds like a lack of hazard analysis / fault tree analysis and or fault-tolerant design in the design process.
Where are we going and why are we in a handbasket?