Slashdot Mirror


Bad Code May Have Crashed Schiaparelli Mars Lander (nature.com)

cadogan west writes "In the accordance with the longstanding tradition of bad software wrecking space probes (See Mariner 1), it appears a coding bug crashed the ESA's latest attempt to land on Mars." Nature reports: Thrusters, designed to decelerate the craft for 30 seconds until it was metres off the ground, engaged for only around 3 seconds before they were commanded to switch off, because the lander's computer thought it was on the ground. The lander even switched on its suite of instruments, ready to record Mars's weather and electrical field, although they did not collect data...

The most likely culprit is a flaw in the craft's software or a problem in merging the data coming from different sensors, which may have led the craft to believe it was lower in altitude than it really was, says Andrea Accomazzo, ESA's head of solar and planetary missions. Accomazzo says that this is a hunch; he is reluctant to diagnose the fault before a full post-mortem has been carried out... But software glitches should be easier to fix than a fundamental problem with the landing hardware, which ESA scientists say seems to have passed its test with flying colours.

92 of 163 comments (clear)

  1. Mark my words by phantomfive · · Score: 5, Funny

    This wouldn't have happened if they'd used imperial not metric!
    New age hippie liberal airheads. If it's not a hogshead, it's not fresh!

    --
    "First they came for the slanderers and i said nothing."
    1. Re:Mark my words by plover · · Score: 1

      This wouldn't have happened if they'd used imperial not metric!
      New age hippie liberal airheads.

      Naahh, just Europeans.

      --
      John
    2. Re:Mark my words by Waffle+Iron · · Score: 1

      I think this problem is a manifestation of a scientific units monoculture.

      We need diverse units to keep our calculations robust.

    3. Re:Mark my words by Man+On+Pink+Corner · · Score: 2

      Not naming it after a crater might've helped, too.

    4. Re:Mark my words by michelcolman · · Score: 4, Funny

      Now whenever an ESA scientist wants to talk about the Schiaparelli crater, you can ask "which one?" to shut him up.

    5. Re:Mark my words by wiretrip · · Score: 1

      I think you'll find it's *metres* - there's the problem...:-)

    6. Re:Mark my words by RockDoctor · · Score: 1
      Cruel! Good, but cruel.

      I take it that you'll be holding yourself down and beating yourself to death, to save us the effort?

      --
      Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
  2. Martians by meglon · · Score: 4, Funny

    They're still unwilling to concede that their defenses against the Martian's OBDS (Orbital Bombardment Defense System) is inadequate.

    --
    Fascism: An authoritarian and nationalistic right-wing system of government and social organization. See also: NAZI's
    1. Re:Martians by phantomfive · · Score: 2
      The most illustrious Council of the Elders met beneath the purple sky. Fields of loyal adepts filled the gathering grounds, as many loyal civilians waited on the perimeter, pushing ever forward to hear to words of the great one speaking, to even catch a glimpse of one of his most reputable gelsacs. As K'breel, speaker for the Council stood to speak, a hush fell over the crowd, and all stood in rapt attention, speaking thusly:

      Behold how the weaklings have fallen! Our priests and soldiers have toiled many days to finish our planetary defenses, and now they are operational! Our prayers during the last eclipse were especially effective.

      A junior reporter who asked a question about 'metric' was hastily removed from the scene.

      --
      "First they came for the slanderers and i said nothing."
    2. Re:Martians by Black+Parrot · · Score: 1

      They panicked when they heard we were sending a "probe" to learn more about them.

      --
      Sheesh, evil *and* a jerk. -- Jade
  3. There is no bad code. by ZecretZquirrel · · Score: 2

    Only bad testing.

    1. Re:There is no bad code. by hcs_$reboot · · Score: 1

      Testing on another planet is not that easy, though.

      --
      Slashdot, fix the reply notifications... You won't get away with it...
    2. Re:There is no bad code. by Anonymous Coward · · Score: 2, Interesting

      Working in a company that makes automotive electronics, I can say that any problem without an obvious hardware assembly cause becomes defined as a software problem.

      Faulty sensor causing false readings that cause the software to detect that the craft is on the ground? That's the software's fault for not detecting that the sensor was faulty and using magic as a backup method to get the right result.

    3. Re:There is no bad code. by m00sh · · Score: 3, Insightful

      Testing on another planet is not that easy, though.

      Yes, test it all in production.

      Since testing is sooooooooo hard.

      Landing is the most complicated part and Beagle and others have failed exactly here. There should be x100 or even more code for unit and integration testing than the actual code itself for the landing code. And, those tests should run through every permutation possible of every possible failure point or bad sensor readings.

      There is no way it thinks it has landed with that many sensor inputs. It is simply code that is not put through a good enough testing system.

    4. Re:There is no bad code. by Anonymous Coward · · Score: 2, Interesting

      If the people writing the simulation are too close to the people writing the control software, I can see this happening.

      When I worked on this stuff (and I did, including a Mars probe) we had three independent teams on different sides of the building, each with their own set of requirements, design, code and tests. Not only that, but the development environments and languages were different to avoid common mode bugs. Fun times; I have no idea how things are done today.

    5. Re:There is no bad code. by the_Bionic_lemming · · Score: 1

      Yeah, and this occurs on earth too.

      "No, sorry, you have to replace your 1200 dollar catalytic converter"

      "Can you just replace the 02 sensors since they are a problem on my jeep as evidenced by online posts???" //invalidates money spent to fix the problem by adding the false error the o2 sensors made

      "Yeah sure - 200 bucks in labor for two sixty dollar sensors. Jerk"

      "ok, thanks. (passes emissions test) "

      --
      _ _ _ Go for the eyes Boo! GO FOR THE EYES!
    6. Re:There is no bad code. by michelcolman · · Score: 1

      Well, it's not like this is the first time a system thought it was near the ground while in fact it wasn't. Turkish Airlines flight 1951 just to name one.

    7. Re:There is no bad code. by cellocgw · · Score: 1

      As someone else who works designing and building automotive radar systems, I take umbrage at your defensiveness. For one thing, car parts have to be dirt cheap to be competitive. parts for a Mars Lander don't. We don't know whether there were redundant altimeters (but normally there's three of everything ), and we don't even know that it was in fact a faulty altitude signal, regardless of root cause, that led to the crash.

      --
      https://app.box.com/WitthoftResume Code: https://github.com/cellocgw
    8. Re:There is no bad code. by Fragnet · · Score: 1

      It's actually worse than that. If there's a problem with the hardware, i.e. it's known to be failing to do what it's supposed to, it's the software people who're tasked with making a "workaround", i.e. frigging their own code to correct the error rather than the (often more expensive) hardware mod. I've got so many software projects behind me with hacks for hardware bugs you wouldn't believe. "isn't that what software is for?" is the inevitable bollocks you get from hardware engineers when confronted with the problem.

    9. Re:There is no bad code. by Gr8Apes · · Score: 1

      Seems a lot of app code gets tested on another planet, because it certainly doesn't work on this one.

      --
      The cesspool just got a check and balance.
    10. Re:There is no bad code. by cwsumner · · Score: 1

      Um... there is bad code AND bad testing! 8-P

      All code should be considered bad, until it passes testing. All testing should be considered bad, until it finds some bugs. Wash and repeat... ;-)

      Of course, testing costs money and it's better to buy the boss a new big desk... right??

    11. Re: There is no bad code. by cwsumner · · Score: 1

      Googled DATDP and couldn't find anything. What does it stand for?

      He must work for a eupopean space agency, they don't define their acronyms. Or maybe it was nasa...

    12. Re:There is no bad code. by cwsumner · · Score: 2

      It's actually worse than that. If there's a problem with the hardware, i.e. it's known to be failing to do what it's supposed to, it's the software people who're tasked with making a "workaround", i.e. frigging their own code to correct the error rather than the (often more expensive) hardware mod. I've got so many software projects behind me with hacks for hardware bugs you wouldn't believe. "isn't that what software is for?" is the inevitable bollocks you get from hardware engineers when confronted with the problem.

      Detecting when the hardware fails -is- part of the software's job. Industrial software does this routinely. Why can't aerospace software?

      But launching with known-bad hardware is criminal... 8-P

    13. Re:There is no bad code. by adhdengineer · · Score: 1

      space is one of those areas where the "we need it ready by X!" actually applies tho. they usually have a very tight launch window and missing it means delays measured in years.
      It's not like the normal software release scheduling governed by trade shows, salesmen selling shit that doesnt exist yet or the normal "we have to ship so we can book revenue for this quater!".
      From all i've heard tho in this case it does seem to be "blame the software until we find the root cause", which i think any software dev is familiar with.

  4. Re:easier to fix? by chuckugly · · Score: 1

    Except hardware requires actual manufacturing and all that goes along with THAT.

  5. And suddenly by Ol+Olsoc · · Score: 1
    The ridicule 'Murrica took for it's lander crashing goes silent.

    A code problem eh? Shit happens, and my condolences - it can happen to any of us.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    1. Re: And suddenly by Ol+Olsoc · · Score: 1

      Funny though only US has success on the red planet. This is not only Europe's second attempt, but the same bug. In 1999 the computer shut off its thrusters too early too and it crashed it's probe 100 feet in the air.

      I forgot the name of the probe for that one.

      I'm not certain. The Beagle in 2004 didn't deploy correctly, but otherwise I'm not certain. The Phobos-Grunt didn't make it out of earth gravity in 2011. It is so darn hard to land on Mars. The martian atmosphere is thick enough to make for a lot of heat, but not thick enough to slow you down anywhere near a normal earth type landing. And of course, the distances and reulting delays in communications time were making for around 14 minutes radio time, and 7 minutes of time between atmospheric contact and landing on the surface. So at the time NASA gets word that the lander has contacted the surface, its on the ground. 7 minutes of terror they call it.

      Here's a good video, if a little dramatic, of some of the hassles. http://www.space.com/16265-7-m...

      Anyhow, its an expensive lesson, but eventually they'll get something good on the ground.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  6. Considering the decline in code quality... by gTsiros · · Score: 2

    ...in recent years, it wouldn't surprise me one bit

    --
    Looking for people to chat about multicopters, coding, music. skype: gtsiros
    1. Re:Considering the decline in code quality... by Drethon · · Score: 2

      ...in recent years, it wouldn't surprise me one bit

      Yeah, ever since they stopped using goto and started with these class things, everything is going to hell.

    2. Re:Considering the decline in code quality... by hcs_$reboot · · Score: 1

      Well, we are not talking about the iPhone, which code quality and testing is deteriorating with any new iOS since the late Jobs left. It's about specific hardware that requires specific software to do specific tasks, that require specifically competent people. ESA staff is expected to provide an outcome of the highest quality. Nobody knows yet for sure the cause of the Mars lander crash.

      --
      Slashdot, fix the reply notifications... You won't get away with it...
    3. Re:Considering the decline in code quality... by wonkey_monkey · · Score: 4, Funny

      everything is going to hell.

      No, everything is calling hell() as a function.

      --
      systemd is Roko's Basilisk.
    4. Re:Considering the decline in code quality... by gTsiros · · Score: 1

      the fact that the possibility of "software bug" is even listed as a possibility, speaks volumes.

      --
      Looking for people to chat about multicopters, coding, music. skype: gtsiros
    5. Re:Considering the decline in code quality... by syntotic · · Score: 1

      Please, do publish the names and faces of ESA programmers! Anyone thinks you can program with a few years or even months of expertise and such is not the case. I would still suspect SCHIZOPHRENIC sabotage: the programmer was right, but a voice told him... therefore he does something different. International competition? GO TO HELL. I cannot live 500 years to wait to have more information when all time spans for space endeavour a more than a lifetime long.

    6. Re:Considering the decline in code quality... by Drethon · · Score: 1

      Done right, C++ classes can run pretty efficiently. In any of my classes, I know exactly what memory that class is occupying. Dynamic memory allocation is easy enough to manage, much more flexible than memory maps. Debugging is far easier with today's IDE, rather than doing by watching an I/O pin on an oscilloscope or LED.

      Java and C# can be worse memory wise but still manageable if you know what you are doing and I've found C# in particular allows me to prototype code a lot faster. I'm taking an algorithms class right now that lets us implement our algorithms in any language and C# doesn't run that much slower, despite all of the overhead.

    7. Re:Considering the decline in code quality... by WallyL · · Score: 1

      No, goto hell is still worse!

  7. Nuke the bugs from orbit by penguinoid · · Score: 1

    It's the only way to be sure.

    --
    Don't waste your vote! Vote for whoever you want, unless you live in a swing state it won't matter anyways
  8. sounds familiar by v1 · · Score: 1

    The most likely culprit is a flaw in the craft's software or a problem in merging the data coming from different sensors, which may have led the craft to believe it was lower in altitude than it really was,

    I don't remember which lander, but a previous one somewhere suffered a similar problem, mistaking landing leg deployment for surface contact. (the legs came down, and when the hit full stop the bounced back up a bit, triggering it to think the foot hit the surface) which caused it to shut off the landing retrorockts and it dropped like a rock from a good height, destroying the lander on impact with the surface.

    You'd think they would learn from the mistakes of the past? Lower gravity messes with sensors, and you have to predict how they'll perform on another planet that has different gravity, pressure, etc. "don't rely on any one sensor to tell you anything"

    --
    I work for the Department of Redundancy Department.
    1. Re:sounds familiar by Velox_SwiftFox · · Score: 1

      "So when do we know we're on the ground?" "Just have it switch modes when the accelerometer registers one Mars gravity."

    2. Re:sounds familiar by NormalVisual · · Score: 1

      "Just have it switch modes when the accelerometer registers one Mars gravity."

      All three accelerometers, and that they've registered no movement for long enough to confirm they're not moving. It's easy to armchair-quarterback stuff like this, but really, you'd think there was sufficient hardware on the machine to deal with problems like this.

      --
      Please stand clear of the doors, por favor mantenganse alejado de las puertas
    3. Re:sounds familiar by Tablizer · · Score: 1

      I thought radar was typically used to know how far one is from the ground? Seems a lot more straightforward than detecting thumps and bumps.

    4. Re:sounds familiar by RubberDogBone · · Score: 1

      I thought radar was typically used to know how far one is from the ground? Seems a lot more straightforward than detecting thumps and bumps.

      Radar takes a lot of power to run and a non-trivial amount of weight and space for something you only use ONE time. So when possible they try to come up with other ways to do it.

      In this case I have no idea what they were using but nobody has mentioned radars, but they have mentioned that the lander was running on very small batteries designed to only last a couple days on the surface. This suggests they didn't have the power for a radar.

      --
      Sig for hire.
    5. Re:sounds familiar by thinkwaitfast · · Score: 1

      You don't experience lower gravity until after you land. Everything up to that point is freefall and not detectable by accelerometers other than the force from the engines.

    6. Re:sounds familiar by thinkwaitfast · · Score: 1

      They don't take that much power (maybe a dozen watts) and have been used on all previous landings. Other than 3d stereoscopic estimates, I don't see another way and nobody is sending processors that powerful out into space yet.

    7. Re:sounds familiar by Solandri · · Score: 4, Interesting

      Usually when that sort of thing happens, it's not because the programmer did something obviously wrong. It's usually because the programmer had two (or more) competing scenarios to design for. He tried to design something which would split the difference, and ended up erring too much to one side.

      Lufthansa flight 2904 is a good example. The plane had to land in an expected crosswind on a wet runway. A crosswind landing requires landing with the plane's orientation misaligned from the runway. The plane is pointed into the crosswind, so is actually landing diagonally, then when it hits the ground it has to quickly yaw so it's aligned with the runway (so the wheels are pointed in the right direction). The way this is done is it lands on one gear first, pivots around on that gear to point the nose at the end of the runway, then drops down the second gear, then the nose gear.

      The A320's flight computer was programmed to avoid the disastrous scenario of a thrust reverser deploying in mid-air. It prohibited deployment of the thrust reversers unless both rear landing gear had 6.3 tons of force each on them. Full deployment of the spoilers (disrupts lift to plant the plane firmly on the ground) was prohibited unless the 6.3 tons criteria was met or the wheels were spinning faster than 72 knots.

      Unfortunately, in flight 2904's case, the crosswing landing maneuver placed most of the initial the force on a single landing gear, so the thrust reversers didn't deploy. The wet runway caused hydroplaning so the spoilers failed to deploy, hindering the pilots from getting the second landing gear down. By the time the above criteria were met and the plane began slowing down, it was well past the halfway point of the runway, and ended up going off the end. Design criteria selected to prevent one type of accident inadvertently caused another.

    8. Re:sounds familiar by the_other_chewey · · Score: 1

      Pretty sure you're thinking of Sonar, not Radar, which as the OP says, does use lots of power (and a vacuum tube, which means a separate high voltage power supply, no such thing as solid state radar, because SS can't handle the power needed).

      Welcome, time traveller from the 1970s, to the year 2016, where 3.3V low-power solid-state radar most definitely is a thing, and is mass-deployed in cars and other moving objects.

    9. Re:sounds familiar by v1 · · Score: 1

      if t were a vacuum, you'd be right. But there is no true freefall in an atmosphere. The air slows you down, and you do experience some gravity. (if you can reach terminal velocity, you'll be experiencing 100% gravity, as gravity is trying to accelerate you, but can't) The denser the atmosphere (which varies depending on your altitude) the more gravity you feel. They frequently rely on that to deploy the landing gear, it doesn't need to be powered when landing on a body with appreciable gravity.

      It's admittedly somewhat complicated and has several factors working on it, but it shouldn't have been a surprise to anyone. And yes to an earlier poster, another lander had this same problem with the sensors interpreting jettison of the heat shield as surface contact. I really don't understand how you don't put a sanity check in place there, or wait to 'arm' the ground detection until it's at least getting NEAR the ground. Things like heat shield jettison and parachute deployments should flat out disable a number of sensors for awhile until the physical shocks have had time to dissipate. Surface contact detection has NO business being turned on that early.

      --
      I work for the Department of Redundancy Department.
    10. Re:sounds familiar by sjames · · Score: 1

      I would suggest more defensive programming next time. For example, it isn't reasonable that the probe could make ground contact only 3 seconds after firing the landing thrusters. Determine the minimum possible time that is reasonable and don't even consider shutting the thrusters down until that much time has elapsed.

    11. Re:sounds familiar by sjames · · Score: 1

      Compare to the cost, weight, and power consumption of simple contact switches.

  9. Re:easier to fix? by hey! · · Score: 2

    What the hell is that "easier to fix" comment about?

    How are you going to issue a software patch to the pile of rubble on another planet? This is not a situation where you can ship the product without testing and fix it in firmware later!.

    I've been doing a lot of reading about the early space programs of the US and the Soviet Union, and that context the meaning is clear: you can use the same approach in the next Mars landing attempt; you don't have to redesign an entirely new system.

    "Rocket science" is hard, because you not only have to be smart, you have to be able to stand repeated failure. Normal people when faced with a spectacular fiasco give up, or they wipe the slate clean and start over. But in something as complicated as a mission like this you have to look at it this way: from a vehicular standpoint everything worked like a charm right up until the last three minutes or so of the trip.

    --
    Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
  10. Re:easier to fix? by plover · · Score: 4, Funny

    How are you going to issue a software patch to the pile of rubble on another planet? This is not a situation where you can ship the product without testing and fix it in firmware later!.

    It's Agile. The product owner will raise this issue as a priority in the backlog, they'll fix it in this sprint, and it will ship in the next release.

    --
    John
  11. bad code bad code by turkeydance · · Score: 1

    what 'cha gonna do

  12. Well if that's the only problem... by MiniMike · · Score: 1

    But software glitches should be easier to fix than a fundamental problem with the landing hardware

    Oh good, then just fix the glitch, recompile, and restart the landing sequence. Lucky they sent the Debug version the first time, maybe they should try the Release next?

  13. Than why black oily explosion? by Billly+Gates · · Score: 2

    A quick glance at the low resolution screenshot showed an explosion with black soot. Engineers said it was caused by the rockets still being on.

    If they were turned off it would leak fuel in It's crater but would not ignite

    1. Re:Than why black oily explosion? by Anonymous Coward · · Score: 1

      The lander used monopropellant engines fueled by hydrazine, which decomposes into hydrogen and nitrogen exothermically (~800ÂC) when passed by an appropriate catalyst. In the case of a crash landing, it probably decomposed a lot more vigorously, and that much heat presumably would've affected the composition of the Martian soil enough to turn it black.

  14. QA by bradgoodman · · Score: 4, Informative
    I've been in organizations that had pretty light SQA departments. I used to say that the "really" good shops had 1-to-3 ratios - 1 engineer doing QA for every 3 doing implementation. When I started working for more "mission critical" stuff - that ratio went even higher.

    I know people that work in companies that design chips. Those manufacturing cycles are MUCH longer and expensive - you can't just recompile when you test and find a bug. This, their QA is probably more like 10 people doing simulation (behvioral, thermal, timing, power, emissions, RF susspetabiliy, etc) before a design is even fabricated.

    I would imagine that in Space Exploration - this would go even higher - given the time and expense of these missions. The point is - saying "it's just software" doesn't help you here. Software is *very* complex and the intricacies of advanced logic, variability of factors - trying to do this stuff probably dwarfs that of the hardware components in this day and age.

    1. Re:QA by ShakaUVM · · Score: 3, Informative

      >I would imagine that in Space Exploration - this would go even higher - given the time and expense of these missions.

      It is. Well, at least it is at JPL - I've gone through their coding standards and testing process for spaceflight, and it's extremely intensive.

      I watched a video on their standards before, and without rewatching it I don't know if this is the same one, but it looks pretty good skimming through it.

      https://www.youtube.com/watch?...

      I'd be really interested in seeing someone go through the process and finding out where it went wrong.

    2. Re:QA by thinkwaitfast · · Score: 1

      My experience is the opposite. One instance we had 3 development engineers to about 8 test engineers. This was for a safety critical system.

    3. Re:QA by johannesg · · Score: 5, Informative

      I work for a company that writes those simulations. Generally a simulation consists of a CPU emulator that runs the onboard software, and a whole bunch of models for each aspect of the spacecraft and environment: the orbit model, the communication model, various instrument models, etc.. These systems are generally set up to allow gradual replacement of each model with real hardware as it becomes available, so the software development is already underway long before the spacecraft hardware has even been built. Each model is a hard real-time program (to allow drop-in replacement of hardware), and has extensive capabilities for error injection in order to simulate things like flipped bits, broken communication channels, broken sensors, etc.

      I don't know what happened on Schiaparelli and they weren't a customer of ours anyway, but a scenario where a sensor breaks and sends bogus information could and should have been tested for during development.

      I'm not sure what the software engineer:QA ratio is - most of that happens internally by the spacecraft people. You run into their QA people everywhere though, while I have yet to personally meet my first flight software engineer.

      Oh, and back in the day I wrote the very first software-only environment for testing flight software on the ground. Up until then, the test environment used real hardware for the flight computer, thus requiring an expensive second set of flight computers just for doing the onboard software development. I hacked together a proof of concept that showed that you _could_ in fact model and simulate the flight computer as well, leading to a substantial cost saving on space projects since...

      The flight computer _simulator_ generally speaking runs on Linux. I'm not sure what the models use these days, but I have seen IRIX and Sun systems around for this purpose. As for the flight computer itself, VxWorks is not an uncommon choice of OS, and the on-board CPU is usually something like ERC32 or Leon - both are radiation-hardened SPARCs.

    4. Re:QA by NotAPK · · Score: 1

      Great post, thanks.

  15. We need the old programmers back. by Opportunist · · Score: 1

    The kind that grew up in a world where the code you delivered had to work because you can't simply ship an update after you find out it barfs in all but laboratory conditions. I am guilty of it myself, I have to admit, I start to slack and deliver bananaware because, hey, a cursory test will do, if everything fails, just send a patch to the customer!

    We need programmers back that knew how to write code that, you know, WORKS!

    --
    We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    1. Re:We need the old programmers back. by NormalVisual · · Score: 1

      You said it yourself - the problem isn't so much the programmers as it is management. Programmers often want to validate/verify/test their work a lot more, but if management's attitude is "ship it!", the coders don't stand a chance no matter how good they are. My experience is that if you're stuck on a difficult problem that only shows itself under very hard-to-duplicate conditions and are spending lots of time trying to fix it, you get written up instead of appreciation for trying to create a quality product.

      --
      Please stand clear of the doors, por favor mantenganse alejado de las puertas
    2. Re:We need the old programmers back. by zapadnik · · Score: 1

      A lot of scientific software isn't written like commercial software. Scientific software does amazing things, but the source code is often completely awful. I know, I used to write scientific software. Once I started looking at how commercial people wrote software I got much better - but very many scientists are great at science but write terrible, terrible code from the perspective of professional commercial developers. It may not be the case in the instance of this lander, I'm just putting it out there that people great at science can be completely awful and undisciplined when it comes to software (like not writing automated Unit/Integration/System Tests and relying on manual testing only - often because they don't know about Unit Testing - few physicists know about modern software engineering practices because they are not taught them).

    3. Re:We need the old programmers back. by NotAPK · · Score: 1

      I actually make a good chunk of my living re-writing scientific software. My physics PhD allows me to come rapidly up to speed on the domain knowledge that went into the original code, and then I can apply software development best practices to produce a final product that is actually maintainable. However, I would be surprised to hear that any actually scientists would have contributed functional code to this space probe mission. At the end of the day, such software is an engineering item: said scientists would not have been tinkering around with the rocket engines, fuel tanks, or flight path, any more than they would have been pissing about with the landing module descent code. Of course, the loss of the payload suggests otherwise... :)

    4. Re:We need the old programmers back. by Opportunist · · Score: 1

      The difference is that an old progger will probably be more inclined to say "screw you, pimple face, I don't give a shit what you learned in your MBA courses, obviously it's short for mastering bullshit annoyingly. Now go back into your office, play with your numbers and don't stand in the way of people who're actually working. It ships when it's done".

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    5. Re:We need the old programmers back. by Opportunist · · Score: 1

      Come to Europe. We're hiring!

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    6. Re:We need the old programmers back. by NormalVisual · · Score: 1

      And I've done just that, followed by a visit to HR for being "insubordinate". My favorite was when we had a project that got cancelled halfway through. I took great care to make notes in the source that it wasn't finished, hadn't been properly tested, and under no circumstance should it be shipped. About a year later, I got written up because "my code" was causing crashes in a another customer's mission critical system. They didn't tell me specifically what the problem was, so I went exploring through the commit logs. Apparently some knob found a class in that source that he thought would do what he wanted, incorporated it into production code for a customer, and then when the customer started having problems, *I* was the one that got written up for it because I was the one that had checked it in a year prior. The fact that I'd committed to a non-production branch and clearly marked it as "not finished and not to be used in its current state" didn't matter at all.

      Me: "So, you'd rather I just discarded thousands of dollars' worth of work instead of committing it to a development branch (also clearly marked as such)?"
      Dev mgr: "No, of course not."
      M: "So why am I being written up?"
      DM: "Because you're the one that checked it in in an unfinished state."
      M: "Really? You told me to move off that project to something else."
      DM: "Yes, but that code wasn't finished and shouldn't have been committed."
      M: "But you just said you didn't want me to toss that code and also didn't want me to spend any more time on it."
      DM: "Right, but you should never be committing code that isn't finished."
      M: "It was committed to a fricking INACTIVE DEV BRANCH!"
      DM: "Doesn't matter, the code made it into production."

      The idiot that actually merged that code into production got off scot-free. Needless to say, I'm no longer associated with those morons.

      --
      Please stand clear of the doors, por favor mantenganse alejado de las puertas
    7. Re:We need the old programmers back. by dunkelfalke · · Score: 1

      Old programmers also made mistakes. Remember the Morris worm? It has used vulnerabilities in several Unix applications.

      --
      "It's such a fine line between stupid and clever" -- David St. Hubbins, Spinal Tap
    8. Re:We need the old programmers back. by Opportunist · · Score: 1

      "I have to give you 2 weeks notice. I have more than 2 weeks of vacation time standing. See you. Or rather, never gonna again."

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  16. this is why they need to drop test it and then... by WindBourne · · Score: 1

    send it to space to land on earth.
    The fact is, that any approach that will work on Mars, with minor mods, will work on Earth. So, it is easy enough to test this.
    Once ESA has a REAL FULLY TESTED LANDING SYSTEM, then everything else is 1 offs.
    And considering the money that ESA has spent on going to mars, only to crash, it would be worth their effort to full test one.

    --
    I prefer the "u" in honour as it seems to be missing these days.
  17. Government work = lowest bidder by pubwvj · · Score: 1

    Government work = lowest bidder
    You get what you pay for, sometimes.

  18. Re:easier to fix? by Waffle+Iron · · Score: 1

    Easier to fix - in that, "we don't have to spend 10 years redesigning the entire fucking lander, we just need to fix the fucking software and send a new fucking copy of the same hardware in six fucking months."

    Jesus fucking christ, are you really that fucking Asperger's?

    I don't see much evidence that the OP has Asperger's, but you may want to get checked out about your possible case of Tourette syndrome.

  19. Re:easier to fix? by Anonymous Coward · · Score: 2, Informative

    As a manufacturing engineer I can tell you from experience even in tightly regulated industries the instances of the print not matching the part is more common than you would think, even on parts produced for decades. When you are talking about one-offs that just self-destructed on another planet and cannot compare the as-produced part to the print it becomes exceedingly difficult to account for last-minute design changes.

  20. What you git by Tablizer · · Score: 1

    I told them not to run Microsoft Lander, but nobody would listen. "But everyone else is using it blah blah."

    1. Re:What you git by david_thornley · · Score: 1

      They didn't even upgrade to Kerbal Space Program.

      --
      "When you have eliminated the unacceptable, whatever is left, however improbable, must be the truthiness" - Holmes
  21. Re: easier to fix? by Anonymous Coward · · Score: 1

    That one indeed. I remember years ago in the manufacturing facility where my father worked a machinist retired and suddenly the quality of a certain part went down to unusable. What they found out that the part was rotaionally casted and the original machinist didn't read the actual casting instruction that came with the blueprints. He was an experienced machinist and knew what part he had to deliver and he always delivered it just right. The ones that the guy who came after him produced were brittle and not well formed. It turned out that the old guy regulated the rpm by feeling of material splatter and vibration, and on average he spun the mold about 3 times more than prescribed. The engineer who did the blueprint newer knew.

  22. Re:easier to fix? by dbIII · · Score: 1

    I've been doing a lot of reading about the early space programs of the US and the Soviet Union, and that context the meaning is clear: you can use the same approach in the next Mars landing attempt; you don't have to redesign an entirely new system.

    You can't if the people who did it before and were helping since 2009 pull out in 2012, take their bat and ball and go home.

    NASA pulled out due to budget cuts and IP restrictions meant the ESA couldn't use their stuff after they pulled out.
    So in this case politics was the enemy of your common sense idea.

  23. Re:this is why they need to drop test it and then. by dbIII · · Score: 1

    The xplane software has an interesting Mars atmosphere simulation mode that shows how wildly different things are.
    Having to fly near the speed of sound to avoid stalling and controls not having much of a grip on the air to respond are two things that rub in the many differences.

  24. My Theory by laing · · Score: 2
    Here's my take on it: The lander's radar got a reflection from the plasma from the decent engine and indicated close proximity to the ground. The software then did exactly what it was programmed to do -- it shut down the thrusters. Once that event occurred, the software entered a new state and possibly even shut down the ground radar. Thus it was doomed from that point forward. No amount of pre-mission testing could have detected this scenario.

    The above is complete speculation, but I believe that there's a good chance I am correct.

    1. Re:My Theory by ThosLives · · Score: 1

      Kind of reckless if they did not have plausibility checking in there. Standard practice is to put checks in that won't even start looking for the ground until an appropriate amount of time has passed - to avoid exactly that sort of thing.

      I can't imagine why you wouldn't put that sort of check in. Well no, I guess I can imagine it, and it makes me sad.

      --
      "There are a dozen opinions on a matter until you know the truth. Then there is only one." - CS Lewis (paraprhase)
    2. Re:My Theory by DerekLyons · · Score: 1

      The lander's radar got a reflection from the plasma from the decent engine and indicated close proximity to the ground.

      Not very likely, even if the engine exhaust was plasma, the geometry of the spacecraft and the low density of the exhaust mitigate against that occurrence.

  25. Software bug, or... by Cochonou · · Score: 1

    From what is written in TFA, it could be a software bug, but it could be as well a sensor fault. It's probably too early to figure exactly what happened. Nevertheless, it is likely the best to present it that way for now, as a software bug is easier to fix than re-designing the sensor suite.

  26. Known issue by TJHook3r · · Score: 1

    Did they have any coders recently imported from console gaming backgrounds? They have a very relaxed view on fixing bugs after go-live!

  27. Re: easier to fix? by NotAPK · · Score: 1

    This is common everywhere: how to capture specific domain knowledge? And remember, it's a two-way process. The knowledge of the current expert has to be recorded in some way, and then, the new guy has to be trained to the intricacies of the previous procedure.Mix in the observation that some players may not want this process to be successful, and you're probably boned before you even realise it.

  28. Re:easier to fix? by Fragnet · · Score: 1

    You can do it with something called a Version Control System, which is actually quite easy to set up. Granted a lot of engineers find it tedious to have to commit their designs but you know, whatever.

  29. Mundane Details by bozzy · · Score: 1

    Thrusters, designed to decelerate the craft for 30 seconds until it was metres off the ground, engaged for only around 3 seconds before they were commanded to switch off

    I must have put a decimal point in the wrong place or something. Shit! I always do that. I always mess up some mundane detail!

  30. Re: easier to fix? by cwsumner · · Score: 1

    This is common everywhere: how to capture specific domain knowledge? And remember, it's a two-way process. The knowledge of the current expert has to be recorded in some way, and then, the new guy has to be trained to the intricacies of the previous procedure. ...

    The place that records the previous expert's knowledge is called "Source Code". Destroy that and you will start as an ignorant beginner. 8-)

    Don't believe what "everyone says". ;-)

  31. Re:easier to fix? by dbIII · · Score: 1

    So not just the ESA, but you want to concede defeat in Korea and watch Seoul burn?
    I think you are a little messed up.

  32. Unsubstantiated gossip by Zoxed · · Score: 1

    Seems a little unprofessional for the head of solar and planetary missions to be publicly spreading theories for which he has zero evidence, even if they are qualified ?

    On another note, as a Software Engineer I know for certain that 99% of all failures are Hardware related ;-)

  33. Software bug on Mars. by manifestdestinynow · · Score: 1

    Isn't it astounding that in 1968 NASA sent a mission to the moon, with hand wired graphite memory ropes.. termed by NASA as "Little Old Lady" memory, and with less memory than a commodore 64 they sent men to the moon, landed, toured, came back to the ship, then came back to earth? Anyone who believes any of NASA's lies, has to be among the most gullible people in the world.

  34. Re: this is why they need to drop test it and then by WindBourne · · Score: 1

    and yet, anything designed for Mars, will land here and all of the systems would have been tested.
    this particular issue would NOT have happened.

    --
    I prefer the "u" in honour as it seems to be missing these days.
  35. Re:this is why they need to drop test it and then. by CanEHdian · · Score: 1

    Can you fly a modified VTOL aircraft (Super MarsHarrier) in a special semi-vertical mode so you have upward thrust AND forward thurst?

    --
    When the copyright term is "forever minus a day", live every day like it's the last.
  36. Re:this is why they need to drop test it and then. by dbIII · · Score: 1

    If you have some incredibly huge jet engines (to compress that not very dense air) that can pivot why not, but rockets sound easier. It's not so much a plane then as a "flying bedstead" like the Eagle lander simulator of the 1960s.