Slashdot Mirror


The Software Behind the Mars Phoenix Lander

chromatic writes "Imagine managing a million lines of code to send over seven hundred pounds of equipment millions of miles through space to land safely on Mars and perform dozens of experiments. You have C, 128 MB of RAM, and very few opportunities to retry if you get it wrong. O'Reilly News interviewed Peter Gluck, project software engineer for NASA's Mars Phoenix Lander, about the process of writing software and managing these constraints — and why you're unlikely to see the source code to the project any time soon."

45 of 152 comments (clear)

  1. Re:Great software! by QuantumG · · Score: 4, Informative

    Nope. VxWorks.

    These questions and more answered in TFA.

    --
    How we know is more important than what we know.
  2. Re:128MB? by Anonymous Coward · · Score: 5, Funny

    / You seem to be trying to make a joke \
    | would you like me to:                |
    |  o Help me come up with new jokes    |
    |  o Help me bash Bill Gates           |
    |  o Help me spell Microdollarsignoft  |
    |  o No thanks, let me keep making     |
    |     a redundant ass of myself        |
    \   P.S.: **** open sores             /
         \
          \
           \     ____
            \   / __ \
             \  O|  |O|
                ||  | |
                ||  | |
                ||    |
                 |___/
    --
    cpu0: Microsoft Clippium ("GenuineClippy" ChromedMetal-Class). Paperbinding, lockpicking, fish-hook-hack support.
    template greedily stolen from this guy: http://slashdot.org/~ClippySay

  3. wow, long article, here's the answer to the teaser by deft · · Score: 5, Interesting

    basically, its because the code is part of a space vehicle regulated by international arms and trafficking laws. That means Joe Blow doesnt get it.

    Sorry dude, you're Joe Blow. Unless you're reading this from a JPL/NASA'ish sort of place. Then you're just smirking.

    ===================

    FTA:
    Sort of on a different topic, I have a quote here. One of our editors talked to Frank Hecker from the Mozilla Foundation the other day.

    Okay.

    In that talk, he suggested that all software developed by the Federal Government should be released to the public domain or a very, very liberal open-source license. That's not even a copyleft license. Does the American public have any access to the source code currently on the Phoenix? Are there plans to make some of the source code available?

    Well, no. There are no plans to make that available. And one of the issues that we have is that our spacecraft are designated as subject to international trafficking and arms regulations. So even --

    Crypto regulations in exporting and such?

    Yeah. Yeah. I mean even though these are not military spacecraft, the technology used in them is space technology. And so the State Department does not allow us to release anything that we've done in terms of technical details to foreign scrutiny. Now, in fact as I said, we have a team of Canadians. The Canadians delivered our meteorology instruments, and we had to be very careful about our relationship with them and how much we could disclose to them.

    Really?

    Yeah. Yeah.

    I can see that in applying control software, but how about the payload software?

    Even the payload software -- in this particular case, remember that the payload software operates within the confines of the RAD 6000 that contains the spacecraft software. And although the newer versions of real-time operating systems allow you to compartmentalize better, the older ones are just global name space. So there really wasn't any way to allow them to provide software for the MET instruments. So we had to define an interface and build the software at JPL, and then do our integration testing. And we worked closely with the Canadians in terms of the integration testing and making sure that the software was going to do what they needed it to do.

    Right.

    But we could not actually release the source code to them.

    --

    There's nothing Intelligent about Intelligent Design.
  4. Re:Millions of lines? by QuantumG · · Score: 4, Interesting

    Well, of course, the proper response to your query is "it doesn't work like that" or "neither are a good metric" or something, but that's a big boring, so let's consider an empirical result.

    liblink-grammar.so.4.3.5 is 616129 bytes. It is built from 23289 lines of code. So that's about 26.4 bytes of code per line.

    So 128 MB of RAM can hold about 5,084,005 lines of code :)

    --
    How we know is more important than what we know.
  5. Not like the olden days by Average · · Score: 4, Interesting

    I'm curious how many old kinds of code we're still communicating with. FTA, Cassini is ADA-based. I know the Voyager craft are in FORTH (my first programming love).

    1. Re:Not like the olden days by frieko · · Score: 4, Informative

      I wouldn't call Ada obsolete. C is only good because a lot of people know C and lots of COTS parts are C. But you can't beat Ada if you need a language that was built from the ground up for mission-critical reliability.

    2. Re:Not like the olden days by Tablizer · · Score: 2, Funny

      I'm building a PDP-11 for a nuclear reactor.

      Divide by zero and run like hell!
         

  6. Nope, its written in C. by deft · · Score: 3, Funny

    But you didnt read the article, you were more just hoping for a slashdot linux rally cry or something, werent you.

    But if someone crys in a dark basement creepily lit by a monitor, does anyone here it?

    Damn, i guess I did.

    --

    There's nothing Intelligent about Intelligent Design.
    1. Re:Nope, its written in C. by Anonymous Coward · · Score: 2, Funny

      Thats OK...Too much Wine crashes my Linux too

  7. Re:wow, long article, here's the answer to the tea by QuantumG · · Score: 5, Insightful

    This is basically the reason why space technology is so primitive. The science has been stifled for years by government regulations.

    --
    How we know is more important than what we know.
  8. How hard can it be? by Dex5791 · · Score: 4, Funny

    Just open the existing code base for the previous lander and cut&paste.

  9. Re:Millions of lines? by neokushan · · Score: 3, Insightful

    For legacy reasons that have just sort of stuck, the maximum line size in C is often no more than 80 characters long, although plenty of people ignore this unwritten "rule" these days.
    I'd say you can safely assume that each line is around 80 characters, though, as a lot of lines will use very few.

    But it's all irrelevant as I doubt they'd bother transmitting the entire source code to Mars when they can just compile it into a good ol' binary that's probably a hundredth of the original source code's size, if not more.

    --
    +1 IDisagreeSoHeMustBeATrollOrAnAstroturferOrAShill
  10. Re:wow, long article, here's the answer to the tea by megaditto · · Score: 4, Insightful

    It's a tricky balance though. Nuclear missile launch codes are also -- technically -- public property, yet I am not sure it'd be a good idea to release that in the public domain.

    I think the way things are handled right now is the best we are going to get: basic science is open, applied scientific results are secret.

    --
    Obama likes poor people so much, he wants to make more of them.
  11. Re:wow, long article, here's the answer to the tea by QuantumG · · Score: 4, Interesting

    Neither the basic science, nor the applied science (aka engineering) is open.

    The only reason any of us know the rocket equation is because it was invented before these laws were.

    --
    How we know is more important than what we know.
  12. Your statement is flawed. by flattop100 · · Score: 5, Insightful

    Space technology is not "flawed." It is rigorously tested to survive A)Lift off B)Months and years of dormancy C)Descent D)Operation on another planet millions of miles away, with minutes-long latency. Beyond that, it has to be tested time and again to make sure there are NO errors. If you computer at home freezes, you hit reset. Trying pushing the reset button on a Mars rover--let me know how that works out for you. Space technology is not primitive. It may seem simplistic, but that's to guarantee functionality. Read the definition of "mission-critical" and think about what you typed there. It's a little different that "recreational software development."

    1. Re:Your statement is flawed. by mystik · · Score: 2, Funny

      Well. They really can make the software as complex + bug ridden as they want.

      They only have to make sure that the code that runs the robot that presses the reset button is bug free :)

      --
      Why aren't you encrypting your e-mail?
    2. Re:Your statement is flawed. by Lumpy · · Score: 4, Informative

      I know exactly how to do that. I program Crestron systems remotely as well as some ham radio embedded stuff. Cresron gear across the country cant be reset without paying someone to do so, and an embedded PC in the norther wild of michigan on a tower that takes a weekend to go and fix are very much the same thing.

      You check, recheck, and test on the copy system. you also build in fail safes so that if your upload fails, it reverts to the old code or fails to the loader so you can upload again.

      --
      Do not look at laser with remaining good eye.
    3. Re:Your statement is flawed. by Kerkyon · · Score: 3, Insightful

      Beyond that, it has to be tested time and again to make sure there are NO errors.

      Well, no, not really. It's far more economical to ensure that the system can recover from serious errors and that errors are fixable, rather than try to assure zero errors. Mainly because the former is possible and the latter isn't.

      If you computer at home freezes, you hit reset. Trying pushing the reset button on a Mars rover--let me know how that works out for you.

      This is addressed in TFA -- the watchdog timer takes care of it; they're ubiquitous in high-reliability embedded systems.

    4. Re:Your statement is flawed. by coop0030 · · Score: 2, Informative

      If you computer at home freezes, you hit reset. Trying pushing the reset button on a Mars rover--let me know how that works out for you.

      Actually, in the article, he basically stated that they essentially do have a reset button. The software pings the hardware every 64 seconds. If the hardware doesn't receive the ping on time, it resets the software to potentially resolve the error/lockup.

      Now in the event that that doesn't work, we have a whole second set of avionics onboard. So the hardware will try to boot to the same side, and if the same side doesn't come up and start stroking the watch-stop timer, then it will swap to the other side and boot the first side.

    5. Re:Your statement is flawed. by imsabbel · · Score: 2, Informative

      Also, people often forget that by the time a probe or rover makes the news, many years have passed since the "feature freeze".
      Years of design, testing, waiting for a launch possibility, travel.

      --
      HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
    6. Re:Your statement is flawed. by Lumpy · · Score: 2, Informative

      Yes I can, if I go to the tower to connect to it. Although last time I disconnected all those wires because lightning uses those 200 foot wires coming down as antennas and destroys the computer running the digi. so right now you have to drive to the location, climb the tower, get the box, climb down, work on it, test it, climb up reinstall it and then climb down.

      i'd rather make sure I get everything right and upload via 6 meters at 1200bps overnight from 30 miles away.

      --
      Do not look at laser with remaining good eye.
    7. Re:Your statement is flawed. by DerekLyons · · Score: 2, Interesting

      Beyond that, it has to be tested time and again to make sure there are NO errors.

      Well, no, not really. It's far more economical to ensure that the system can recover from serious errors and that errors are fixable, rather than try to assure zero errors. Mainly because the former is possible and the latter isn't.

      Well, the shuttle software has zero bugs - or seemingly as close to it as to be indistinguishable from zero. The software for the [nuclear tipped missile] fire control system I used to work on in the Navy, as well as the firmware in the missile had absolutely zero bugs... Its not impossible to get zero bugs, its merely very damn expensive.

  13. Canadians! by vimm · · Score: 3, Funny
    FTFA

    We have a team of Canadians... we had to be very careful about our relationship with them... how much we could disclose to them

    In soviet canada, mars probe software discloses you?

  14. Re:Great software! by Durindana · · Score: 4, Funny

    you mean...

    Answers to these questions and more, rendered inaccessible by /.

  15. Re:128MB? by Anonymous Coward · · Score: 2, Funny

    Micro$soft?

    Good job.

  16. Re:Why unlikely to see the source? by rk · · Score: 4, Informative

    NASA releases all kinds of code. As an example, many people in the space science community rely on SPICE from JPL's Navigation and Ancillary Information Facility, and you can play from home. I think the newest version of Celestia has a CSPICE interface to get extremely accurate planetary positions and spacecraft pointings into it.

  17. Re:Why unlikely to see the source? by exley · · Score: 2, Informative

    I stand at least partly corrected then... Although still not sure they would release code like what is running the Mars lander. On the other hand, what would anyone do with that code if it was available? I suppose there could be some homebrew interplanetary lander projects out there... :)

    And for anyone else initially confused and unwilling to click links, the reference to SPICE in parent's post isn't about the circuit simulator!

  18. Re:wow, long article, here's the answer to the tea by deft · · Score: 2, Informative

    I was told that the reason space craft run on very old perating systems is not because of the ideas you're thinking, but because old operating systems have basically no unknown quirks. They are having anything unknown pop out.

    the actions the have to undertake are just fine on old tech, because OLD = PROVEN in alot of cases.

    --

    There's nothing Intelligent about Intelligent Design.
  19. Huh? by oneiros27 · · Score: 4, Informative

    Maybe because it's freaking NASA?

    What's that have anything to do with it?

    NASA has an OSI approved license:

    It could probably be easier to find NASA software, and I doubt this particular software would ever be released, but there's lots of NASA software that's been released:

    There's issues because much of NASA stuff is done as part of grants, and so it's officially owned by the academic / research institution that won the grant ... as such, there might be other NASA funded code that's out there, that you don't know is NASA code... at least one program (AISRP) has started a place to collect software by grantees.

    I've been to NASA workshops where there's plenty of code that's being written where people would LOVE to have their software find a broader audience. At the last one, we had an hour debate on if we were allowed to release code as GPL, as that'd place restrictions on the use of the code (that derivative copies have to be open), which should not be done as the software was developed w/ federal money and as such citizens should be free to do whatever they want with it. I think someone was assigned to talk to NASA's legal department and find out what we had to do to release our code.

    --
    Build it, and they will come^Hplain.
  20. My eyes, they burn! by geekoid · · Score: 5, Insightful

    What a horrible interview.

    WHen interviewing someone, you don't tell them what they know, you ask them.

    An example:
    "That's not a really beefy embedded board actually. It's what, thirty-three megahertz?

    Yeah. That's â" yeah.


    About 128 megabytes of RAM?


    That's right.

    I imagine that produces some interesting challenges, getting all of that software to run together on that board while also having it land on the planet successfully.
    "

    Painful.

    --
    The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
    1. Re:My eyes, they burn! by rho · · Score: 2, Insightful

      But just think how smart and well-informed the interviewer sounds!

      I stopped reading halfway through. Useless interviewer leads to useless interview.

      --
      Potato chips are a by-yourself food.
  21. Re:Um, right by Anonymous Coward · · Score: 2, Funny

    So how much of that 128MB does the VBRUN60.DLL take?

    The same amount it takes on Earth.

  22. You know by geekoid · · Score: 5, Funny

    if they wrote it in perl, it would only be 1 line.

    --
    The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
    1. Re:You know by MadnessASAP · · Score: 5, Funny

      Yes, but it would use the unholy letters of the dark lord himself, and bring death and destruction upon any man who gazed upon its ghastly source.

      --
      I may agree with what you say, but I will defend to the death your right to face the consequences of saying it.
    2. Re:You know by dkf · · Score: 2, Funny

      Yes, but it would use the unholy letters of the dark lord himself, and bring death and destruction upon any man who gazed upon its ghastly source.

      Ah! I see you know APL...

      --
      "Little does he know, but there is no 'I' in 'Idiot'!"
  23. hello.c by xx_chris · · Score: 5, Funny

    #include void main() { printf("Hello Mars\n"); }

  24. Misleading summary by techno-vampire · · Score: 2, Interesting

    I suppose it's inevitable that the summary of a Slashdot article is inaccurate, but in this case it's highly misleading. The code in the Phoenix Lander has nothing whatsoever to do with getting it to Mars. The Spaceprobe Navigation Package (Are they still using MOPS and TRAM, I wonder? After all, they were good enough for Voyager I and II.) run on mainframes at JPL, in Pasadena, and course corrections are sent from their to the space craft. This is because the same programs doing the navigation for Phoenix can be used at the same time for other missions, instead of wasting valuable memory (and the energy needed to run them) on putting a separate copy of the program on every, single probe.

    --
    Good, inexpensive web hosting
  25. Re:Reset button on a lander by Buran · · Score: 4, Informative

    It's called a dead man's switch and is implemented in rail locomotives, for example. A horn sounds a tone at (probably) random intervals and you must press a button within a certain amount of time or the engine is throttled to idle. Previously, you had to keep your foot on a pedal at all times but it was defeated by just putting a brick on it.

    The idea is that if you're dead, you can't hit the switch, so the train you're supposed to be controlling will stop rather than plow through a stop signal at some later time and hit something or go off the track.

  26. Re:Reset button on a lander by ezzzD55J · · Score: 3, Informative

    It's called a dead man's switch and is implemented in rail locomotives, for example.

    Dead man's switch for humans, watchdog timer for computers. TFA mentions the phoenix watchdog going off every 64 seconds.

  27. Nah, space technology is just as junky as us. by tjstork · · Score: 2, Insightful

    Dude, the lunar lander program crashed repeatedly on Neil Armstrong and Buzz Aldrin right when they were trying to land on the moon. It was so bad, that mission control basically told them to ignore it and Neil went ahead and landed the thing really by the seat of his own pants. You really can't have a bug much more worse than that!

    When you think about it, space software is probably the most unreliable software there is. I mean, it is a classic cathedral design, has only a handful of users, and so, yeah, they can do a lot of testing, but, they miss stuff. Look at how often they have to upload patches to the ship while it is in flight.

    --
    This is my sig.
  28. So Peter? by codepunk · · Score: 2, Funny

    You're the project software engineer. I noticed that was singular. Is there just one product software engineer? Like sort of the managing engineer for the project?

    Yeah. Project Software Systems Engineer is the title. And our software was developed -- the flight system software was developed in three different locations. Lockheed Martin developed the spacecraft software, and then we had payload software developed by both the University of Arizona and the Jet Propulsion Laboratory.

    So Peter what is it that you do here?

    I hand specifications to engineers that write the flight software.

    So you take the papers to them?

    Well no my secretary does that...

    --


    Got Code?
  29. Re:Millions of lines? by ArcherB · · Score: 2, Interesting

    How many lines of code can 128 MB of RAM hold and what is the average 'line' for C?

    I don't know, but 15 years ago, I would have killed for 128 MB of RAM or even a 128 MB HDD. My first "PC" had 4 MB RAM and a 102 MB HDD. It ran DOS 6.2, Windows 3.1 and a host of crappy DOS games. (Actually, I don't think the DOS games used more than the first MB)
    Strip the GUI and even the CLI, and you'll find that 128MB is quite a bit if your main concern is code. Data could take quite a chunk of that, but if you're just talking about text files with data and configuration, a few MB could handle it with not problem.

    Now, once you're on the ground and you want to start storing some hires pics to send back to Houston... you better have a flash card stashed away on that thing somewhere!

    --
    There is no "I disagree" mod for a reason. Flamebait, Troll, and Overrated are not substitutes.
  30. Re:128MB? by ron+sepun · · Score: 3, Funny

    Micro$soft?

    Good job.

    Help me spell Microdollarsignoft == Micro$oft... Fail is you.

  31. Re:wow, long article, here's the answer to the tea by Tablizer · · Score: 2, Informative

    Here's an interesting web-page on the processors used in various space probes over the years.

    http://www.cpushack.net/space-craft-cpu.html

    It seems Viking was the first to use micro-processors. Before that they used TTL, which is sort of a roll-your-own CPU based on bunches of simpler logic chips (NAND gates, multiplexers, etc.).
       

  32. Re:Reset button on a lander by joeslugg · · Score: 2, Insightful

    6 bit counter (2**6) ?

    Just a guess...