'Extreme Programming' Controls Phoenix Mars Lander

pbd points out a story discussing the challenges faced by NASA engineers in designing the code sent to operate the Phoenix Mars Lander. Quoting Computerworld: "On Wednesday, engineers sent up the code to run an actual analysis, but a satellite orbiting Mars, which transfers the data down to the Lander, was offline. Robinson explained that the satellite had been hit with radiation, knocking it into safe mode. 'Space is a harsh environment, and sometimes they just go into safe mode,' he noted. 'It's a minor problem. [The satellite] aborts whatever it was doing and waits for future commanding.' Engineers successfully resent the code on Thursday."

Comments

That's coolness

[zappepcs]

Not sure about others, but designing software and systems (software & hardware) that understand the data they are using, and when conditions are not right AND know how to go to safe mode rather than just fall apart is awesomeness in physical form.

Then just considering it a minor problem is an even higher form of awesomeness :)

As a hobby I build small robotic creations. I can tell you that working with limited resources, and having to deal with the completely unexpected is just brain damage for the fun of it. Testing your new code on the same hardware you've had for months and thinking "I really did not know that you could bend this metal piece with so little force" or "why the hell is it doing that" and find out that you missed a decimal point on force calculations or a typo leaves you reacting with subroutine x instead of y.

Those engineers have to be fscking proud of their work. I know I would be. Some days I look at code I've had running 24/7 for several years and am amazed, not just at myself, but that I managed to find the bugs and fix the "I didn't know that would happen" issues. It's a lot of type and bounds checking to be certain, but something always creeps in.

I say they should be proud. The Mars missions have been nothing short of exemplary awesomeness. When they figure out the failure they did have, I'm certain that the absolute goodness they built into it will help reveal totally amazing discoveries about why there was a failure. Nothing simple like metric conversions, or wrong alloy for the screws.

Makes me want to work with them. :)

Re:That's coolness

[zappepcs]

Actually, considering it a minor problem is not irresponsible risk. When you are that cock sure of the equipment you designed that you can think of it as a minor problem when space radiation knocks your system into safe mode for a bit, you have awesome mad design skills, not risk taking behaviors.

Yes, they used tax payers' money, and I'm pretty glad that I helped contribute in some way. Exploring our solar system and beyond with instrumentation is an absolutely needed step to find more space for all the humans on this planet. Not sure if you have noticed, but there are a few things we are running out of here despite China's one child rule.... room, food, fuel, and some other less dramatic things. Mother nature has a way of balancing things, so she'll kill a lot of us off. If we want to continue growing, expanding, space is the only viable option with current trends and technology. Improving the technology we command and the information that we have is the only way to viable improvements. The space exploration programs aim to do this in very calculated ways, very methodical steps to discover new information on a cost per answer type basis.

It is NOT irresponsible. There are a lot of irresponsible uses of tax payers' money, but trying to expand human knowledge and capabilities is not.

Re:That's coolness

[satoshi1]

Linux may not be ready for the desktop, but it is ready for the surface of Mars.
-Me

You know, I am actually cool with that. A stable operating system running a mission critical device thousands of miles away is important, everyone who has contributed to the Linux kernel in some way should be proud that their work is enabling us to learn amazing new things about our red neighbor. I think it says a lot that Linux was chosen over some internallly developed OS.

Re:That's coolness

[zappepcs]

For a Friday night, that is a very nice thought. A project funded by the public, and at least in part, built by the public. Nothing says OSS works better than that. At least I can't think of anything that does... very nice.

Re:That's coolness

[Innova]

The Mars missions have been nothing short of exemplary awesomeness. I couldn't agree more....with one exception

Re:That's coolness

[kperson]

"A stable operating system running a mission critical device thousands of miles away..."

Just where the hell do you think Mars is??

Re:That's coolness

[funwithBSD]

I just hope none of it was LGPL 3.

I think that would mean that RMS owns Mars.

Re:That's coolness

[harlows_monkeys]

TFA appear to be wrong. It runs VxWorks 5.2.

The confusion probably arose because Wind River also sells a Linux version, and the press sometimes confuses that with VxWorks.

Re:That's coolness

[iminplaya]

Not sure if you have noticed, but there are a few things we are running out of here despite China's one child rule.... room, food, fuel, and some other less dramatic things.

We are not running out of them. We are only mismanaging them. There's lots of fresh water falling onto the oceans and empty land we can bring together.

Re:That's coolness

[Tolkien]

Better him than the gov't, odds are they'll claim ownership, despite not having the right to do so.

Re:That's coolness

[roguetrick]

Hey, after it gets larger than 10 basements, we lose the picture.

Re:That's coolness

[homer_s]

Yes, they used tax payers' money, and I'm pretty glad that I helped contribute in some way.
That is the issue, isn't it? You might be glad, but there are people who do not want to contribute and they were forced to do so.

Not sure if you have noticed, but there are a few things we are running out of here despite China's one child rule.... room, food, fuel, and some other less dramatic things. Mother nature has a way of balancing things, so she'll kill a lot of us off.
This might interest you.

No mention of Extreme Programming

[Anonymous Coward]

This article doesn't state the lander was programmed with Extreme Programming techniques.

There's just one sentence which says something about "presents extreme programming challenges" which is the closest this article comes to mentioning Extreme Programming.

Re:No mention of Extreme Programming

[serge587]

Then simply add some suspensefully dangerous music in the background and you have the next generation of thriller movies!

Re:No mention of Extreme Programming

[Ethanol-fueled]

...whilst hand-coding a real-time, multitasking OS in assembler while suspended over a tank of water filled with hungry sharks. Laser beams optional.

Embedded programming is hard!

[compumike]

Sure, there's complicated software systems, but when something has to deal with hardware too, there's just a lot more to think about in every action. That's why engineers make abstractions -- enough to keep their part of a particular project in their head.

But shouldn't the "safe mode" limits be independent of the particular operation as much as possible? In software engineering, the people writing test cases are often not the same as the people developing the code, and for the reason that they want to match both to the spec, not to each other.

--
Hey code monkey... want to learn electronics?

Units?

[pythonist]

I'm wondering what units they are using? American? SI? or Martian?

Call me a pragmatist...

[Dice]

... but when I hear terms like "EXTREME PROGRAMMING!!!1eleven" I just roll my eyes and stop listening.

Re:Call me a pragmatist...

[justinlee37]

I don't think the actual article has to do with the XP methodology, though, the headline writer was just too dumb to avoid referencing XP 'cause he didn't know about it.

Re:Um what

[roman_mir]

It's definitely more extreme than the so called 'Extreme Programming'. Extreme in a sense that you have to get things right or an extreme amount of time and money will go to extreme waste. That's Extreme. 'Extreme Programming' is mostly about covering your ass, not about getting the code right.

Re:Um what

[K. S. Kyosuke]

"And somehow I doubt NASA is doing all that."

And you might be right. That is because this submission was processed with an editorial process called "Extreme Submission Editing", which is a subset of the "agile journalism" methodologies, driven by principles like "test-first publishing" (first publish it, then fix the bugs if there is a stronger "boo!" than usual), "pair editing" (pick any two editors, each one will believe that the other one is going to fix it) and "do the simplest editing possible that could work" (cross yourself, close the eyes and click the "OK, Publish the damned thing!" button). Thanks to the Extreme Submission Editing, Slashdot continues to bring you quality news at an unbeatable price!

Why do they have to do this much coding?

[Anonymous Coward]

According to TFA, each member of a team of 30 engineers must write ~1500 lines of code, per day, in order to control the robotic equipment on board. This coding is done in C. Does this strike anyone else as a brain dead way to control a robot on a day to day basis? Why do they have to write the commands in such a low level language. Can't they abstract this into a more manageable format?

Re:Why do they have to do this much coding?

[Tablizer]

I agree, there's something odd about using C to "program" daily instrument movements. They just need coordinate tables, not C programming. If there are conditionals, they cannot be complex enough to justify C, can they? The rovers didn't have very many conditionals for their sequences based on what I read. If something is not as expected, they usually have the rover stop and wait for Earth confirmation, NOT make the robot select contingency options (unless automated driving is on, but it still has low tolerances before going into wait-mode). I suspect the article is confusing the work of two different teams: the "operating" software developers and the instrument sequencers. Some journalists don't know C from a horse shoe......oh wait.

Re:Why do they have to do this much coding?

[GrayNimic]

From what I understand, Phoenix's on-board memory is sufficiently limited that they cannot keep the entire set of "normal" operations on-board (ie, the library would take up too much space). So for each day's actions, they have to include the relevant subsets of the library in with the command code. If those library excerpts are included in the line count (it would make sense from a "upload size" perspective, if not from a "programming" perspective), that could explain it.

Also, be careful with comparisons with the rovers. While the rover arms and the Phoenix arm are related, they've got some critical differences. One that makes the Phoenix arm movements far more complex is lack of any kind of touch/pressure sensor - the rovers' have them, so you know that you've contacted the rock simply by it being triggered. With Phoenix, they instead have to base "contact" on the change in torque values at the arm joints. That means a lot more calculations, and more daily variation (based on what you expect to be impacting/digging into).

Phoenix also doesn't seem to be very self-aware (ie it can't corelate two kinds of data, such as using the stereoscopic camera to understand what the arm is doing), meaning that it has less capability to do contingency cases - it can't even notice that something has gone wrong. All of that processing, evaluation, and decision making has to be done back on Earth. The rovers are much more self-aware, in terms of sensors about their state (wheel revolutions, etc) and more complex uses of the data it acquires (hazard-avoidance cameras).

Re:Why do they have to do this much coding?

[Saeculorum]

I'd suggest looking at pricing for radiation-hardened parts sometime. A radiation-hardened 2 Megabyte SRAM on a programs I am working on costs around $40,000, and it's not even particularly quick. There are very real code size considerations for space programs, especially when all of this has to run on a battery!

Re:Why do they have to do this much coding?

[dodobh]

That close to the metal, with the limited memory space and CPU power? C is probably the highest level language which works.

!extreme programming

[religious freak]

Aside from having a strong general dislike for anything 'extreme', this is wrong because this is obviously (at least from what I saw) NOT extreme programming, it's programming for harsh environments (no not Windows :).

Crappy title.

I hope they mean...

[Anonymous Coward]

I hope they mean this type of Extreme Programming.

Re:No mention of Extreme Programming?

[Anonymous Coward]

Having a plan that requires 1000 lines of new C code every day is f'ed up beyond all belief. Someone doesn't know what they are writing about.

Wow, yeah. You actually inspired me to read the article. I thought "LOL, yeah right, AC must've made a typo". But I'll be god damned, the article really says that.

I always thought I was a fairly decent coder, but I might have to re-evaluate if other people can do 1000+ lines of debugged, production quality code every day.

Why didn't they automate Dumping The Scoop, etc?

[Anonymous Coward]

Some of the things shouldn't require detailed programming each day,
because they could have been programmed into a firmware library!

The geometry between the arm, scoop, & the 8 ovens doesn't change,
so that could have been automated.
Instead of coding the whole damn thing
& hoping the signal can get there that day,
they could have told it
Invoke "dump scoop into oven # 1 routine"
and saved the interplanetary bandwidth+delays
for something that required customization...

Also, this business of EVERYTHING waits if one link in the chain gives,
suggests that JIT is the wrong paradigm for this case...
at least slightly.

The life of the machine is known to be finite,
the conditions are extreme in temp & superfine dust,
the thing's been jostled a bit in getting it there,
loss of function-time is costly, isn't it?

Say What!?

[Sam36]

Why did they use vxworks instead of linux?

Re:Say What!?

[Xiaran]

There can be lots of reasons. I like Linux as much as the next guy and have been using is since back in the days when I installed it on my 386 from dozens of floppies(and nearly fried my CRT monitor several time configing X :) Those were the days :) ). But I used to work on a project that used QNX. I was often pestered by people saying "Why cant you do that in WIndows" and then "Why cant you do that in Linux". Frankly we didnt because

1. We had years of libraries that were QNX specific
2. There were real time requirements that Plain Linux was not up to(real time linux may be... but you are talking a major porting project)
3. We had literally man-decades of programming experience in our team.
4. We were using a database soultion that was not available on Linux(and is still not)

SO the upshot is... it is often a decision to either have something working now to do the job or wait another few man-years before a (probably buggier) version is done for Linux. Its not anti-Linux. Its just common sense.

Re:Say What!?

[JamesP]

I believe the main issue with using linux is memory protection.

"Real Time OS" is (kind of) "marketing speak" for a OS that does pretty much nothing. Not because it is bad or poor, but because it was designed that way.

And really, up in Mars you don't need hundreds of processes, user interaction, etc, etc

What you need is a basic framework for working with (and that VxWorks does), and you ABSOLUTELY NEED the OS to STEP OUT OF THE WAY when you need it!

Because sometimes the only way to fix a problem is binary patching something on the fly, or rewriting most of the system data, etc. Try doing that in Linux.

Some other interesting points about that article

[Fallen Andy]

1. Only 8 ovens, which can't be used more than once. Hence all the painstaking deliberations

about when to really go for a scoop of soil.

2. Only 3 months before it will get too cold and the lander will (probably) die.

3. Martian day, (roughly 24hrs 40mins).

The NASA programmers have been my heroes ever since the hacks they did to Voyager.

I guess after they've finished the programmers will take up something more relaxing (like working for EA).

Andy

Obligatory PA reference...x2

[adamofgreyskull]

Google > NASA for directions
Mars is roughly 3715 miles from me, as the deepcrow flies.

Re:Some other interesting points about that articl

[Lumpy]

Exactly, I get a kick out of guys that second guess programmers that are so more advanced than they are that they can only HOPE to aspire to be as good as the guys at NASA.

That and From my experience, less than 10% of all programmers have ever done anything embedded or robotics wise. The rules change when you are writing software for engine management, robots, or space probes, than when you are writing a new consumer toy app or spreadsheet program.

Bummer Alert! Incoming Facts Detected!

[Anonymous Coward]

A couple of details to clarify:

1. The "code" the folks at the Pheonix SOC are using is called "VML", which is a script language developed by a fellow named Chris Grasso and is similar in some reards to the earlier command language "CDL" from the 70's. They are not doing any programming in C or anything like it, and VML isn't all that sophisticated (it has no array type, for instance). It's designed to do one thing well, and that's control a spacecraft by issuing command messages at specific times.

2. The software on the spacecraft that actually runs everything was written entirely in ANSI C, and it all runs on the VxWorks RTOS on a radiation-hardened PowerPC CPU in a VME card cage. Linux isn't anywhere near the actual spacecraft (so all you Linux fanboys can calm down now). Linux sits on some folk's desks, and it was used by some of the instrument teams in the development phases for file servers, metrics tools and so on. The actual Windriver compilers for the target CPU run under Solaris. Not Linux. Sorry.

3. The challenge facing the operations folks is centered around getting the various instruments to sing and dance together at the correct times. Most of the team members are people with PhD's in planetary science, atmospheric science, geology, optical science, math and so on.

How do I know all this? Well, let's just say that a little part of me is now sitting on Mars.