Stress-Testing Software For Deep Space

kenekaplan writes "NASA has used VxWorks for several deep space missions, including Sojourner, Spirit, Opportunity and the Mars Reconnaissance Orbiter. When the space agency's Jet Propulsion Laboratory (JPL) needs to run stress tests or simulations for upgrades and fixes to the OS, Wind River's Mike Deliman gets the call. In a recent interview, Deliman, a senior member of the technical staff at Wind River, which is owned by Intel, gave a peek at the legacy technology under Curiosity's hood and recalled the emergency call he got when an earlier Mars mission hit a software snag after liftoff."

Comments

Seems like a rationalization

[gadzook33]

While I buy that the landing systems need an RTOS, I doubt Curiosity does. Image processing that happens with "precision"? Do x86 processors not process images precisely enough? I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this. The rest of this seems like a rationalization for using old hardware. However, as an engineer for the government it's possible I'm just old and embittered.

Re:Seems like a rationalization

[Anonymous Coward]

Remember, too, that Curiosity has been in the works for almost a decade. They had to commit to a spec for the computers a long time ago, so it's no wonder by today's standards things seem out of date.

Re:Seems like a rationalization

[Sasayaki]

My understanding is that the thinking goes like this.

Sure, there are newer processors that claim to fit the bill. But space hasn't changed so much since the Apollo days that we need all new processors; by and large anything that needs "heavy lifting" CPU wise can be transmitted back to Earth. For unmanned probes, there's very little demand for high speed CPU tasks that can't be offloaded to Earth. And even if there was, when your latency back to your operator is about 14 minutes (with an extra 14 to receive further instructions, plus the time it takes to interpret the previous data set, determine new instructions, then program those instructions), that's a lot of down time to work on various tasks.

The Mars rover CPUs, I imagine, spend the vast majority of their time idling.

However... the old stuff works. It has its faults and flaws, sure, but they're extremely well known and documented. You can work around them. You have the old grognards that have been kicking around since Apollo who know every damn thing about them. They're risky, sure, but it's a managed, controlled, limited and understood risk. But new processors are *new*. You lose that element of certainty, and the CPU is the heart of a probe. You lose it, you're fucked.

You're trusting the mission, a mission that costs billions of bucks, to a new, untested device that hasn't been field tested, hasn't got that certainty, and *you just don't need*.

Re:Seems like a rationalization

[Anonymous Coward]

that's why land-based projects like SKA for example which also take decades to complete are designed taking moore's law into account, leading to a very funny situation in which the project starts, they start building stuff but the computers that will run the thing are still 10 years away... (and I guess everybody just hopes computers will keep up or else...)

Also you must take into account that the actual instruments are being built fairly early i.e. 5 or more years before launch since there is a LOT of testing calibration more testing etc. Additionally, when the stake is a billion dollar project like these you tend to leave fancy new things and favor old proven and well documented tech. Just in case...
If not you just mount two instruments if you have space and money a fancy new one and the old usual thing (such is the case for Solar Orbiter for example)

Re:Seems like a rationalization

[Grave]

That's just it - this sort of computing task cannot get by with 5 9's or 7 9's or a hundred 9's of reliability. It needs to be 100% reliable, which means that every potential hiccup, flaw, or design quirk is understood and documented to the nth degree, and thus can be worked around. It also means you can reliably simulate the hardware and throw all sorts of stress testing at it.

Re:Seems like a rationalization

[Anonymous Coward]

Reminds me of about 10 years ago when I was working at Motorola on cell phone base stations. We switched from VxWorks to Linux and got... nothing. No performance gains, no reliability gains. Just a free OS instead of something we had to spend money licensing.

Of course, all the extra time spent switching and testing certainly cost a lot of money in man hours.

Re:Seems like a rationalization

[gadzook33]

Oh good, someone more embittered than me. I especially like how rather than provide any sort of argument as to why an RTOS is required (because...a pedestrian might walk in front of the rover?) you'd rather insult me.

Not for nothing but I'm about a step away from being a hippie and I've served the government faithfully for many years. I work with some of the best and brightest and if you weren't able to cut it there, the fact that you're incredibly negative and seem like a jerk would likely only be a few of the reasons why.

Re:Seems like a rationalization

[Anonymous Coward]

And that's where you (and most people) are mistaken.

A RTOS is not an OS that acts "quickly", it's an OS which provide a 100% guarantee that a task will be executed in a definite time-frame, whether this needs to be 1 micro-second or 1 hour ; and which provide guarantees if the task can not be completed in this time-frame. A job neither Windows nor any flavor of Linux can achieve.

Re:Seems like a rationalization

[TubeSteak]

First of all, there's a typo in TFA.
They state the chip is a "RAD760" but they link to the RAD750 wikipedia page.

Do x86 processors not process images precisely enough? I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this.

The problem with x86 technology is that it has gotten too advanced.
The chips have become so dense that radiation hardening is much much more difficult than it used to be.
Increased difficulty = increased expense

Further, I don't think you appreciate the specs of that old PowerPC chip.
It's tolerance to 1 megarad of radiation exposure is a lot.
You literally get what you pay for with this cup, ranging from 200 rads to 1 megarad.
Even 500 rads is more than most space applications require.

So in order to save money, some companies use cheaper hardware in a triple redundant configuration, in order to avoid paying out big bucks for radiation hardened boards + chips. But for a mission to mars, where reliability and power usage are critical, two old 133mhz processors are better than any of the other choices.

The rover has just enough processing power to talk to NASA, look around, and do one other thing. And that's just fine.
They've partly split up the workload between two processors, but if one processor failed, NASA could juggle everything with one hand.

Re:Seems like a rationalization

[khallow]

It's worth noting that in the overall mission they generally get by with one or two 9's of reliability. There's no 100% reliability out there and nobody would be able to afford it, if there was.

Re:Seems like a rationalization

[Meditato]

Look, that guy ("Required Snark") might have been an asshole, but you didn't really acquit yourself well either in your original post. I cofounded and work for a real-time telemetry contractor. We use Android, but the Linux kernel isn't built to handle read-time applications reliably. There are too many things to handle in terms of time-safe task-switching, execution, multi-processing, and internal consistency in order for it to be a good RTOS. So keeping that in mind, I had to implement a real time environment in userspace that uses root and some native code in order to collect data, send data, and operate hardware in a safe, timely manner. But this isn't the best solution because I still have to deal with the fact that it's all just a frustrating abstraction sitting on top of a kernel that isn't at all concerned with what I'm actually trying to do, despite my best efforts to single-handedly make the necessary changes.

Your "newer processors" bit is also completely off the mark. Radiation-hardened processors lag generations behind owing to the need for extensive redesign and testing. Complicating this picture is the fact that even then, they still have varying levels of reliability and power efficiency. You don't want a processor that has a microcode architecture that makes your targeted code difficult to semantically evaluate and verify. You don't want (or need) a recent processor that hasn't had extensive real-world user testing. You want a processor in the goldilocks zone, one that you've worked with before and has a community behind it.

Keeping that all in mind, they chose a good processor, and already had an OS largely built for it based on previous missions with earlier versions of the same processor.

Re:Seems like a rationalization

[Electricity Likes Me]

The classic example is the Pentium math error: imagine if you were 2 years into the mission and then discovered that the new high speed chip you put in gives incorrect floating point calculations.

Re:Seems like a rationalization

[lordholm]

Newer missions collect too much data to transmit everything back to earth. They typically need to do local processing of for example images and other data. There is also AI aspects, for the ExoMars rover (made by Europe), the onboard computer will have a virtual scientist embedded. This virtual scientist look at the camera pictures and decide if something is worth an extra look, and may order the rover to carry out opportunistic science. I am not sure as to whether this is the case with Curiosity, by I could easily imagine this is the case. In fact, newer missions have substantial need for computational power. But, there is no software reason to do these computational tasks on the main computer, the task may as well be sent to a soft realtime helper computer, that may as well run Linux or something else. A lost image is typically not the end of the world.

In many cases the spacecraft and rovers are also not hard realtime, but they are also not soft realtime either (i.e. we compute thruster response for t=0, only to have the thrusters fired at t+0.1 or something in that range, whether they fire within this time does not really matter except during docking, landing and separation), I was trying to push through the notion of firm realtime when I was working in the space sector, but the main problem with this notion is that we do not yet know what effects it has in terms of sw design. Any way...

The primary reasons for running 10 year old CPUs is that, 1) specs are chosen early in the project, this is important as the CPU specs are guiding the development of the SW requirements and the actual implementation of the SW and 2) as you say, the older CPU will be battle tested before they are sent into deep space.

Re:Seems like a rationalization

[Animats]

I get the idea of being hardened to radiation but it was my understanding we have newer processors that fit the bill on this.

Radiation-hardened processors are hard to get. For one thing, they're export-controlled, so if you make them in the US, you can't sell many. Atmel makes a rad-hard SPARC CPU, and they've sold 3000 of them. Nobody seems to have built a modern x86 design or even an ARM in a rad-hard technology.

There's a basic conflict between small gate size and radiation hardness. The smaller the transistors, the more likely a stray particle can damage or switch them. So the latest small geometries aren't as suitable. Also, the more radiation-hard processes, like Silicon on Sapphire, aren't used much for high-volume products.

As a result, rad-hard parts are an expensive niche product. It's not inherently expensive to make them, but the volume is so small that the cost per part is high.

Re:Seems like a rationalization

[pedestrian crossing]

Pedestrian crossing?

You rang?

Re:Seems like a rationalization

[chihowa]

Shielding is heavy and expensive to launch (and to land softly). Then, for every extra mm of lead shielding you add, there's a more energetic photon just waiting to flip a bit. It ends being up cheaper to make radiation hardened electronics than to accommodate for the extra shielding.

"earlier Mars mission" == MER-A Spirit

[Anonymous Coward]

''recalled the emergency call he got when an earlier Mars mission hit a software snag after liftoff."
From TFA:

Back when Spirit Rover landed on Mars in 2004, it experienced file systems problems. I got a call on landing day while I was in Southern California. I fired up my laptop and worked with three groups who were dealing with a variety of time zones: California, Japan and Mars. Since I had a RAD 6000 systems on my desk running simulations, by the end of first week we figured it out and were able to fix it.

VxWorks has a nice track record in space

[Anonymous Coward]

At least one instrument running VxWorks has been flying on the ISS since 2001. I'd be surprised if it were the only one.

Re:I'm glad I don't program for NASA

[jimmydevice]

So, You work for Microsoft?

Re:"earlier Mars mission" == MER-A Spirit

[AaronW]

With my long experience with VxWorks this doesn't surprise me. VxWorks is not the most robust RTOS. Think of it as a multi-tasking MS-DOS. The version they used has no memory protection between processes and I have found numerous areas of VxWorks to be badly implemented or downright buggy. Up through version 5.3 the malloc() implementation was absolutely horrid and suffered from severe fragmentation and performance problems. On the platform I was working with I replaced the VxWorks implementation with Doug Lea's implementation (which glibc was based off of) and our startup time dropped from an hour to 3 minutes. I was also able to easily add instrumentation so we could quickly find memory leaks or heap corruption in the field, something not possible with Wind River's implementation. After reading about the problems with the filesystem I looked at the Wind River filesystem code. It was rather ugly. They map FAT on top of flash memory (not the best choice) and the corner cases were not well handled (like a full filesystem).

Similarly, their TCP/IP stack sucked as well. If you can drop to the T-shell through a security exploit you totally own the box (i.e. Huawei's poor security record).

VxWorks is fine for simple applications, but for very complex applications it sucks. At least the 5.x series do not clean up after a task if it crashes because it does not keep track of what resources are used by a task. A task is basically just a thread of execution. All memory is a shared global pool. At the time it did have one feature that was useful that was lacking in Linux, priority inheritance mutexes. These are a requirement for proper real-time performance and I believe are now included in Linux.

pshaw, we use RTEMS

[Anonymous Coward]

the other big player in space RTOS: RTEMS.
Free, open source, rtems.org.

Has all the same problems as VxWorks.. no process memory isolation (because space flight hardware doesn't have the hardware to support it usually)....

One thing that VxWorks has that RTEMS doesn't, and I wish it did, was dynamic loading and linking of applications. You're basically back in 1960s monolithic image days, not even with overlay loaders.

Re:pshaw, we use RTEMS

[Anonymous Coward]

FORTH is great. From about 2 dozen core instructions an entire operating environment can be built. Unfortunately, FORTH takes in-depth knowledge of not only the hardware, but also a firm grasp of scope of the tasks that need to be performed. Most programmers today cannot handle FORTH -- imagine building your own TCPIP stack, filesystem, and RTOS operating environment from scratch. That talent is found only in a dying breed of programmers, literally.

For a robust 100% reliable radiation-hardened space environment, even the processor, data paths, and memory need to be self-correcting for each data bit. Sapphire-On-Silicon processors are only the beginning of solving the reliability issues. Commercial Off The Shelf solutions are an invitation to disaster, but nobody wants to invest the time and money for proper solutions any more.

You can thank Just In Time supply chains, quarterly corporate focus on maximizing profits, and Globalization for the current sad state of space exploration. Without a paradigm shift in attitude, there will be no more Voyagers. I know. I used to work for rocket scientists.

Ever seen a time table for a space mission?

[dutchwhizzman]

They start planning this years, years and years ahead. It is not uncustomary to have decided on a hardware platform five years before launch. Since there's a lot at stake for these bigger missions to succeed, they usually don't take risks and put stuff up there that hasn't proven itself. Maybe some evolution like a higher clock rate or more memory or something like that, but a new processor architecture gets tried on other things that have redundancy, lower cost or less exposure and preferably a combination of those.

I have been discussing some technology that was possibly put in an instrument on a weather/climate sat with the primary investigator of the then current mission and named to be the one of the next mission as well. This was around 2007. They had to choose the technology then, so they could work on plans and get funding around now. Once they get their funding, it will still be three to five years before it goes up there. Back then, due to the reliability demands they had for the sensor and the relative unproven state of using CMOS sensors for photon capture (common used in digital consumer cameras in 2007) they chose to go with the previous solution, that was in the current instrument. That means that they will probably launch a pre-CMOS sensor equipped instrument around 2015, because that was the best option available to them when it was decision time.

Unless we change the way we "go to space" in a radical way, I don't see the latest and greatest tech make it in missions like this. It's up there, sure it is, but only a handful people know it is and they don't want their precious black ops budget exposed or taken away from them. Once the statistics they get from the successes and failures (failing in secret "testing missions" once in a while is allowed) to a rating that makes it commercially viable to sell the tech to civilian usage, plus the state of technology used for espionage and military use is such that there isn't any tactical threat to do so, more modern tech will be used for missions like this.

Re:"earlier Mars mission" == MER-A Spirit

[Jeremi]

Up through version 5.3 the malloc() implementation was absolutely horrid and suffered from severe fragmentation and performance problems.

I talked to one of Curiosity's software engineers the day it landed... he mentioned that one of their coding rules was: no malloc() allowed.

Re:"earlier Mars mission" == MER-A Spirit

[kauaidiver]

No malloc()? Interesting, I worked on a project at NG and we had same policy. Everything was on the stack or global. We had the chance to run with Monta Vista embedded Linux but someone higher up decided to go with "tried and true" VxWorks. I agree with a poster above about re-training costs and all that adding up.. but if embedded linux became standard with big companies I don't think it would take too long to make-up the costs of re-training and all the other stuff that goes with it.

It demonstrate how inefficient desktop software is

[Viol8]

An old Power PC can fly a spaceship to mars, execute a difficult landing and now semi autonomously drive a robot across the surface of a planet 30 million miles away , yet its not up to the job of writing documents using the latest word processors. Whats wrong with this picture?

Re:"earlier Mars mission" == MER-A Spirit

[Anonymous Coward]

Malloc is non-deterministic. The request for a pointer to return contiguous free bytes will need to search a fragmented memory map to complete the request. The duration of the search depends upon the algorithms and the amount of fragmentation relative to the size of the request. It is worse if it must rearrange memory to accomodate the request. Thus, use of malloc() is typically avoided for time-critical code in a real-time operating system.