Slashdot Mirror
Mars Rover Spirit Back Online
Skyshadow writes "Just in time for the arrival of its twin, the Spirit Mars Rover is back in working order. Programmers at the JPL have traced the problem to the rover's flash RAM, which it uses to maintain its filesystems. They are using a ramdisk in the rover's RAM to bypass the bad flash memory, and are working on a workaround for the bad flash. Good news, but the rover is still potentially weeks away from full operational status."
Here is the link to the real story. The one given in the /. acticle is getting pushed down spaceflight's page.
If I was sending an embedded control computer to another planet, I would have chosen an OS with memory protection, not VxWorks.
Actually, they might have protected memory if they use VxWorks AE RTOS/Tornado Tools 3.0. Spirit uses VxWorks, but I don't know what version they used or when they had to commit to a particular version of VxWorks.
Also, as the article mentions, memory protection adds overhead and can affect real-time performance. Hard real-time software cannot afford to have a complex layered structure and lots of conditional code that adds unpredictable delays. For that reason, many really real-time applications run very close to the hardware (for better or for worse.)
Two wrongs don't make a right, but three lefts do.
Here's a rant by a JPL guy about appropriate technologies for software on deep space probes. He recounts one story of a failed probe "100 million dollars, and 100 million miles away".
They fixed it. The fact there was a lisp REPL running on the spacecraft helped.
That's cool:
(unwind-protect
(progn (do-science)(talk-to-earth))
(wait-in-repl-for-earth))
I'm watching NASA tv at the moment and they're explaining possibilities now. At the moment, they only have a very broad explanation of what's going wrong. However the newest knowledge is;
There are two separate flash memories on Spirit. At the moment, part of the problem is software which can read part of the flash memories as some of the operational software which is kept in flash ram seems to be coming up before the system reboots.
The system is rebooting no matter which flash memory is being accessed, it has the same bug both ways, so the flash ram itself looks to be OK, but the interface between the flash ram and the software looks to be causing resets.
Even if there were more backup flashrams, it looks like they'd still have this problem. Perhaps many, all on different controllers, and even an entire backup computer would have prevented this. at 100watts total power available for the rover, an entire extra computer may be a bit much to have fit. But then sending two rovers would also negate problems, and thats just what they've done
It seems most likely at the moment, according to NASA, that the family of components that are involved with the hardware addressing of the flash memories looks to be where the problem is.
There is a big difference between standard flash and radiation hardened flash. In fact we are designing a project with one of these VME buss units as a storage array.
If you don't get it on cable, you can watch NASA TV here.
Litigious bastards
A lot of the comoponents in this craft came from my former employer, www.InterPoint.com, who laid off half their staff a few years ago (me was one of those). Little boxes the size of a pack of cards, hand built. Really amazing stuff.
"Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck