What's Inside the Mars Rovers

Captain Zion writes "Space.com has a story about the hardware and software of Mars Rovers Spirit and Opportunity. Basically, they're radiation-shielded, 20MHz PowerPC machines wirh 128Mb RAM and 256Mb of flash memory, running VxWorks. I wonder if I could make a nice firewall with one of these for my home network..."

Comments

Radiation hardness

[swordboy]

Does anyone know what the deal was with the flash memory that caused the outage? I heard something about a "solar event" that caused a problem with the flash memory that led to the outage. It was subsequently resolved by disabling the flash. If so, BAE Aerospace has a possible solution with their upcoming line of rad-hard memory.

Re:Radiation hardness

[TwistedGreen]

It appears to be a software error, and not hardware-related. It actually looks like it ran out of swap space and the OS crashed. This article explains what they think happened, and this article has more information on their recovery plans.

A quote:

It is now believed that the rover's flash memory had become so full of files that the craft couldn't manage all of the information stored aboard. Spirit bogged down because it didn't have enough random access memory, or RAM, to handle the current amount of files in the flash -- including data recorded during its cruise from Earth to Mars and the 18 days of operations on the red planet's surface.

Raises some interesting questions about software reliability, I think. Did nobody think about running out of disk space?

Re:Radiation hardness

[AaronW]

If anyone saw my earlier posts on VxWorks they would see I am not at all surprised about the problems NASA is having.

As someone with first-hand experience with VxWorks let me say that VxWorks' memory handling code sucks. Their malloc implementation has got to be the worst one ever designed. It fragments horribly and when fragmented has unusable performance. A malloc call can take many milliseconds when memory gets fragmented. Our box used to crash due to the fragmentation. I replaced Wind River's code with DLMalloc and that fixed the memory issue. We went from many tens of thousands of fragments to only a few dozen. Our reliability significantly increased as well after dumping Wind Rivers brain-dead malloc code. BTW, glibc uses a variant of Doug Lea's malloc code, so it's been widely tested.

Furthermore, in VxWorks there is no way to identify what process malloced a block of memory. There is no memory protection either (think DOS). If a task has a memory leak, VxWorks does not provide any method of tracking down the culprit. I had to add that support to the DL Malloc code I ported so we could find memory leaks and general memory usage by task.

Since malloc is critical to the working of VxWorks, if you run out of memory you are basically completely dead. Often the only way to recover is to wait for the hardware watchdog timer to kick in and reboot the system.

If a task dies in VxWorks with a bad pointer there is no way to recover other than reboot. The OS will not clean up after a task (i.e. free memory, close files, release semaphores).

As far as flash goes, VxWorks supports the FAT16 file system. As you know, FAT16 sucks. It only supports a limited number of files in the root directory. It's relatively easy to corrupt, and when corrupt it tends to corrupt itself even worse. There is no wear leveling support. If the FAT table or root directory gets corrupted, you're screwed. VxWorks is even worse since there arn't tools to fix a corrupt file system.

VxWorks is not a scalable OS. The OS gets slower as the number of tasks increases. Realtime support sucks. Although it has support for things like priority inheritance to prevent priority inversion, the best guaranteed realtime latency is half the system tick rate (the tick rate is usually 10ms).

Also remember that unlike open source operating systems, the source code to VxWorks is not available unless you pay some major $$$. Without the source you're basically working blind.

VxWorks is an old RTOS, and its age is definitely showing. It is not a robust OS.

As far as turning VxWorks into a firewall, you'll need to write all your own code. The VxWorks TCP/IP stack is an archaic vulnerable version of the BSD stack. TCP sequence number guessing is trivial. There is no built-in support for firewall support, NAT, or anything else. I have heard many many complaints about the VxWorks networking code. Although the box I'm working on is a router and broadband remote access server, we don't use the VxWorks TCP/Ip stack much. I am sure the VxWorks stack is vulnerable to many of the current DOS attacks as well.

Some of you might say why not use VxWorks AE, which adds memory protection. Think hacked-in memory protection far worse than Windows 95. It's slow, very buggy, and poorly supported. We tried AE and after many months we dumped it and went back to the non-AE version. Very few companies actually went with AE.

Today there are many other choices for an RTOS that are better than VxWorks. For our next major project we're looking at both QNX and TimeSys Linux. QNX is a true microkernel design with the core being around 70K. Every driver is protected and can be restarted if it crashes. I think you can even buy a medical grade version of QNX. TimeSys Linux is also pretty cool, with excellent real-time support and all the advantages of Linux. For something like the Mars rover, QNX would be better due to the limited amount of memory and greater robustness.

Wi

Radiation Shielding

[kyknos.org]

How is it done? Some external armor, or even insides of the chip are different?

---

Re:Self-warming

[Cyclopedian]

To survive the frigid Martian night, MER computers are housed in warm electronics boxed heated by a combination of electric heaters, eight radioisotope heater units as well as the natural warmth from the electronics themselves.[Emphasis added by me]

If obsessed environmentalists don't like NASA sending up probes with any radioactive material ('it might blow up, ohh..'), then how did this little tidbit get by them? Do they consider it non-radioactive? If they're only concerned by radioactive propulsion systems, then I think they're a bunch of hypocrites. Radioactivitiy is radioactivity whether it's propulsion or heating.

If they don't mind it, then let's send up those dune buggies with RTG and 18-inch wheels and cover a lot more of Mars.

-Cyc

Re:I hope the flash memory was not commodity hardw

[Rootbear]

There is very little on the Rovers that is "commodity" in any sense. The CCD image sensors, the computers, everything, is all custom made. Everything has to be made to withstand the rigors of flight and the harsh environments of space and Mars. The CPU does not have a backup, which is a bit unusual for NASA (I'm a contractor at NASA/Goddard, but not involved in any flight missions). However, the particular computer used on the rovers (the RAD6000) has a very good record. There are something like 150 in use on various spacecraft and they've all worked very well.

And the flash memory has probably not failed. It seems to have been a software problem, not hardware.

Rootbear

Save HST! Sell Sojourner knock-offs

[bhima]

What NASA should do is to hire a Taiwanese firm to build inexpensive knock-offs of Sojourner. They already have the design, I'm sure a few bright minds could cut the chassis price down significantly; after all we don't need all the exotic materials. I'm sure IBM still makes a PPC variant that would make a new cheap board layout easy. As far as the OS: of course we don't need VxWorks (Nor could the project afford it) we have NetBSD!

The profits from Slashdot alone could extend the life of HST or launch the James Web Space Telescope early.

I thought about the current rovers, but I think they are a bit large to be successful!

space shuttle uses 1969-vintage ibm 360 computers

[peter303]

The space shuttles run on five AP-101 computers, originally designed in 1969. The started with 32 kilowords of magnetic core memory for radiation protection, since upgraded to semiconductor memory. These computers were chosen due to their success in the Apollo, Skylab, and B52. For science and personal work the astronaut specialists usually bring personal laptops which are thousnds of times more performant.

FPGA's

[retostamm]

There are also Xilinx FPGA's in the Rover. Cool thing because they can be reconfigured if you find a bug while the thing is in transit.

Xilinx radiation-tolerant Virtex(TM) FPGAs are being used in the "main brain" of the rover vehicle, controlling the motors for the wheels, steering, arms, cameras and various instrumentation, enabling the vehicle to travel about the planet.

They also controlled the Pyrotechnical stuff during landing.

[Disclaimer: I work for this great company.]

Re:Processor is *not* a PowerPC

[addaon]

The Rad750, btw, is a deeply cool chip. Once it's mature enough to start using for scientific-level stuff, it will be a real revolution in what we can do. One of the limitations with Hubble was that it had so little processing, a full data dump needed to be done for even checking orientation; there was no ability to offload processing to the sat. If the 750, or something similar (not that I know of anything too similar) is up there for our next big telescope, it will make a real difference in the efficiency of how it is used.