What's Inside the Mars Rovers

Captain Zion writes "Space.com has a story about the hardware and software of Mars Rovers Spirit and Opportunity. Basically, they're radiation-shielded, 20MHz PowerPC machines wirh 128Mb RAM and 256Mb of flash memory, running VxWorks. I wonder if I could make a nice firewall with one of these for my home network..."

Comments

Re:What's the bus speed on that thing?

[CaptainAlbert]

> Does a 20mhz processor really need 128mb of ram?

A processor of any speed doesn't need RAM of any size.

The application you want to run needs both processing power and memory. How much of each? Depends on the application.

Re:Radiation Shielding

[the real darkskye]

The CPU is fabricated to withstand the radiation, a brief summary can be found here or by googling

Flying VxWorks to Mars

[Hiroto. S]

I googled across following presentation with a little more details.

Flying VxWorks to Mars

Re:Self-warming

[JDevers]

If I'm not mistaken, virtually all probes have some sort of radioisotope heater...

Radioactivity is NOT radioactivity when you are considering things like this. Saying the people who don't want nuclear powered rockets should hate this as well or else they are hypocrites is tantamount to saying that the people who don't like oil spills should bitch about how some motor oil ALWAYS stays in the plastic container it is shipped in. Not quite the same problem. Afterall, these things aren't much more radioactive than a Coleman lantern wick or a smoke detector element...

Re:Redundency Check?

[vofka]

If I recall correctly, the Shuttle has 5 GPC's (General Purpose Computers), three of which are "online" at any one time.

The online GPC's each carry out the same set of calculations (potentially each uses code designed to do the same thing, but written by different programmers), and they compare each others results. If any single GPC is considered to be too far wrong, the offline GPC's submit their answers. The three GPC's that are in closest agreement then become the new online GPC's, and the remaining two go offline. The GPC's can reboot themselves if they are too far out of whack, if they fail in one of the "results elections", and of course when they are told to do so by the crew.

Also, whenever a GPC is sent offline by one of the others, a specific caution indicator (and potentially the master caution indicator and klaxon) is activated, and the relevant error codes are shown on one of the forward CRT's. The error codes, along with other information such as the currently running program and the current mission phase, determine the crew's actions. Actions can be as simple as disabling the master caution klaxon for the current alert, all the way to hand-checking certain results and manual GPC restarts.

This is all from memory (from about 5 years back), so some of this may have changed recently, particularly on Atlantis with the "glass cockpit" upgrade that happened 18 months or so ago, but the general gist should be about right (and I'm sure I'll soon know if it isn't!!)

Re:Wait a second...

[Rootbear]

Actually they are running at 20MHz. I've seen several write ups which clearly state that. The RAD6000 can apparently run at up to 33MHz, with a claimed 35MIPS. The rovers are "underclocked", probably due to power budget concerns.

Go to
http://www.iews.na.baesystems.com/space/rad600 0/ra d6000.html
and click on the rover picture to get a PDF brochure, which gives the 33MHz/35MIPS figure.

Rootbear

Processor is *not* a PowerPC

[GileadGreene]

radiation-shielded, 20MHz PowerPC machines

No, they're not.

The processors in MER are RAD6000's, which are radiation-hardened versions of the RS/6000, the predecessor to the PowerPC (see this for details). The RAD6000's younger brother, the RAD750, is indeed a rad-hardened PowerPC.

As an aside, there is a big difference between a radiation-shielded processor and a radiation-hardened processor. Shielding implies just sticking some kind of rad-absorbent material between the processor and the environment. A rad-hardened processor is actually manufactured in a different way - different gate layout, different design rules, often different materials (Silicon-on-Insulator is popular). These things are done to minimize or prevent the effects of single-event upsets (when a bit is flipped by high-energy particles) and single-event latchups (which basically turn a couple of gates into a glorified short-to-ground). The materials changes may also improve the overall total dose tolerance of the processor. The work required for redesign is one of the reasons that space-qualified rad-hard processors lag the commercial market. The NASA Office of Logic Design has some good papers on space processors available online if you're interested in learning more.

Re:Self-warming

[The Fun Guy]

Radioisotope thermoelectric power units need to be hot enough to allow for electricity to be generated by thermocouples placed between the unit and the heat sink (space). A quick Google search gives 200-500 watts of power generated from multiple interleaved stacks of plutonium-238 or strontium-90, average radioactive source strength of around 50,000 curies, depending on design.

Radioisotope heaters use much less material, as they only need enough heat to keep the warm electronics box above -40F or so. From the Environmental Impact Statement in the Federal Register ([wais.access.gpo.gov][DOCID:fr10de02-54]):

"Each rover would employ two [calibration] instruments that use small quantities of cobalt-57 (not exceeding 350 millicuries) and curium-244 (not exceeding 50 millicuries) as instrument sources. Each rover would have up to 11 RHUs that use plutonium dioxide to provide heat to the electronics and batteries on board the rover. The radioisotope inventory of 11 RHUs would total approximately 365 curies of plutonium."

Nothing you'd like to swallow, but still, much smaller than a radioisotope power unit.

They used to use sapphire...

[emil]

...as the substrate of the chip, rather than a silicon wafer, so the chip was a "sapphire" chip rather than a silicon chip (although doped silicon could then be used to form transistors, as could Gallium Arsenide or Germanium, through the regular lithographic process).

This is the classic "Silicon On Insulator." IBM has a process of embedding a layer of glass beneath the surface of a standard silicon wafer, allowing SOI using silicon substrates. This and their work with copper set them apart from the other large silicon transisitor foundries (TSMC, Intel, etc.).

The processors on the rovers are probably SOI, but I don't know which process is used.

RAD6K

[Anonymous Coward]

I am an engineer that works with the RAD6K processor boards. A couple of observations here.

1. The RAD6K really does run at 20 Mhz. They're creakingly slow. They're spec'd to run up to 33 Mhz, but the customer can get them to clock at lower speeds (I've seen them run at 12.5 Mhz). The only drawback is the PCI bus is also clocked as the same speed as the CPU. This is a mixed bag - but a slower PCI bus helps improves signal integrity and decreases power consumption.
2. The board is PCI, but NOT compact PCI. There is a proprietary PCI connector and a proprietary PCI backplane. You cannot plug commercial PCI products unless you have an adapter to interface to the proprietary PCI connectors.
3. For those who are not aware, there are three types of memory being used on the rovers. There is the SRAM (the RAD6K boards use SRAMs, not DRAMs), the EEPROM, and apparently, FLASH RAM. The EEPROM and the SRAM are on the processor board itself - there is probably more EEPROM memory in the system on another board. The EEPROM usually holds the flight code, and there are usually two copies. An original version that was launched with the spacecraft, and one patched version made possible via uplinks.
   
4. I am amazed at the presence of FLASH RAM's. I am not aware of any rad-hardened FLASH RAM devices for spaceflight use. In addition to radiation hardness, the device must be made reliable with an approved QML-Q or V manufacturing flow. Radiation hardness is just icing on the cake, but the key is that the device must be reliable to withstand temperature extremes, shock and vibration. So, I have yet to see a FLASH RAM device that can be used. I am aware of the Chalcogenide based RAM's which are essientially uses the same substrates on CD-ROMs as memory cells. These products are hard to come by right now and are a high risk because we don't have sufficient data and flight heritage. A catch-22 in flight design is if it hasn't flown before, we don't want to fly it. But at some point, someone has to fly the first generation (someone who is willing to take a huge risk). Anyway, the FLASH RAM's on the rovers are in all likelihood upscreened commercial products. In other words, a mass buy of an entire lot of dies of commercial FLASH RAM's may have been bought, packaged in-house or through a vendor, and then screened for reliablity at extended specifications. This is not the same as the manufacturer who can guarantee the specs by design by designing it from the outset with increased reliability in mind.
5. Radiation shielding? Minimal at best! The RAD6K shields its SRAMs by placing it on the underside of the processor board and orienting it such that the particles hit the processor side of the board instead of the RAM side of the board. There is some degree of radiation shielding for particles of sufficiently low energy. The truly high speed particles are going to get through and the only thing that will truly stop them is shielding whose thickness is measured in feet. That amount of shielding is too heavy for launch. The best we can do is mitigate the effects of radiation by guaranteeing devices can withstand a certain amount of radation dosage (measured in kRads) and design for latchup protection (latchup is a parasitic condition in which an ionizing particle impacts a transistor structure in a way that causes a SCR to be formed and a runaway current condition is initated leading to the device being burned out by high currents). Radiation effects in the form of SEE's (single event effects) such as bit flips can be mitigated by redundancy and voting circuits, memory scrubbing, and error checking using checksums/CRC's.

Re:RAD6K

[demachina]

"A catch-22 in flight design is if it hasn't flown before, we don't want to fly it. But at some point, someone has to fly the first generation (someone who is willing to take a huge risk)."

Or you fly it as a non mission critical experimental payload which is what we did back in the day I worked on avionics. You fly it as an experimental package so it gets the stress but if it breaks either its not in the mission critical loop, or if it is in the loop you can switch back to proven hardware. I kind of assumed this would be a standard part of qualifying electronics for space flight as well though its obviously a lot more expensive. Its not feasible to test it on Mars due the expense but you could test it in geostationary orbit where it will get lots of radiation and temperature extremes, as well as launch vibration and G's.

The CPU is POWER, not PPC

[rshadowen]

The RAD6000 is based on a POWER CPU called RSC (RIOS single chip) - it's not a PPC chip. This was a design that consolidated the 5 - 6 chip RIOS processor complex onto a single lower performance die for low end workstations. I worked on the development team at IBM.

The RSC design played a key role in bringing Apple and Motorola together with IBM to create the PowerPC line of CPUs. The 601 was the first PPC and was basically a redesign of RSC. It supported both POWER and PPC architectures, although there were deviances from PPC since the architecture was actually being defined at the time we were working on the chip.

The RAD6000 version of the design happened because IBM wanted to pursue some government contracts, so had the RSC specially qualified. Another group then took the design and performed the radiation hardening.

After Pathfinder we had some cool IBM/Mars posters hanging around the building, but oddly enough they vanished very quickly...

Re:Radiation hardness

[AaronW]

While I would agree with you that avoiding malloc and preallocating memory is the way to go, but it is not always possible. In my case, we are using various 3rd party libraries, and changing them to use static memory allocation would be prohibitave. In at least one case, the third party library source code was not available. Also, in many cases the dynamic nature of some algorithms requires dynamic memory management. You cannot statically allocate everything, especially in a limited memory environment.

I know we're not the only ones to have been burned by Wind River's malloc. I know several major companies that also had to replace Wind River's code.

As far as being able to dynamically replace code, VxWorks isn't alone in that. Numerous other RTOSes out there can do the same thing, including QNX. QNX even supports the concept of a hot standby process to take over if the main process dies.

To give you an idea about how Wind River's malloc works, they keep a sorted linked list of fragments from the smallest to the largest. When you try and allocate a block, it walks the linked list until it finds a block large enough. Likewise, when you free a block it checks if it can coalesc the block with a neighboring block. It then goes through the linked list looking for a slot to insert the free block.

Yes, VxWorks may have been around since the 80's, but that's part of the problem too and it is showing its age. In the 80s embedded processors typically did not have MMUs. Now MMUs are quite common in the more powerful embedded processors.

You say you can't have low latency and memory protection? QNX proves that you can. It is low latency and *very* robust. If your driver dies, no problem, restart it. Timesys Linux also has a very low latency, although not as low as QNX. Timesys also has an interesting feature where you can guarantee CPU and networking resources. I can schedule a task to be guaranteed 5.8ms of execution every 8.3ms and it will guarantee that that task will get the CPU time allotted to it with the desired resolution. This is without increasing the system tick rate (usually 10ms). Timesys can also schedule a task to be higher priority than an interrupt. I'm not as familiar with QNXs scheduler, but it's also quite flexible from what I've heard.

As far as FAT, it is not a robust filesystem. It never has been. If the FAT gets corrupted or a directory entry gets corrupted it's difficult to recover. Other than possibly having 2 copies of the FAT cluster table, any corruption can be difficult to repair. If the FAT table gets corrupted, which table is corrupt and which is not? If a directory entry gets corrupted, it can be impossible to fix. For flash memory, unless you are using a device with special wear-leveling, FAT is about the worst choice since any file write that changes the size of a file requires a write to the directory entry and possibly the FAT table. If the table gets corrupted and you don't run a repair operation (which often ends up leaving orphaned files as lost clusters), the file system can happily corrupt itself to death. Why do you think every time DOS/Windows9x/ME crashed it had to repair the disk with scandisk? FAT is a poorly designed file system that was originally designed for 160K floppies and scales poorly. FAT32 is an improvement, but it's still not very robust. For flash, something like Linux's journalling flash file system 2 (JFFS2). More information on VxWorks file system support can be found here.

Basic VxWorks information can be found http://www.slac.stanford.edu/exp/glast/flight/docs /VxWorks_2.2/vxworks/guide/.