NASA Update Will Deal With Opportunity Flash Memory "Amnesia"

BarbaraHudson writes Computerworld has some details on NASA's latest fix to allow the Opportunity Mars Rover to store data while in overnight "sleep mode." Opportunity has been suffering from a glitch that's causing what NASA scientists describe as memory and data loss — or robotic "amnesia" — caused by flash memory deterioration since early December. Currently any information gathered is stored temporarily in RAM and must be sent to Earth so it's not lost when Opportunity powers down.

Comments

Flash memory sucks

[Russ1642]

This is why I'll never use an SSD in my computer. They're just unreliable.

Re:Flash memory sucks

[Shakrai]

And you think a spinning hard drive with platters and heads would be a better choice for a spacecraft that has to endure several G forces worth of acceleration at each end of the trip? Or perhaps it's a better choice to use magnetic media than solid state in high radiation environments like interplanetary space?

Re:Flash memory sucks

[gstoddart]

While that may or may not be true, it's kind of random.

These things were intended to last 90 days or so. It's been 11 years.

At this point, if they're still able to apply updates to make fixes to the damned thing it means this has outlived its originally planned lifespan by a massive amount of time.

I hardly think that's a fair critique of SSD in general. I'd say pretty much every part of that rover has performed well beyond anything it was ever expected to.

Re:Flash memory sucks

[Redbehrend]

I run SSD's on all my computers and never had one fail yet and when one does, it takes less than 10 min to get a new one up and running. (Already tested my restore plan with my backups) In the meantime I enjoy the awesome performance and get back my 10 min and money in x1000 from the time it saves lol.
Most of the ones that fail are the cheap clearance, no name ones, the good Samsung and Intel ones are developing a great track record.

But on topic, the rovers are all custom and it's been 10+ years, i believe they already know about this flaw now and they have found ways around it with newer flash memory.

Re:Flash memory sucks

[Russ1642]

I've had an SSD in my desktop for years now. It's fine. I think that poster might have a sense of humour.

Re:Flash memory sucks

[Shakrai]

I'd say pretty much every part of that rover has performed well beyond anything it was ever expected to.

All except for The Illudium Q-36 Explosive Space Modulator.

Re:Flash memory sucks

[bluefoxlucid]

Actually, the amount of NAND in Opportunity is 256MB, which isn't a lot. RAM is only 128MB; NAND is routinely given a full wipe and rewrite. Given it was only meant to last for 90 days, it's probably a low-grade MLC with under 5000 erase cycles reliability.

By contrast, a desktop SSD of 32-256GB will see a daily write cycle of under a gigabyte per day. The documents, cache, e-mails, and updates per day total very little--hundreds of megabytes, at best--for most workloads, up to and including intensive workloads such as 3D modeling and multi-layer 2D image editing, which produce massive work sets but only make minor changes to them. Large workloads include video editing, which does write multi-gigabyte files out on each large render operation (which may be frequent in some workflows).

System drive SSDs see small workloads even when used as a swap device: swap offloads a lot of stale memory from RAM, which is either hardly ever used, never written to, or simply stale. The brk() area can have fragmented holes too small to take new allocations, and so may page out entirely unused RAM to disk; much RAM (such as GUI elements, textures, models, and audio assets) contains load-once data that's written to disk when swapped, and then only read later, such that it's left on swap and evicted from RAM without writing out again when memory gets scarce. That means 2GB of swap might be 2GB of writes since boot time, plus maybe a hundred megabytes or less per day of continuous system run.

Finally, system drives also wear level internally. Writing to the same 1MB area over and over will spread the writes out: the controller will internally account for those blocks being erased and rewritten elsewhere, often by additive writing (writing without erasing) to avoid wearing the drive (e.g. you can have a used and erased map, in which anything erased or not used is free; you can then write to the used map and to the erased map, until you run out of free blocks and need to erase a newly used block, and thus need to mark all used-erased blocks free and mark no blocks as erased, costing one write-erase cycle even though you've written thousands of times). You'll get a full write-erase cycle every full data width write: even if you write to the same spot repeatedly, you only use one write-erase cycle writing 32GB to a 32GB drive, or 256GB to a 256GB drive.

Accounting for all this, the drives are quite long-lasting. For a 10,000 cycle drive, you'd have to write 320,000GB to a 32GB SSD or 2,560,000GB to a 256GB SSD to wear it out. That's 876 years at 1GB per day for a 32GB drive, or over 8 years if you're writing 100GB per day. For a 256GB drive, it's 7000 year-gigabytes, or 70 years if you're writing 100GB per day. Modern MLC NAND can survive 100,000 write-erase cycles before failure, so these lifetimes may be 10 times higher; high-end SLC drives can survive 10,000,000 write-erase cycles, and can back high-traffic SANs for decades.

They actually burn out less frequently than hard drives.

Re:Flash memory sucks

[Kjella]

Actually, the amount of NAND in Opportunity is 256MB, which isn't a lot. RAM is only 128MB; NAND is routinely given a full wipe and rewrite. Given it was only meant to last for 90 days, it's probably a low-grade MLC with under 5000 erase cycles reliability.

I very much doubt that. If you're going to spend that amount of money on sending it to Mars, you don't skimp on off-the-shelf technology that costs a few hundred bucks. You may have to ditch developing that custom system that'd increase the mission budget with a million dollars, but that's different. And if you want to make it as radiation-hardened as possible I doubt they'd go with anything but SLC for maximum signal strength. I doubt it has anything to do with write cycles which they presumably have full control over at all, it's probably from operating in the harsh environment for 11 years. No wonder it's getting a bit flaky.

Re:Flash memory sucks

[The Grim Reefer]

several G forces worth of acceleration at each end of the trip?

That must be what caused that WHOOSH.

Re:Flash memory sucks

[Anonymous Coward]

Voyager's magnetic tape drives would like to have a word with you

Re:Don't worry, NASA is really good at this

[NoNonAlphaCharsHere]

How many rovers do YOU (or **anyone** else for that matter) have on Mars?

Re:Don't worry, NASA is really good at this

[bobbied]

Keeping old relics and forgotten dreams alive is what they do best.

And here I thought it was being able to successfully launch people in billion dollar vehicles built by lowest bidders, loaded with millions of pounds of high explosives without killing anybody (usually)

Re:Don't worry, NASA is really good at this

[ArcadeMan]

(puts Mars bar on the floor)
"Come here boy!"
(rover sits on Mars)

(clicks on big Staples button) That was easy

Re:Don't worry, NASA is really good at this

[NoNonAlphaCharsHere]

Heh. I took the train into Chicago for many years. There's a Mars plant next to the line, with a passenger stop. Every morning the robot-voice would announce "The next stop will be ... Mars".

FRAM vs NAND

[volvox_voxel]

I've never been a big fan of flash memory, given that it has a finite number of write cycles before a memory bit fails (varying between 1 and 100million write cycles). The probability may be low that an individual bit may need to flip so many times in it's lifetime, but it's still an issue.. A lot of care must be taken by the firmware engineer to handle this. There are a lot of job postings for firmware engineers that understand flash..

I'm a huge fan of FRAM. It has a lifecycle limit that is quoted at being 10 trillion write cycles (some mention at it being infinite). The memory density is lower, but is a lot more reliable. It's biggest issue is that the density is lower. For a spacecraft, I'd much rather have a board of these 2Mbit FRAMS then a large flash chip. They use these things in smart meters, etc. In embedded systems, you have to be really careful not to write to the flash too often out of risk of damaging the flash. Most fast SD cards have their own dedicated microcontroller (ARM9, etc) to do what they can to extend the life of the flash..

A datasheet of an FRAM device: http://www.fujitsu.com/downloa...

One question I have is how FRAM compares to NAND-flash in a harsh radiation environment, and what are the radiation differences on mars vs the earth. How many vendors offer rad-hard processes for FRAM, and how do they perform?

Here is one link I could find on FRAM, but the report from 2011 is not clear:

http://cdn.intechopen.com/pdfs...

Re:FRAM vs NAND

[Solandri]

I've never been a big fan of flash memory, given that it has a finite number of write cycles before a memory bit fails (varying between 1 and 100million write cycles).

I'm a huge fan of FRAM. It has a lifecycle limit that is quoted at being 10 trillion write cycles (some mention at it being infinite). The memory density is lower, but is a lot more reliable. It's biggest issue is that the density is lower.

No, the biggest issue is likely cost. A quick Google search says a 4 megabit (0.5 MB) FRAM module costs about $50. That's $100,000 per GB. You can load your spacecraft with a couple dozen extra reserve NAND memory banks at 1/1000th the price. As one bank wears out, bring the next bank online, and it'll last far past the life expectancy of other components on the spacecraft.

In embedded systems, you have to be really careful not to write to the flash too often out of risk of damaging the flash.

That's only an issue if you have very small amounts of flash memory. i.e. You're making a conscious decision to limit the amount of storage, at the cost of the memory wearing out quicker. The more storage you put in, the longer it takes to wear out.

I can see it being an issue in certain extreme applications. e.g. extremely low power embedded systems where it may have to last 10 years on a single battery, so you don't want something like a SSD wear leveling controller or an overabundance of NAND sucking up extra power. Or high frequency database storage, where having a large amount of storage can actually slow things down, but you still need good write endurance. But for most applications, it's cheaper just to throw more NAND at the problem.

Re:FRAM vs NAND

[Solandri]

Forgot the tl;dr: Basically this is a problem which will disappear on its own. Opportunity has only 256 MB of flash memory (which was quite a lot when it was launched in 2003 - I paid $100 for a 512 MB flash card in 2004). So each individual NAND cell is being used a lot. As you put in larger amounts of flash memory, each cell gets used less frequency, and the problem goes away on its own.

Re:FRAM vs NAND

[necro81]

I did a bit of reading on the subject from TI, which has FRAM integrated into some of its MSP430 microcontrollers. If anything, the technology seems to be well-suited to the space environment, because bit storage is accomplished via a crystal structure change (polarization), rather than through charge storage.