Reformatting a Machine 125 Million Miles Away

An anonymous reader writes: NASA's Opportunity rover has been rolling around the surface of Mars for over 10 years. It's still performing scientific observations, but the mission team has been dealing with a problem: the rover keeps rebooting. It's happened a dozen times this month, and the process is a bit more involved than rebooting a typical computer. It takes a day or two to get back into operation every time. To try and fix this, the Opportunity team is planning a tricky operation: reformatting the flash memory from 125 million miles away. "Preparations include downloading to Earth all useful data remaining in the flash memory and switching the rover to an operating mode that does not use flash memory. Also, the team is restructuring the rover's communication sessions to use a slower data rate, which may add resilience in case of a reset during these preparations." The team suspects some of the flash memory cells are simply wearing out. The reformat operation is scheduled for some time in September.

Comments

Hey, Bob, this is Jim

[Anonymous Coward]

We're gonna need you to go out to the rover and reboot it. Yeah, it got stuck. You should probably leave ASAP.

Simple fix

[SternisheFan]

Easy, just gotta' replace the button battery.

Re:Simple fix

[sillybilly]

It's running on solar power, that's how it lasts 10 years. Though the rechargeable battery must be tough to take so many recarchings.

Ideally, you have redundant systems for such a situation, where you can take one of them down and use the other to do the booting, formatting, programming, as if there were a user sitting right next to it. They say it has a flashless mode of operation, but the way I think of it, as in a regular PC, with a BIOS, you can reformat the harddrive without booting off of and using the harddrive, such as booting from a floppy, or even ROM chip they used to have back in the 80's (ROM-DOS 3.3 or ROM BASIC). So when flashing a BIOS or a ROM chip, there is no lower level to boot from, but if you have Tandem, dual redundant systems for everything, you can boot from the lowest of lowest levels and have the partner system execute all the commands. So with Tandem failure is less frequent, as in, you're down to 50% capacity but still fully functioning ok, and can work on regaining the 100% capacity, while not using regular operations, for two days and the like. The problem with Tandem is the double or higher cost, and, in space missions, the extra power consumption and extra weight, and in space missions, weight is almost everything, as each lb has to be paid for dearly, on the order of $10,000/lb low Earth orbit, and who knows how many gazillion dollars per lb for a Mars mission.
There used to be a company named Tandem, designing dual CPU redundant resilient failure tolerant systems, but they fell behind on chip design because of small size, plus high expense, and did not compete well in the computing field. For instance back in 1999 when Google started, they started with regular pc's of whateve the vogue of the day was, I don't know, 700 MHz PIII, maybe? And just jerry rigged a bunch of them into a daisy chain and voila, you have a Tandem-like, more than dual, more like thousandfold or millionfold duplicated, resilient supercomputer. But the principle of tandemness and fault tolerance was there. Maybe for space missions that need fault tolerance like that, it may be worth the extra rocket fuel weight in the first place to double the weight and duplicate most critical systems. The human body duplicates kidneys, lungs, but not liver, or heart, so there is a balance on what you want to go redundant on and what not. Life is easy with 2 kidneys, some people can live with only one kidney, but it's really difficult to live with zero kidneys.

Re:Simple fix

[Anonymous Coward]

Wow. Talk about missing an obvious joke and over-thinking the response. Seriously epic *WHOOSH*

Re: Simple fix

[Anonymous Coward]

Ass-burgers.

Re:Simple fix

[davester666]

what's this step 4??

Press the reset button.

Who the hell designed this stuff?

Re:Simple fix

[sound vision]

tl;dr on the whole post BUT... I've had my iPod nano in daily use for the past 8 years and it's still going strong. True, it doesn't need to power any motors - but the design specs probably also allocate a lot less weight to the battery.

Re: Simple fix

[LinuxLuver]

Don't think he missed it. The previous comment probably just provoked a related tangent.

And I thought I was cool...

[toygeek]

When I reboot machines in Asia or UK/EU using IPMI from the US.

Re:And I thought I was cool...

[marcello_dl]

And I thought I was cool when I reboot servers around the world thinking I am rebooting mine.

If there is a problem and need to call "support"

[Anonymous Coward]

do they get sombody in or from India?

Send someone

[TheDarkMaster]

With a replacement SLC SSD and a screwdriver

ECC?

[TechyImmigrant]

They didn't do any ECC on the flash memory? I thought these people were rocket scientists.

Re:ECC?

[Anonymous Coward]

As it happens, for flash, read errors are often transient. A better model than DRAM style ECC is to treat it more like a disk drive with checksums on each block. If you get an error, reread the block. And if you have a problem writing a block (e.g. the readback is wrong), just use a new block. Surely you've noticed that your USB thumbdrive gradually gets smaller with time as blocks wear out. (In space hardware, back in 2000, wear leveling was done manually.. still is as far as I know.. there's no nice rad-hard flash controller chips to make a big pile of MLC flash look like a disk drive, etc.)

The long duration radiation performance of flash memory (particularly back in 2000, when these things were being designed) was/is not particularly well understood. There are a lot of what is called Enhanced Low Dose Radiation Effects (ELDREs) that are poorly understood for all semiconductor devices: you can't just blast the part in an accelerator at 1kRad/hr for a few days to get to a few hundred kRad and expect that this is the same as taking a few tens of Rad/hr over days and days and days, with 12 hours off after the sun goes down to anneal and heal.

And, because resources on spacecraft are very precious, one doesn't blindly head off and say "let's just TMR everything". You make a rational choice based on the expected design life and the data you do have and pray for the best.

And, of course, the design life was 3-6 months, and here we are 10 years later, still cranking along. I think it's done pretty well, all things considered.

Re:ECC?

[Anonymous Coward]

The rocket scientists did their job ten years ago. They're working at McDonalds now.

Re:ECC?

[schlachter]

Well, in their defense, ECC on the flash memory isn't exactly rocket science.

Re: ECC?

[Anonymous Coward]

This would make an interesting movie plot where they have to recall all the older, laid off rocket scientists working at McDonald's and bagging groceries at the supermarket to reboot an idle probe on a far away planet because it's the only one that can be repurposed to save the earth from an asteroid impact. But only the old guys know the hardware and can reprogram the firmware.

Yeah I'm a laid off old guy. Get off my lawn!

Re: ECC?

[rickb928]

And add in the volunteer group that decided to save the project, working out of an abandoned McDonald's.

Oh, wait....

Re:ECC?

[Nimey]

You're a poster child for Dunning-Kruger: some random on the Internet who thinks he's smarter than the folks who designed a Mars rover that lasted over 10 years past its 90-day expected life.

Re:ECC?

[DMUTPeregrine]

There's also the matter that better ECCs cost more overhead. You can detect single bit errors with a simple parity bit, but double errors will go undetected. And even something like Reed-Solomon can't correct all the errors it can detect. Spacecraft going to mars have very limited mass budgets, there are often better places to spend the extra mass than on an additional redundant flash chip (and associated circuitry).

Re:ECC?

[Noah Haders]

They clearly overspecced it. Maybe if had been designed more reasonably they could have sent more.

Re:ECC?

[M1FCJ]

Most of the hardware cost is the launch vehicle, not the rover.
Most of the people (salary) cost is the people working on the data generated (all accross the universities around the world who analze the data and write papers), not the designers.

Underspeccing it wouldn't have saved much.

There's one that breaks this rule, the JWST. Just the endless redesigns have gobbled up so much money, I don't believe there will be enough Science generated by it to cover the build & launch costs.

Re: ECC?

[daremonai]

No wonder the last quarter-pounder I had tasted a little odd.

Odder than usual, I mean.

Re:ECC?

[lightbounce]

ECC use is standard with all flash storage. Flash is so unreliable that it can't be used without it, and it has nothing to do with the hard radiation environment on Mars. As for wear leveling, it's been standard since at least 1990 with the first attempts at flash storage. Why the rovers don't do it, I don't know. Maybe because it requires too many cycles of an already limited processor, plus dedicated storage space to keep "use counts" of all the flash blocks.

Re:ECC?

[Agripa]

The long duration radiation performance of flash memory (particularly back in 2000, when these things were being designed) was/is not particularly well understood.

Flash is another form of floating gate memory. Wouldn't the known long duration performance of EPROM and EEPROM apply?

Re:ECC?

[Agripa]

ECC use is standard with all flash storage. Flash is so unreliable that it can't be used without it, and it has nothing to do with the hard radiation environment on Mars.

NOR Flash does not normally use ECC and has reliability closer to that of EEPROM than NAND Flash.

Re:ECC?

[TechyImmigrant]

>You can detect single bit errors with a simple parity bit

You can detect (2^32-1)/(2^32) of every possible failure pattern with a CRC. With a combination of a multiple bit error correction algorithm (with most correction schemes n bits can be corrected with 2n redundant error correction bits) and then the CRC can be used to tell if you correctly corrected the data.

Re:ECC?

[WaffleMonster]

You're a poster child for Dunning-Kruger: some random on the Internet who thinks he's smarter than the folks who designed a Mars rover that lasted over 10 years past its 90-day expected life.

Not too often but occasionally the stupid get lucky and in some perverted way lack of knowledge and consideration of detail can lead to better outcomes.

After awhile one has to admit having to be careful when you transmit for fears it would even be possible for commands to be misinterpreted or designing something which knowingly continually writes to flash memory using DOS era FAT filesystems is not a winning play no matter how much you throw the reliability arguments at the wall and expect them to stick.

And all those commenting about what they instinctively noticed with their ignorant eyes as curiosity's chintzy wheels turning out to in fact be objective reality.

The engineers might be smarter than us fools and idiots yet it does not automatically follow they were actually correct to make a particular tradeoff or the fools and idiots don't have a point.

Usually best to stick to the facts and make arguments from merit vs accusing people of staying at a holiday inn express last night.

Re:ECC?

[TechyImmigrant]

>You're a poster child for Dunning-Kruger

Actually I am an engineer who has designed many error correction circuits for communication and storage systems. I think I know how much I know about error correction systems, which is plenty for this conversation.

While the statement was made in Slashdot jackass style, the question is legitimate. Why didn't they do any or more ECC on the flash that is failing. There is probably a perfectly fine answer like "We knew the expected error rate and It was designed to last 10 times longer than the system", but the system lasted 40 time longer so the ECC correction capacity was exceeded". or "We had TMR, but the give then age of the system the error rate is now such that the error collision probability is too high". or "This flash claimed to be rad hardened but it turns out it isn;t".

I'd like to know the answer because I like techy shit.

Re:ECC?

[romons]

On the other hand, it is all still working. It reboots occasionally. My computer does that. By reformatting, they will map out any bad sectors, which is probably the issue, and it'll run for another 10 years. Sounds like a smart technology tradeoff to me. Use cheap, off the shelf hardware, and KISS it to death. Write a special driver, or build special hardware to do ECC, and you end up with a bug that causes the system to freeze in an unrecoverable way.

Alternative Title

[wisnoskij]

How to brick a 2.5 billion dollar device.

Re:Alternative Title

[TapeCutter]

Not sure if it was opportunity or its twin, but one of them required a modem reset not long after landing.

Re:Alternative Title

[rasmusbr]

I would imagine that the system probably boots itself off of a ROM chip that has a routine for receiving data from Earth and storing it in RAM and then flashing that data onto the flash chip.

If the rover does not boot from ROM then it is a miracle that it hasn't bricked itself yet.

Re:Alternative Title

[wisnoskij]

And the state of the hardware. Some unknown number of systems on the real curiosity are degraded to the point of malfunctioning; And they have little to no way of exactly measuring what and where.

Re:Alternative Title

[dotancohen]

And the state of the hardware. Some unknown number of systems on the real curiosity are degraded to the point of malfunctioning; And they have little to no way of exactly measuring what and where.

Opportunity. Curiosity is on the other side of Mars, nuturing holes in its wheels and looking for cats to kill.

Re:Alternative Title

[Zarhan]

Not modem reset. The filesystem on Spirit had bunch of temp files and other stuff from the Earth-Mars flight, and apparently it just ran out of inodes. So basically they had to remote into whatever constitutes a bootloader with 20 mins of latency and remove some of the no-longer-needed files.

See http://science.slashdot.org/st...

Re:Alternative Title

[Nimey]

Never attribute to malice that which is adequately explained by stupidity.

Re:Alternative Title

[Agripa]

I would imagine that the system probably boots itself off of a ROM chip that has a routine for receiving data from Earth and storing it in RAM and then flashing that data onto the flash chip.

I wonder if the ROM would actually be a floating gate ROM instead of mask ROM or fuse based PROM in which case it would be more like EPROM or NOR Flash.

Does anybody even make mask ROM or fuse based PROM any more?

Re:Alternative Title

[rasmusbr]

I checked and it is EEPROM. And there are two EEPROM:s, I presume those are for redundancy in case one gets zapped.

Is it running Windows?

[mark_reh]

Is it?

Re:Is it running Windows?

[Psychotria]

You're probably joking, but the OS is VxWorks.

Re:Remote management

[ledow]

Not really...

The chances are that "reformat" isn't what we think and includes one of more of:

1) Rewriting cells and allowing wear-levelling and sector-replacement to take place, and make bad sectors as bad.
2) Write-testing and manually avoiding those sectors that don't perform as expected.
3) Rewriting all the critical storage functions to avoid the already-known bad sectors.

It's the kind of thing that anyone can play with. Not saying it's not risky on a remote device, but BadRAM etc. patches have been in places for years and that's a way to run Linux on machines with faulty ***RAM****, not just long-term storage.

Many years ago, a bad sector on your hard drive was something you found out with scandisk (or previous tools) and then it was marked as bad and that was the end of that. Your PC wouldn't use it and so long as it wasn't the boot sector, that was the end of that. It was only the "creeping" bad sectors, where you got more bad sectors over time, that would really worry anyone.

I imagine that it's not at all difficult to make sure that multiple boot sectors were in place if you really wanted to but why bother? The chances are billions to one. Chances are this hardware has MUCH better fault tolerance and multiple hardware watchdogs, firmware, and boot attempts to make sure it eventually gets back up SOMEHOW.

There's a reason that even FAT stores two copies of the allocation table, why Linux ext filesystems store multiple copies of the superblock, etc. They come from a legacy where the occasional bad sector wasn't a problem and where 20Mb of hard drive cost more than the computer did so it was better to cope with the fault than just tell people to buy a new one. And their predecessors were (and still are) mainframes with hardware that's just that fault-tolerant in the first place anyway.

It's not at all hard to write a filesystem that can cope with not only damage, but even recurring damage. You've seen PAR files presumably? The same could easily be done on a filesystem-level basis (and I imagine, somewhere, already is for some specialist niche).

It's not that big a deal once they KNOW that's the problem. The biggest problem is that they only "suspect" that's the problem.

Err, if you're a system admin..

[Viol8]

... you're not cool. Period. Sorry.

Re:Err, if you're a system admin..

[Anonymous Coward]

Everybody is a system admin when linux is involved, and I like it that way. But I digress.
alias halt='echo Use shutdown instead'
alias reboot='echo Use shutdown instead'

Re:Err, if you're a system admin..

[Anonymous Coward]

Typing out "Period" makes you look retarded.

Re:Err, if you're a system admin..

[marcello_dl]

Err, if taking a server offline, no matter the reason, is a serious problem, then you are not a good - or properly funded - sysadmin.

Re:Err, if you're a system admin..

[TeknoHog]

Chill out. They're just having that time of the month.

Re:Err, if you're a system admin..

[bbsalem]

I think he was talking about "In the current directory" :-)

2.5 billion?

[Viol8]

I dunno so much these days. Its 10 years old and got a few miles on the clock plus collection for the new owner would be an issue. On the plus side vandalism won't be a worry. For a few centuries anyway.

Alternative Title

[Whiternoise]

They will almost certainly do a dummy run on an identical piece of flight hardware on Earth. The only difference is how the data are sent.

Why is it not trivial?

[nurb432]

Why didn't they plan ahead for this sort of operation in the beginning, making it painless and 'reliable' ( as possible ).

Re:Why is it not trivial?

[beelsebob]

Who says they didn't?

Re:Why is it not trivial?

[SeaFox]

Why didn't they plan ahead for this sort of operation in the beginning, making it painless and 'reliable' ( as possible ).

That's a joke, right? We are talking about one of the two rovers that was sent to Mars on a mission planned to only last 90 days. They didn't see "flash memory wearing out from use" as a contingency they needed to plan for.

Re:Why is it not trivial?

[nurb432]

What kind of handwaving armchair wannabe are you?

One that plans ahead well enough that this would not considered 'news'. Instead it would be just SoP.

Re:Why is it not trivial?

[Nimey]

Put up or shut up: who are you, really?

Re:Why is it not trivial?

[nurb432]

Don't need to prove anything to a potty mouthed coward.

Re:Why is it not trivial?

[Nimey]

If you're going to troll and be full of shit, save us both time and say so up front.

Re:Why is it not trivial?

[Nimey]

Further, if this is you:
http://www.fiero.nl/cgi-bin/fi...

You're conclusively an idiot. Only an idiot believes in homeopathy.

Re:Why is it not trivial?

[Nimey]

Boring troll is boring. Put your back into it, boy.

Re:Why is it not trivial?

[nurb432]

Sorry, not the same person. No "nickname" is overly unique in the online world these days. Its not 1980 anymore.

But i dont want to blow my karma.

Re:Why is it not trivial?

[Nimey]

Yeah, bullshit. Your nickname has enough entropy that it's exceedingly unlikely this is not you.

Re:Why is it not trivial?

[nurb432]

Nah, its not. But believe what you wish, it is a free country. ( assuming a US citizen here )

Re:Why is it not trivial?

[Nimey]

I don't believe a word of it.

Re:Why is it not trivial?

[Nimey]

You really need to level-up your trolling. This is 101-level shit, son.

Re:Why is it not trivial?

[nurb432]

You are more than welcome to get a court order and demand IP addresses, then compare them. I'm sure both places log IP+posts.

Re:Why is it not trivial?

[Nimey]

*snore*

Re:Why is it not trivial?

[sound vision]

The specifications for this mission were that the rover should last 6 months on the surface. Currently we are at over 100 months.

Re:Why is it not trivial?

[TheDarkMaster]

Do not waste your time with the Nimey, he's just another troll who can not understand what others write and he take pleasure in offending others when writing.

Re:Why is it not trivial?

[nurb432]

Ya, i figured that out, a bit too late.

Re:Remote management

[pegdhcp]

Ultrix used to mark bad sectors on the fly, as far as I could remember, if the disk was not a SCSI...

Assumptions

[DaMattster]

I believe NASA is operating under the assumption that the rover's on board flash memory is still serviceable. 10 years ago flash memory was still in its relative infancy. A reformat and reload risks bricking the rover completely.

Re:Assumptions

[beelsebob]

I believe you're assuming that the flash used on a rover that went to mars, and encounters all kinds of crazy radiation, is in some way similar to the crappy OCZ thing you stuck in your PC 10 years ago.

Re:Assumptions

[M1FCJ]

Don't forget, we don't hear what the techies are talking about. What we're hearing is what the techies told to the PR guy distilled down to a journo, being summarized in The Register (!) and some other soft-tech sites, finally an inaccurate summary on the frontpage of Slashdot.

I wouldn't be surprised if it were just a "fsck.ext4 -cc" (I know it's not an ext4, it was't even released when Opportunity soft-crashed and bounced around on Mars nor it runs Linux).

Re:Assumptions

[M1FCJ]

Assigned a macro to this reply, haven't you? Clever boy, want a medal?

Re:If there is a problem and need to call "support

[sillybilly]

I'll be glad to help you with that Sir.

Failing flash cells?

[Twinbee]

I didn't realize they used OCZ for the storage tech. ;)

Re:Failing flash cells?

[Anonymous Coward]

It was designed to last 3 months and failed after 10 years.
If OCZ was involved, it'd be the other way around. ;)

I"d hate to be the guy

[TigerPlish]

I'd hate to be the guy a) pitching this operation at the change control meeting, and b) the guy signing off on this change.

Re:Remote management

[lgw]

You've seen PAR files presumably? The same could easily be done on a filesystem-level basis (and I imagine, somewhere, already is for some specialist niche).

While all hard drives now do their own Hamming error correction (or something better), RAID2 is the same idea for "raw" storage that doesn't: you write explicit ECCs to redundant volumes to allow recovery from both drive loss and bad sectors.

RAID5 with modern drives gives all the same resiliency, as the drives do the block-level ECC themselves, so you never see RAID2. But for a pile of flash memory, that's the filesystem-level equivalent of PAR files.

It worked on Spirit

[lemur3]

they had to do this type of thing on spirit shortly after it arrived on mars..

read more here: http://trs-new.jpl.nasa.gov/ds...

or the PDF linked therin here http://trs-new.jpl.nasa.gov/ds...

its got all sorts of awesome details.

We commanded a shutdown, which terminated the
current communication window, and the loss of signal occurred at the predicted time. Fifty minutes later, we commanded a beep at 7.8125 bps to alert us if the shutdown command did not work, and much to our disappointment, the beep was received!

really a fun read. ..im guessing theyll be doing a lot of similar stuff

Deploy the Paperclip!

[apraetor]

I'm picturing something akin to those Shuttle missions to repair flaws in the Hubble telescope's optics, except involving a NASA-engineered paperclip.

stressful job

[lkernan]

Man, hope they don't select the wrong partition.....

Capacitors

[njhunter]

I'm sure they didn't get any of their capacitors from that bad batch a few years past.

Re: Protip

[Anonymous Coward]

They should have used what genius?

Re: Remote management

[M1FCJ]

Hah, I remember running the DOS debugger, poking into a certain address in the memory to access the MFM BIOS, then you could do a low level format where you could enter the sectors to mark as bad. Those were the days...

Re: Remote management

[M1FCJ]

"g=c800:5."
Hah, I almost remembered that one, good old Seagate controllers. I had the 800 but not the rest. :)

Sort of like ...

[PPH]

... handing over remote desktop access to tech support in Bangalore.

Now if only we could get a Martian to IM during the process: "Yes. The little red LED is blinking ....."

sad

[electrosoccertux]

can parent be modded funny not insightful? Insightful is too depressing...

Do unto others...

2014-- year of Linux on Mars?

[electrosoccertux]

???

anything starting with "why didn't they just..."

[electrosoccertux]

shoot the asker?

Re:Remote management

[MasterOfMagic]

It's not at all hard to write a filesystem that can cope with not only damage, but even recurring damage. You've seen PAR files presumably? The same could easily be done on a filesystem-level basis (and I imagine, somewhere, already is for some specialist niche).

You mean like RAID-5? Because RAID-5 was part of the inspiration for the PAR2 format.

Re:If there is a problem and need to call "support

[sillybilly]

Sometimes when I sound mocking, ironic and sarcastic, I'm actually serious, as in ironic-ironic, or sarcastic-sarcastic. A lot of Americans simply smack the phone down on Indian tech support, saying gimme somebody who speaks English. I patiently listen to them struggle through it.

Re:If there is a problem and need to call "support

[Noah Haders]

that makes no sense.

Re: Remote management

[lightbounce]

I always thought that the disk controller should do idle scrubbing. Are there any modern SATA disks that do this?

No, the drives themselves don't do this because it pulls the head away from where the host wants/expects it to be. This would result in a lot of unexpected thrashing. If scrubbing is to be done, it is best done by the OS as a background task.

The Real Rover Problem Explained

[magicandjewel]

Flash memory isn't the Rover's problem. It's still running XP and there are no more hot fixes. At this point the Rover's system has massive "bit rot," not to mention that it's been hacked countless times by the Chinese. Undeterred by this seemingly insurmountable problem, Microsoft has donated a Windows Phone for communications back to earth and a Surface Pro to power the Rover "because it's just like a computer." They didn't say just who's going to operate their touch-only interfaces. It all makes perfect sense because nobody in their right mind buys those things down on earth. Thus, new markets like Mars are vital to both products' successes. You might wonder how they will get into space. Microsoft has also kept mum on that, but the word is that there is still so much gas leftover from the Ballmer era that achieving liftoff is a trivial undertaking. -- Cary R., Microsoft Senior Technical Writer (ret.)