Slashdot Mirror
Mars Rover Spirit Back Online
Skyshadow writes "Just in time for the arrival of its twin, the Spirit Mars Rover is back in working order. Programmers at the JPL have traced the problem to the rover's flash RAM, which it uses to maintain its filesystems. They are using a ramdisk in the rover's RAM to bypass the bad flash memory, and are working on a workaround for the bad flash. Good news, but the rover is still potentially weeks away from full operational status."
They signed up for Mars Online with 3000 free hours. What they didn't realize was that the free 3000 hours only applied to the first month of service. Once they paid their MOL bill, they got hooked back up. All the probes friends on Mars use MOL!
They should boot faster, using linux. Then they'd only be ten seconds away :-)
"'I pass the test,' she said. 'I will diminish, and go into the West, and remain Galadriel.'"
- JRR Tolkien.
/riff/Move over Rover, let the ramdisk take over!/riff/
Wonder wehre they got they flash ram from?
--
I think they should return the bad flash part to where they got it and exchange it for a new part... although getting the memory back to the store by the 30 day warranty might be a little difficult.
I hope they bought the extended warranty.
/^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$/i
During all of the "Spirit is broken" columns, I kept reading /. comments saying that it was likely a memory error due to the non-consistent errors...I guess a million monkeys with a typewriter can be correct :-)
Doh!
Engineer 1: Ho-hum.. Little bit of ... whatever it is, 'ere... Hand me that thingamajig, will you?
Engineer 2: Yah, sure... Hey, remember that employee last month who got laid of within a week?
Engineer 1: Who? Vincent?
Engineer 2: Yeah, Vinnie... With the Italian accent?
Engineer 1: Yeah, him. What about the guy?
Engineer 2: Well, he has this offer on cheap RAM we just CAN'T resist!
Engineer 1: Really now? But-
Engineer 2: Look, our budget is already comparable to social welfare. We need to save some loot.
Engineer 1: Fair enough, buy the crap and hand me the other twisty-turny thingy over there? I need to screw on this name tag reading... "Spirit"?
Engineer 2: Look, it's either that or my wife's name.
Hate me!
If I was sending an embedded control computer to another planet, I would have chosen an OS with memory protection, not VxWorks. VxWorks is like DOS, and early versions of Windows, where one pointer problem in one task can corrupt the whole system. Sure, we don't know that's the problem now, but it would be nice to know for sure that it wasn't.
It's all good until tech support says, "So... Do you have a boot disk?" :-)
A cheerful little bird is sitting here singing.
Is there a chance that the problem could've been caused by electrostatic discharge? Rover bounces on rubber airbags on sand, bags fold up, Rover rolls off, Rover touches rock - zap!??
Here is the link to the real story. The one given in the /. acticle is getting pushed down spaceflight's page.
I mean like beagle isnt using its flashram anymore, just go and jack some off it. While your at it TAG the Beagle with some PRO-US graffiti :) hell maybe its got nicer rims too
Seriously, can you imagine the first manned expiditon seeing the Beagle Jacked up, tagged , up on little martian cinderblocks, All that and we already got a head start on building martian cities
Ive been unable to find any hard information on the design of the MER memory systems. If anyone can point me to a technical brief id be very happy.
From what ive pieced together the MER system is something like this:
One RAD6000 powerpc cpu.
Connected via probably compact pci to 128 mb of ecc sdram.
256 mb of flash. No info on what make of flash, but likely Intel since they are the biggest. There was some info from the press conference that there are actually two flash chips and that the flight software is redundantly stored on each. So does this mean that there is actually 128mb of redundant flash? Also it was said that they had problems even with the redundancy, could they possibly have overwritten something? We all know that even a redundant raid does not stop filesystem corruption.
No information on how the flash is connected, parallell / serial? How the redundancy works?
Btw, I guess flash is rather radiation hard since they require 10 - 20V to erase / write.
I have had some tough calls in my time but I have never had to walk a robot 283 million miles away through brain surgery. Man I am glad I did not get that call. This is going to blow there call averages all to hell. I raise a cup of Joe to you, Rover Help Desk man.
Papa Legba come and open the gate
This is the last image received prior to the recent issues with Spirit...
I remember in the last thread about the rover, someone opined that it was bad memory, then proceeded to give a half dozen reasons why. Totally nailed it.
Yeah, in the future NASA should just submit an Ask Slashdot whenever something goes wrong..
Here's a rant by a JPL guy about appropriate technologies for software on deep space probes. He recounts one story of a failed probe "100 million dollars, and 100 million miles away".
They fixed it. The fact there was a lisp REPL running on the spacecraft helped.
That's cool:
(unwind-protect
(progn (do-science)(talk-to-earth))
(wait-in-repl-for-earth))
where the russian cosmonaut says "American components, Russian components. They're all made in Taiwan!"
I'm watching NASA tv at the moment and they're explaining possibilities now. At the moment, they only have a very broad explanation of what's going wrong. However the newest knowledge is;
There are two separate flash memories on Spirit. At the moment, part of the problem is software which can read part of the flash memories as some of the operational software which is kept in flash ram seems to be coming up before the system reboots.
The system is rebooting no matter which flash memory is being accessed, it has the same bug both ways, so the flash ram itself looks to be OK, but the interface between the flash ram and the software looks to be causing resets.
Even if there were more backup flashrams, it looks like they'd still have this problem. Perhaps many, all on different controllers, and even an entire backup computer would have prevented this. at 100watts total power available for the rover, an entire extra computer may be a bit much to have fit. But then sending two rovers would also negate problems, and thats just what they've done
It seems most likely at the moment, according to NASA, that the family of components that are involved with the hardware addressing of the flash memories looks to be where the problem is.
Funny you mention that. I'm taking a class on design of digital systems at my university, and my professor works for JPL. He helps design the control systems onboard space vehicles such as the Mars rover. Anyway, a majority of the class grade is based on an end-of-the-quarter project, which we complete in groups of 2 to 4. On Wednesday he expressed interest in a group developing some sort of redundancy for FPGAs that would be suitable in spacecraft. You see, on Mars, you're not shielded from huge doses of radiation as you are on earth. A healthy dose of radiation bombardment could easily reprogram an FPGA chip on the surface of Mars; ASICs chips are used to overcome this problem.
Maybe he was gung-ho about anti-radiation redundancy because he already knew the likely problem of the Spirit. Who knows?
- sm
The Spirit is willing, but the flash is weak.
(Posted by Jane Slee and John Stracke in separate usenet postings.)
There is a big difference between standard flash and radiation hardened flash. In fact we are designing a project with one of these VME buss units as a storage array.