Slashdot Mirror
Mars Rover Spirit Back Online
Skyshadow writes "Just in time for the arrival of its twin, the Spirit Mars Rover is back in working order. Programmers at the JPL have traced the problem to the rover's flash RAM, which it uses to maintain its filesystems. They are using a ramdisk in the rover's RAM to bypass the bad flash memory, and are working on a workaround for the bad flash. Good news, but the rover is still potentially weeks away from full operational status."
During all of the "Spirit is broken" columns, I kept reading /. comments saying that it was likely a memory error due to the non-consistent errors...I guess a million monkeys with a typewriter can be correct :-)
Doh!
Engineers guessed that Spirit's troubles were in its Flash memory and set about sending the rover a complex series of instructions to see if they could get it to bypass the corrupted memory. Theisinger said engineers sent Spirit a command just before its daily "waking up," telling it to shut down and restart in what is known as "cripple mode," using RAM instead of Flash for its start-up instructions.
Some people may take this sort of thing for granted, but I for one find it remarkable that we can essentially reboot and perhaps even fix a system that is on a whole other planet.
Just wait until we have Interplanetary, Interstellar, Intergalactic Remote Desktop. I'm only half-joking.
The coolest voice ever.
Shouldn't they have like 5 Flash RAM's? Really,they shouldn't have one of anything. In my computer if my BIOS fries, I pop open the box and replace it. If it fries on mars, obviously I kiss my megamillion dollar project goodbye, all for a $5 Flash ROM.
Is there a chance that the problem could've been caused by electrostatic discharge? Rover bounces on rubber airbags on sand, bags fold up, Rover rolls off, Rover touches rock - zap!??
...will apparently cause one out of every trillion bits on Earth to flip randomly... I guess with less of an atmosphere, it is a bigger problem on Mars! ;)
libertarianswag.com
I have a friend who works in the field. Space travel hoses electronics bad. Triple redundancy and over-engineering is the name of the game. This is nice to hear. I would imagine that something went wrong intransit or on-landing, but they can keep going,
Great ideas often receive violent opposition from mediocre minds. - Albert Einstein
I know a lot of ppl are using flash ram in smaller computers for booting linux or what not. Well if they are writing their logs and other things to that flash be aware that you can only write to it so many times before it fails.
Was NASA writing to that flash or just reading? A ram drive in flash sounds like it will access/write thousands of times a ?minute? This should wear it out quickly.
Ive been unable to find any hard information on the design of the MER memory systems. If anyone can point me to a technical brief id be very happy.
From what ive pieced together the MER system is something like this:
One RAD6000 powerpc cpu.
Connected via probably compact pci to 128 mb of ecc sdram.
256 mb of flash. No info on what make of flash, but likely Intel since they are the biggest. There was some info from the press conference that there are actually two flash chips and that the flight software is redundantly stored on each. So does this mean that there is actually 128mb of redundant flash? Also it was said that they had problems even with the redundancy, could they possibly have overwritten something? We all know that even a redundant raid does not stop filesystem corruption.
No information on how the flash is connected, parallell / serial? How the redundancy works?
Btw, I guess flash is rather radiation hard since they require 10 - 20V to erase / write.
I know those guys at NASA are smart... so does anyone know why they sent opportunity right after spirit? I would think it would be better to wait and see if any problems occured with the Spirit, and learn from their mistakes. I'm sure there is an optimum aligning of the planets for a launch... but is that really so rare that they couldn't wait?
Opportunity is fast approaching the red planet. It should be an interesting night at JPL. Execellent work guys, good luck.
[alk]
I don't know that much about VXWorks, but I heard that one of its main assets is having a very small tight multitasking kernel.
They were able to regain the system, despite loss of a major computational component. Remotely. Through a debug link. That sure says a helluva lot for the robustness of the OS and how they configured it.
Good job, JPL.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
So... I wonder if they'll consider validating MRAM more quickly if Flash is found to be more error prone.
You know how NASA works. The Space Shuttle running on 486's and whatnot. I understand the science behind that reasoning, as sad as a 66 MHz processor seems to us geeks nowadays, but I wonder if MRAM will prove more flexible and stable for future space missions.
"...Well, there's egg and bacon; egg sausage and bacon; egg and spam; egg bacon and spam; egg bacon sausage and spam..."
This is a bit OT, but I need to rant:
A quote from his site: "It is incredibly frustrating watching all this happen... I can't even say the word Lisp without cementing my reputation as a crazy lunatic who thinks Lisp is the Answer to Everything"
I feel his pain. I was introduced to Lisp not too long ago, and within a short time, a Lisp-derived language (Dylan) became my favorite. I also found that many of the features I loved from Python were very Lisp-y in nature. Now, I see Java and C# either neglecting all the knowledge garnered from the Lisp-family of languages, or reinventing it --- badly. The features in C# 2.0 have either been in Lisp for decades (lambdas, closures) or are not necessary in Lisp (iterators, enumerators --- which, btw, are theoretically not necessary in C# 2.0 either because of lambdas and closures!) This new "Xen" (or X#) language Microsoft Research is pushing takes a great idea (extending the language to fit the problem domain) that has been a part of Lisp for decades, and chops it off at the knees. Instead of having proper macros, so you can extend the language to fit *your* problem domain, they hack support for a single problem domain (back-end business programming) into the language itself!
That said, the Lisp community is to blame as well. Part of the reason people stop listening the moment somebody says Lisp is that the Lisp community is *so* rabid and *so* unyielding. Especially some high-profile members who are highly respected within the community despite the fact that they are completely obnoxious and lack any human sense of manners.
A deep unwavering belief is a sure sign you're missing something...
Flash RAID array.
(Can this even be done?)
Weeks of coding saves hours of planning.
with all the radiation and very high energy particles zipping thru the spacecraft on its way there, I'm suprised any computerized spacecraft get anywhere intact.
"It's so convenient to have a system where everyone is a criminal" - A. Hitler
Lambdas are just an aesthetics/keystrokes issue. :: jaw drops :: :: gets on knees ::
---
I bow down to your ignorance, oh mighty King of the Cluless!
Seriously, though, please research lambdas. They don't just save typing. They are *everything*. All of computation can be described just with lambdas of a single parameter. Everything else is just syntax suger. If you ease one restriction of the lambda calculus (no side-effects), lambdas can do procedural code, functional code, and even object-oriented programming. That's why I said iterators, enumerators, etc, are not necessary in C# 2.0, because it has proper lambdas. All of thsoe can be implemented very easily on top of lambdas.
and start focusing on really useful subjects - like formal verification
---
You can do formal verification in Lisp. Look up ACL2 (first link in Google search for that term). I'd still say that Haskell or Clean are a bit better for such purposes, but mainly because they are designed from the beginning to be comfortable to program without side-effects, while Lisp was not.
and strong typing (which go very well together).
---
C'mon. You're making this too easy. Lisp has a strong type system. All type errors are caught, unless you disable type-checking in your compiler. Maybe you mean *static* typing instead?
That's probably because the only people who think Lisp Is The Answer to Everything are a little bit insane.
---
Um, wasn't that my point? Its the "Lisp is the Answer to Everything" people that make it harder for normal people to push Lisp to areas where it would be really useful.
A deep unwavering belief is a sure sign you're missing something...
Besides robot exploration software would be handy right here. It would be neat to be able to send a research bot out in the deserts, deep oceans and jungle canopies of the world. Machines can go where we can't.
Individually you can be damn annoying sometimes, but I'm constantly amazed and delighted by the collective intelligence of the /. pack.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
I was going to mod this one up, but I decided to give this reply some more emphasis by actually replying with some thoughtful encouraging words instead.
It would be nice to be able to have some folks at JPL throw down the source code and engineering schematics and say to the geek/space/engineering community at large "We have a problem here and could use your suggestions to see if we can get this fixed."
This (the mars missions) is obviously a big hit, as measured by replies on Slashdot, the number of hits on the website at JPL, stories in mainstream media, and other reasonable metrics to gague popularlity of a project. I'm sure that there are several geeks out there that wouldn't mind digging into the source code.
The only reason I could see the engineers not wanting to do that is to open themselves up to obvious scrutiny for poor engineering and coding. (Whadda you mean the global variable named temp is the only variable. We also have temp2, temp3, and temp4. What do the numbers mean in those mean? You can get it from context, can't you?) That and some people just aren't used to allowing other into their "domain".
Being 100% funded by public money should also be further reason for why this should be opened up. I also totally agree.