Slashdot Mirror
Debugging The Spirit Rover
icebike writes "eeTimes has a story on how the Mars Rover was essentially reprogrammed from millions of miles away. 'How do you diagnose an embedded system that has rendered itself unobservable? That was the riddle a Jet Propulsion Laboratory team had to solve when the Mars rover Spirit capped a successful landing on the Martian surface with a sequence of stunning images and then, darkness.' The outcome strikes me as an extremely Lucky Hack, and the rover could have just as likely been lost forever. Are there lessons here that we can use here on the third rock for recovery of our messed up machines which we manage from afar via ssh?"
That's the thing that amaze me. Any technology having to do with space seem that much more advanced.
Here on earth we can't even build cars that require no maintainance and last more than 10 years.
If it was the hardware that got fried and they miraculously fixed that, I would understand but this was just a software glitch.
I routinely reboot and reprogram machines in our data-center that is 2000 miles away from me.
As long as all hardware components are working and there is connectivity to the machine, it doesn't matter whether the machine is a few miles away or a million miles away.
...would have been to have "fixed" the problem before the hardware left earth. This "bug" (or more accurately, known limitation of the filesystem) should have been discovered here on earth if the rover had been properly tested.
The only real bug was the inability of the system to properly handle running out of file entries (or more specifically, consuming too much RAM as the number of file entries increased). However the software should have never have stressed the filesystem to that degree in the first place.
Dan East
Better known as 318230.
Granted mainstream media have to keep their coverage dumbed down if Joe Public are going to read it. But what really bugs me is the lack of follow-up. We hear about poorly understood events as they are unfolding, then never heard about them later when they are completely understood.
A recent example is the gangway between ship and shore at the QM2's drydock. It collapsed killing lots of people, an investigation was launched. Why did it collapse? At the time it wasn't known. I'm sure it's known now, but there's been absolutely no followup.
This article about the rover is great not so much because of the level of detail but because it reports on an event with the benefit of hindsight.
Slashdot monitor for your Mozilla sidebar or Active Desktop.
No the didn't use SSH. However, a lucky hacker
would have to have access to a every large radio atennae, like the one atop a volcano in Hawaii.
I don't think they would bother using anything to do with TCP. Anything you do send you will have to wait 9 minutes for. Just imagine the ping times:
:(
Pinging mars-rover with 32 bytes of data:
request timed out
request timed out
request timed out
64 bytes from mars-rover: icmp_seq=0 ttl=64 time=32400ms
If it has anything to do with current internet protocols, it would be UDP.
What filesystem is used? Is wear leveling being used? The directory structure is apparently stored in RAM during the day (why else would it use so much RAM?), that is a good thing for reducing wear on the flash system. But what's the number of writes on the flash chips? When will that number be reached?
"It's too bad that stupidity isn't painful." - Anton LaVey
If you RTFA you will realize that I'm not lying in the least when I say that, effectively, they ran out of flash-based "disk" space! They forgot to delete old files when updating the programs in the flash memory (which is mounted like a filesystem, or hard disk), and the OS was failing because it wanted to use that space. So it rebooted, and still had insufficient disk space, and rebooted again . . . lather rinse repeat. There was no signal because it was stuck in a reboot loop because they ran out of disk. Wow.
:) ), then used low-level (direct access) flash utilities to remove the old files. Reboot, mount, disk check / corruption repair, voila it works again.
They fixed it by telling it to boot without using the flash (safe mode
We have a big 1TB NetApps server where I work, and we have so much disk space that people get lazy and don't delete files or archive old projects, then they get really confused when jobs fail, not thinking disk space until checking everything else first. But it happens, and it's usually surprisingly hard to debug (they check a lot of other things first, sometimes even upgrading tool versions!). It's really kinda funny, in an expensive and mildly embarassing way that the Spirit had the same problem.
everything in moderation
"The outcome strikes me as an extremely Lucky Hack..."
The outcome does not strike me as a "Lucky Hack." They made the system flexible, that flexibility got them into some trouble, and it's also what got them out of it. Anyone else agree?
Your post is the only thing that strikes me as a "Lucky Hack" here. They included the ability in the design to remotely disable booting from flash and upload new boot images, in what way is that a "hack"? All this is just foresight in design to include as many possible recovery modes as they could.
Basically, they rebooted from a recovery image (sent via radio) and then proceeded to do low-level fixes on Flash memory and they a chkdisk. If I do something similar via recovery disk or CD, I don't get a lot of people telling me that it was a "Lucky Hack" that I could boot off of CD!!!
"There is more worth loving than we have strength to love." - Brian Jay Stanley
One lesson we can learn from the Spirit problems that really and truly does directly apply on earth:
Just in case of a worst case scenario, always make sure you have physical access to the machines.
I can't believe that this is the state of the art at NASA - no wonder Shuttles fall from the sky.
I had pretty much the same post - the originator of the story confuses luck with skill, a mistake a find very annoying and committed all too frequently. I'll fully admit when I've been lucky, but I also went recognition for foresight when I've had some! NASA deserves at least that much respect.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Before doing something risky, type this:
:-)
:-)
sleep 600 && reboot &
Now if your risky maneuver makes the ssh session unusable, just wait 5 minutes for the machine to reboot.
This is great for fiddling with firewalls by remote control... through the firewall.
Oh... You say you're not using a POSIX-like system? That's not supported. Sorry.
It sounded like the same type questions non-technical bosses always ask about technical matters.
"We're ordering this brand new hardware that you've never tested before. Can you guarantee it will never crash?"
"Will this database server handle the load of our brand new project?" (without an accurate growth estimate)
"A server 2000 miles away just went down. What happened?" (no ping, no nothing) Hmmm.. Power/NIC/CPU/CPU fan/hard disks?
It really sounds like they did some decent advanced planning on those probes, but from other stories I read, the were shooting for 90 days of reliability, which in itself was a hard one to do. What if it turns the antenna the wrong way and looses connectivity? What if it gets hit by lightning? What if it falls in a hole? (go Beagle!)
Sure, relate this to your web server colocated somewhere you're not. Cross your fingers, hold your breath, and hope there aren't a few fatal systems failures, or a bit of human error. I've been responsible for a bit of that in the past, but at least my equipment wasn't a few million miles away.
Serious? Seriousness is well above my pay grade.
Using the low- level commands, about a thousand files and their directories -- the leftovers from the initial launch load -- were removed.
I think that means they deleted the useless stuff they wanted to delete anyways but didn't get to delete before the crash. I also remember news about science data from before the crash that was received after they got the rover working again.
As for how critical it is, well yeah, it seems the rover didn't need the contents of the flash file system. The operating system and other software was in the same flash memory but I assume that any sane designer would put in some hardware write protect interlock that's not easy to defeat accidentally.
You realize that missions to Mars can only be launched once every two years, right? If they miss their launch window, they've got to wait two years before they can launch again.
You also realize that NASA did do a test mission, right? They built a test rover and put it out in a desert somewhere. They used the mission to test the hardware, test the software, and to help train the team.
And what are the specifications of your network connection? Those *are* measured in base 10: 1 megabit/second = 1000 kilobit/second = 1000000 bit/second.
If there was any revisionist crap here, it was the defining of M, G, and so on to be used in base 2 (1024) in the first place. Those are standardized units!
UDP would be even worse. Interplanetary transmission is difficult, so some packet loss is likely. Under UDP the packets would just disappear-it's an unreliable protocol. TCP would of course be too inefficient. I'd expect them to use a custom protocol designed for the specific application, since their situation is totally unlike anything you'll face on Earth.
WindRiver may give JPL large discounts, but I doubt that's the only reason VxWorks is running on the MERs.
Years ago, when JPL was designing the Mars Pathfinder mission, they asked Wind River to do an "affordable" port of VxWorks to the RAD6000 (a radiation-hardened RS6000), and they agreed. Since the computers on the two MERs are very similar to the computer on the Mars Pathfinder lander, it makes sense that they'd use the same OS that they used on the MPF lander.
I would think the fact that JPL knows VxWorks very well by now would be a major factor in deciding to use VxWorks for the MERs.
I believe the software was in fact working exactly as expected here so no amount of formal verification would have helped.
Perhaps with hindsight it might have worked a little differently but you can't forsee every combination of events and in fact the software worked flawlessly allowing them to recover in exactly the way they designed it.
If this had been sitting on a desk next to them they probably would have sorted this out in 20 minutes but they obviously they needed to do this in a methodical and careful way to verify what they thought this problem was and they needed to do this over a very slow and very delayed link which was being affected by the problem. Which is no doubt why it took a few days.
Could this have not been said more succinctly with a simple quote? Namely:
"What we have here, is failure to communicate."
Mal-2
How is the Riemann zeta function like Trump rallies? Both have an endless number of trivial zeros.
"The irony of it was that the operating system was doing exactly what we'd told it to do"
Funny, that's how it was explained to me by my computer science teacher my freshman year in high school. He said, "The problem with computers is that they do exactly what we tell them to."
I belong to the ______ generation.
allocation errors are the easiest to predict. even if you don't handle them gracefully (which often can be near to impossible), most of the time you can log them. of course, a reliable, redundant log facility is one the most crucial components of such a system...
writting this from my armchair, of course i can't really judge their competence and claim i could have done better. still, the article makes me suspicious.Seriously, from a developer viewpoint, that is all wrong.
I have worked on projects in which there was simply too much logging going on that you couldn't tell head from toe anymore. When a problem arrived, scanning the logfiles proved very cumbersome indeed. Every developer had his own stuff logged, which sometimes proved interesting, sometimes proved utter crap (noone wants to know variable XYZ is increased by 1 for 24943 times).
You should develop a well-thought logging strategy that increases the logging verbosity on a problem-basis, not simply log everything that happens and hoping you get some useful information.
Slashdot: stuff for news, nerds that matter, matter for news, stuff that nerd
> What on earth (or on Mars) could we possibly take away from this experience?
Rule 3: Never ignore the return value from open.