Spirit Sends Debug Information to Earth

gfilion writes "NASA has released a press release that says: 'Shortly before noon, controllers were surprised to receive a relay of data from Spirit via the Mars Odyssey orbiter. Spirit sent 73 megabits at a rate of 128 kilobits per second.'" They've been having communications troubles with Spirit since Wednesday, so it's good to hear from it again, even if the data is just filler.

Comments

No BSOD Jokes, Please

[Anonymous Coward]

Forego the obvious

Re:No BSOD Jokes, Please

[Tablizer]

You know, it occurs to me that maybe instead of having an interactive rover with a billion complicated subsystems and spectrometers and cameras... it might be a good idea to launch a package full of smaller autonomous devices carrying different instrumentation... So you'd have a base that lands on mars, opens up (like the rover bases do) and releases 20 or 30 "dumb robots"

The two main problems I see with that is radio contact with the base and coordinated science. If a roverlet goes behind a hill it no longer has radio contact with the base. The atmosphere is probably too thin to rely on atmospheric bouncing of radio waves.

Second, if all the instruments are spread about, you don't get consistent science for any given rock or soil patch. It is better to look at the same object with different instruments than look at different objects with different instruments. But I suppose pairing could be done.

However, if there is a high failure rate of single-vehicle approaches, then roverlets starts to look inviting. But then again, what if the base lander fails? You still have a bottleneck. I suppose one of the roverlets could serve as the backup base, but at a lower bandwidth.

Interesting to think about it.

Re:ISDN to mars

[tomstdenis]

Most likely it's not a protocol that involves a lot of ACK'ing [e.g. huge packets with FECs]

Tom

Someone was thinking ahead

[Crudely_Indecent]

It's nice to know that NASA engineers threw debugging code in the mix. Otherwise, we'd have a $410,000,000 junkyard on the red planet.

I don't know what I'd do if I didn't get to see high resolution pictures of dirt and rock every day.

got some useful data

[cheezus]

The transmission included power subsystem engineering data, no science data, and several frames of "fill data." Fill data are sets of intentionally random numbers that do not provide information.

They don't say why it's sending fill data, but I bet the NASA geeks are happy about getting that engineering data.

If we could put a man on the moon with slide rulers, we should have no problem figuring out how to three-key a computer on another planet

Re:Can low-power corrupt memory?

[codepunk]

If it will not go to sleep at night it suggests to me that they have a serious hardware / software design flaw. They probably rely on software to initiate a standby vs alive mode. A proper design in this case would be to use standard analog circuits to do this type of job. Think about it you do not have to go out everynight and reboot your street light pole. Now of course this is pure speculation as IANANE
but then again maybe I should be.

400 million and only one CPU

[codepunk]

I saw over at the windriver site that this thing
has a proprietary os and only on cpu and only one set of code. Now come on how frigging smart is that? Hell back at work I even have redundant clusters for nearly everything. Relying on a single computer that is a few hundred million miles away is, should I dare say? Retarded..

Re:400 million and only one CPU

[Hollinger]

Well, you know, what's interesting about that is:
1. you'd have to increase the complexity of the device even more, exposing it to a higher risk of failure statistically
2. you'd need more complicated software and hardware that would require more time and effort (money & delays)
3. the hardware would need more power (limited batteries and solar panel capacity)
4. the system would be heavier and bigger (costs are measured in grams, iirc).

While you have a valid point, the constraints of this design give very strong tradeoffs among safety, feasability, and cash flow (and I'm sure there are others, but I'm not a rocket scientist). I'd imagine that some time was spent on redundant systems, but the adage of "Why have one when you can have two at twice the price?" only works when your budget can support the extra price of man-hours and cash.

I'd argue that where you work has unlimited available power, and if you need more, you can ask your power company for more. You have the money to spend on a X-thousand-dollar sever that's been pre-fabbed by whatever company you like. If you need more, you get more drop-shipped to you within days. NASA had to build these little buggers from the ground up.

<RANT>
You know, if you take your philosophy of simply duplicating the entire machine, there is a backup. It's called "Opportunity." It lands tomorrow.

I highly resent the fact that you've called some of the greatest engineers of our time "retarded." If you can't understand the problem (I certainly don't, but I do understand the concept of tradeoffs in design) you have no right to speak on the issue. Of course, this is slashdot. Everyone can mouth off about everything. Nevermind.
</RANT>

~MCH.

Re:400 million and only one CPU

[fermion]

It is a reliability issue. A second CPU is nice for redundancy, but integrated that second CPU complicates circuity. As things get more complicated, it gets more expensive to maintain the same level of reliability. The increase in expense in not linear. And since the reliability of the CPU is high, it is probably unlikely to be worth the expense when human life is not at risk.

Most offices now have to have redundant computers because the reliability of the machines are so low. This make economic sense for the office. Space travel is not the office. With space travel you buy the highly reliable machines and test the hell out of them to make sure they work. Even with all that they don't always work. But when you are doing something new not everything works.

Unfortunately kids today think Newton made his formulations the instant he got hit i the head. Explorations is about hard work and risks. And some guy in an office who has never done it has no idea of how complex it is.

Re:Wind river

[Anonymous Coward]

Hmmm, even they can do nothing against hardware errors... what this appears to be.
The good news is that this software appears to handle a hardware error situation gracefully, as it should. Bragging time may still be ahead.

should NASA let Wind River write the code?

[polished look 2]

I'm wondering if this is a software glitch running on Spirit and if so this truly does call into question the competence of Wind River, the people that wrote the code in use inside of Spirit. Why doesn't NASA hire its own programmers instead of hiring another firm to write it for them?

Re:should NASA let Wind River write the code?

[Detritus]

If there is a COTS (commercial off-the-shelf) real-time operating system available that meets the system requirements, why go to the risk and expense of writing your own from scratch? Do you expect NASA to fabricate every component in the spacecraft?

Re:Can low-power corrupt memory?

[Reivec]

While I am sure the parent isn't at all involved in the project and is probably wildly off base, I think it is a very interesting observation. I mean the guys as NASA guess the same kind of stuff right? They just have the means to check it and rule it out (or not). I would have to say based on the limited info he has of the rover, that this isn't an all that unlikely guess as to the cause of problems.

And for all those people that say things like "Do you think the people at NASA are just stupid and wouldn't have thought of this in the design?" Well no, they are not stupid, but they are not perfect either. And they have most certainly overlooked some pretty stupid things that caused serious failures. I mean hell, they only need one bug to bring the whole thing to a halt, and it isn't like they can do real world testing beforehand, they can only simulate what it will be like.

Re:Connections

[Anonymous Coward]

they paid 400 million for that link. How much did you spend on yours?

Thought so.

Improving NASA: Get-it-right vs. get experience

[G4from128k]

Is it quite possible that NASA engineers simply have not mastered the art and science of designing hardware and software operable in the harshest of environments?

While I would never claim that NASA is perfect, I think you underestimate the both the engineering challenge of putting a rover on Mars and the impact of more conservative, get-it-right, policies.

Interplanetary missions are the hardest of all because the engineers never get to actually test the whole device under realistic conditions. Although they can test and analyze each subsystem under a variety of simulated or near-realistic conditions, they have no way of building a test rover, putting it in interplanetary space of months, having is aerobrake into a thin atmosphere, parachute in a thin atmosphere, and crashland at high speed, and then operate all its mechanical parts under dusty low G conditions.

Second, get-it-right == conservatism == greater cost == fewer missions == less experience. The last thing NASA should do is spend more money, take more time, and do fewer missions. The only way we will really learn how to operate in space is to go into space. I'm not saying that better engineering won't help, only that more experience (unfettered by excessive conservatism) is a crucial part of learning to operate on other planets.

Re:Improving NASA: Get-it-right vs. get experience

[Jeremi]

given the vastness of time in a cosmological sense, shouldn't NASA be considering 100-year or 1000-year timetables?

That would be ideal, but keep in mind that NASA is funded by Congress, an entity that changes its mind about everything every 2-8 years. Any NASA program that takes too long to complete is very likely to be cancelled halfway through, wasting 100% of the resources that were put into it.

Re:Improving NASA: Get-it-right vs. get experience

[AllUsernamesAreGone]

It seems ludicrous to me that NASA is on a 15-year time table...given the vastness of time in a cosmological sense, shouldn't NASA be considering 100-year or 1000-year timetables?

Unfortunately, if you want to look at things on the scale of cosmological time, we don't even exist. Human beings have been around for a blink of the eye of the universe, and unless we get our backsides off this damp ball of rock as soon as possible there's every chance that within another blink, we won't exist anymore. Between climate change (not even human-caused - the "comfortable" Earth we know is just a fleeting hospitible break between the planet's normal fire and ice), potential self-destruction, impact events and a dozen other risks, our continued persistence in keeping all our eggs in one basket is nothing short of asking for annihilation. How many other "intelligent" species would sit there and watch as enough rock and ice to wipe out life plunges into a planet that is, comparitively, just next door and do nothing? We did when Shoemaker-Levy 9 hit Jupiter. My only comfort is that, should the human race be wiped out while confined to Earth by its own lack of vision and sense, it'll be a service to galactic evolution.

Re:Improving NASA: Get-it-right vs. get experience

[Anonymous Coward]

Beautifully put my dear fellow.

To those that argue "Lets get things right here on earth before we adventure to the great beyond" - I answer that this is extremely short sighted thinking. Things will never be right here on earth. There will always be wars, famine, ecological disasters, pestilence, terrorism, crime, religious strife, etc. As we outgrow our comfortable little womb most of these problems will only intensify.

The purpose of intellegent life, (i.e. the self-aware universe) is to extend, propagate and further intelligent life into the vacant lifeless universe.

Re:Can low-power corrupt memory?

[Anonymous Coward]

You look much worse chiding someone over what was, at most, an unimportant part of the post.

sure would be nice..

[maelstrom]

To have some actual technical discussion on a site that is supposed to be filled with nerds, instead of the same tired jokes about martians.

Re:Spirit rebooting 60 times a day

[Basehart]

If there was another agency out there putting machines on Mars, able to perform flawlessly for extended periods of time, and the NASA machines were the only ones crapping out, then I'd agree there needs to be some serious analysis of why NASA isn't getting it right.

But this just isn't the case.

From what I can tell NASA is doing as good a job as anyone on Earth with the technologies, manufacturing processes and testing programs available to them.

I would hope that NASA be the first ones to run a diagnostic on themselves when problems occur, but the first order of business is to figure out what went wrong with Rover on Wednesday and make sure it doesn't happen again, which is what they are doing right now.

Bushes space program...

[PetoskeyGuy]

Maybe if Bush didn't invade Iraq, he could have given that 87 Billion to Nasa instead. In the mean time they have to do the best with what they have.

I agree it's wrong to just put NASA on a pedestal, but analyze their success as well as thier failures, and be sure to compare it to the other space agencies out there. I think they are doing a pretty incredible job accomplishing lots of things that have never been done before.

With that said, lets see how Opportunity does tonight!

Re:Bushes space program...

[cheezedawg]

Maybe if Bush didn't invade Iraq, he could have given that 87 Billion to Nasa instead.

Ok- on one hand, we can spent the money to free 25 million people from a brutal and oppressive dictator, give credibility to the UN, provide a catalyst for the democratization of one of the most volatile regions in the world, and eliminate a threat to our national security.

-OR-

We can hurl more crap up into space.

Thats a pretty tough choice.

Yes, but there are other possibilities

[dtmos]

Since Spirit is rebooting sixty times per day, a problem that started when an electric motor moving its spectrometer "conked out", one thinks first of a hardware failure, possibly leading to software corruption.

I don't know the boot sequence of Spirit, but in most battery-powered embedded systems with which I am familiar, an elaborate state machine design is made to ensure that, when the boot sequence is complete, the system has sufficient power to perform any task that may be requested of it. Since the power supply is limited, an unexpectedly heavy load on the primary supply could cause the supply voltage to the microcomputer to fall below its specified lower limit, leading to a system reset.

Now imagine that there is a hardware failure associated with some process that runs during the boot sequence--a voltage regulator turn-on, a heating system initialization, an electric motor activation, whatever--that results in excessive current drain. When this part of the boot sequence is reached, the supply voltage falls, and the microcomputer resets. This disables the problem-causing hardware, unloading the power supply. When the supply voltage recovers, the microcomputer reboots (either automatically, with a power-on reset, via a watchdog timer, or via some other means) and, when the critical part of the boot sequence is reached, the supply voltage falls again. The system is now in a continuous loop, in which it can remain indefinitely. (Or at least 60 times per day....)

Note that this situation can also arise due to a defect in the power supply--if the output impedance of the power supply has risen for some reason, its output voltage under lightly loaded conditions can be acceptable, but it may not be able to supply heavier loads.

One expects the Spirit power supply to be complex, with separate regulators for the microcomputer, radio transceiver, and electric motors, so looking for common circuits and systems would be the first thing to do when troubleshooting for this type of failure. Looking for system conditions that can cause a system reset would be another; the JPL people have lived with their systems for years now, and would have had many design reviews to identify possible system failure scenarios--I'm not telling them anything new here. I understand that the system telemetry received yesterday indicates that the power supply is within specification, so that seems to eliminate that possiblility.

The second alternative is a soft memory failure of some kind, either caused by a supply failure as the parent suggests or perhaps by a radiation event of some kind.

Note that these problems can be multi-disciplinary; for example, the problem could be caused by some vibration when a motor runs that loosens a broken connection created by a chemical reaction to something on the surface (to take an extreme example).

Re:Can low-power corrupt memory?

[Wolfrider]

grrroooooooooaaaannn.... +1 funny, reluctantly