Slashdot Mirror
Mars Rovers Have Incorrect Instruments Installed
Christopher Reimer writes "The New Scientist is reporting that the twin Mars rovers, Opportunity and Spirit, has instruments installed in the wrong rovers. From the article: 'While the bungle does not undermine the main scientific conclusions drawn from the data collected by the rovers, it is an embarrassing slip-up for a space agency that once lost a Mars spacecraft because engineers mixed up metric and imperial units.'"
Who knew being a rocket scientist was so tough.
Twin Mars rovers, Opportunity and Spirit landed on the Moon.
They instruments were installed correctly on Earth. It's the Martians that switched them as a prank. :-)
To clarify the summary: it's not that the WRONG instruments were installed, but that the SAME instruments were installed but calibrated for the OPPOSITE rovers. So, the data have been slightly off in a predictable way. In the end, it's not too surprising nor is it devastating. The data is still valid and is being readjusted.
At least they landed 'em on the right planet.
OK, now what?
While the lead scientist says that it wasn't a big deal and no investigation will be held, I think he isn't analyzing the significance of this event. While scientists are more focused on the validity of data, engineers have to analyze not just events that occur (like loss of a rover), but also events that could occur. Putting the wrong instrument into a rover is due to "failure to follow procedure". This is a big deal. Failure to follow procedures could have been caught by a better QA system, better monitoring of the installation, and better training (including walkthroughs on the installation of the instruments).
Even though this minor event that has had no impact on the mission, it has shown that there are holes in JPL's QA system, their monitoring system, and their training program for building these rovers. If you want to dig further you might find that all of these problems were caused by an unnecessary sense of urgency which may have been caused by poor project planning. These exact problems have caused the loss of spacecraft before (and many of them were cited for the loss of Challenger and Columbia).
No investigation? The lead scientist really needs to take a look at his project management priorities. Having experience working in nuclear power I have learned and have been trained that small problems are many times the only symptoms of much larger problems. The lead scientist's attitude on the problem gives me no confidence in his ability to run a more complicated mission. Like in gambling, one or two successes doesn't mean that you are going to win on the next roll.
Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
The sensors are built and configured the same, and the raw data they collect is sent back to earth.
However, nothing is perfect, and each sensor has slight imperfections. Before they were sent up, each sensor was measured so that those imperfections could be accounted for. This calibration data is unique to each sensor. They used the calibration data for Spirit on the data from Opportunity, and vice versa. Luckily, since they still have the original(un-corrected, raw) data, it is easy to correct.
one word: Tolerances.
when you have 100 resistors at 0.5% tolerance, you are gonna have drift, and you will have to calibrate them to the right parts, you can make 100's of those cars of the same make and model, but none will be exactly the same, especially when you have sensitive equipment. (Think odometer)
Identical parts are not exact duplicates. Go to Radio Shack and buy some resistors, all marked for the same value. Take them home and measure their actual values with a multi-meter. You will find that the measured values are scattered over a range that is centered on the marked value of the resistors. That's why each resistor has a tolerance specification. For example, a resistor may be marked 47 ohms, plus or minus 5%. The value of the resistor is guaranteed to lie within that range. It isn't guaranteed to be 47 ohms. The same thing applies to capacitors, transistors, and other parts. Circuits built from these parts inherit some of the variability of their component parts.
Mea navis aericumbens anguillis abundat
So, let me get this straight: NASA has managed to successfully send two completely functional rovers to the planet Mars 45 million miles away. Since they have arrived, the two rovers have expanded our understanding of the planet greatly and have had few and mostly correctable errors. They are now way, way past their expected mission time and are still running, and a few people have the nerve around to here to bash NASA for their horrible, numerous mistakes?
This stuff isn't easy. Just because you reap the benefits of the entire space program from your living room couch via the TV without actually contributing one bit does not mean you have any understanding of how complex and spectacular these great accomplishments are.
To the NASA / JPL engineers and scientists: Thanks.
They don't need to upload the files to the rovers because they're getting raw data here on Earth. They simply need to switch the calibration data in the software.
This is not a major thing. Yes, all the data collected from these sensors will have to be re-analyzed, but that should be a simple thing.
In the old days, if you bought a carbon +/-10% resistor, you could be assured that it was either -10% to -5% or +5% to +10%, and almost never in-between. The reason? They'd mark the ones that fell between as +/-5% and sell them for more moeny.
This doesn't work for +/-5% and the next grade (+/-1%) because the parts are built differently.
"Compared to the fact that the rovers are still running long after they were expected to die, this is a tiny, tiny thing."
Except for the fact that the same organization that made this error is designing other spacecraft. If they don't get to the root causes of the problem, like the failure of the technicians to properly follow the correct procedure to install the instrument and the failure of any other engineer or management to catch their failure to follow procedure, much larger problems could occur. Lets examine a couple of JPL's problem's in the last couple of years:
Galileo: High power antenna failed to deploy resulting in a much lower data transfer rate. This was due to technical specifications in the lubrication of the antenna not being reviewed when the project was delayed.
Mars Climate Orbiter: Burned up because the technical requirements were not met (converting from BES to metric).
Mars Polar Lander: Lost on landing. Cause is not known. Project team was rushed in accordance with faster, better, cheaper plan.
Genesis: Failed to deploy parachute and crashed on landing due to technical requirements not being met (backwards specification for G-force meters).
Mars Exploration Rovers: Software glitch early in mission due to failure to test software for its entire expected lifespan. Instruments swapped due to failure to follow procedure.
Some things we can get out of this analysis are that the QA was unsatisfactory. Procedures were not followed. Technical specifications were not verified. The culture was rushed (go-fever or product push environment). None of these are small problems, but they also point to much bigger problems: failure of the leadership to properly plan the project so that rushed timelines would not occur. This same culture is building new spacecraft. While JPL is a great agency and they do tremendous and incredible feats, they are not perfect and have lost several spacecraft and have had severe faults in others. These problems did not have to occur and more importantly these problems do not have to occur again in the future.
Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
What I don't understand is why this is a big deal.
It isn't a big deal. Instead of "Mars Rovers Have Incorrect Instruments Installed", a better headline would have been "Mars Rover Data Analyzed With Incorrect Calibration Data Files". But the editors would have rejected a headline like that.
It's true that the swap occurred when the instruments were installed. But it's really just a matter of semantics whether you consider the instruments to be swapped in the rovers on Mars, or their calibration files to be swapped in a computer's filesystem on Earth. Once the swap is discovered, it's over.
Small problems lead to medium sized problems which lead to big problems. Example: In the 1970's the NRC was similar to the Department of Transportation or FAA (pre 9/11) in that their job was to help facilitate the nuclear economy, not to beat down offenders. In the early 70's plant managers at a nuclear power plant in Alabamba, Browns Ferry Nuclear Power Plant, received reports that their insulation connecting to a cable room was not in accordance with fire specifications (small problem). Since this was not a significant problem, managers ignored it. Later workers testing the air-tightness of the room failed to follow the correct procedures by using candles to check the air tightness (if the flame is deflected, air is moving in that direction--small problem). Managers were aware but dismissed the problem. During testing for air leaks the flame of a candle was sucked into insulation and a fire erupted. The cable run that caught on fire was non-redundant and carried all of the control features for two nuclear reactors. Control of the reactors was lost and reactor safety was severly compromised. Problems that occured included that the operators of the reactors did not know how to properly respond to this casuality (including attempts to put out a large class A fire with portable CO2 extinguishers). Over $100 million in damages occured, but the reactors narrowly escaped tragedy (medium sized problem). This occured in 1975 and the NRC mostly covered up the problem. No congressional hearing were held. No significant corrective actions were issued and review of the ability of the operators to fight a casuality at a nuclear power plant was not reviewed. Fast forward four years and we arrive at Three Mile Island (big problem), where many of the shortcomings of the Brown's Ferry Plant and of the NRC being able to regulate and control the nuclear industry were exposed.
The lesson to learn here: if small problems exist, dig at them to see how far you can get and then fix *all* of the problems that you uncover. There are many other examples (including the 9/11 incident) but I think the point is obvious: there are problems at JPL that are not being looked at because *nothing* happened. They should be examined and corrected prior to a medium or large problem occuring.
Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
This was a follow up mission to the Viking landers which found no signs of life on Earth.