Slashdot Mirror
Mars Rovers Have Incorrect Instruments Installed
Christopher Reimer writes "The New Scientist is reporting that the twin Mars rovers, Opportunity and Spirit, has instruments installed in the wrong rovers. From the article: 'While the bungle does not undermine the main scientific conclusions drawn from the data collected by the rovers, it is an embarrassing slip-up for a space agency that once lost a Mars spacecraft because engineers mixed up metric and imperial units.'"
Who knew being a rocket scientist was so tough.
Twin Mars rovers, Opportunity and Spirit landed on the Moon.
They instruments were installed correctly on Earth. It's the Martians that switched them as a prank. :-)
It annoys me that so much is made of this problem. This in no way compares to the lost spacecraft error, it's simply a calibration adjustment to a sensor. I think the fact that they have two rovers that have performed extremely well under harsh conditions 4x over their rated life is an incredible accomplishment. This just sounds like someone looking for sensationalism in a non-issue.
It's not intentional. Building something exactly to spec is impossible. The sensors are not identical for this reason. Not really a problem when you do a proper calibration.
Of course, it becomes a problem when you use the wrong calibration curve for the sensor.
Only in a Slashdot fantasy can a Slackware install turn into several hours of sex . . . . .
To clarify the summary: it's not that the WRONG instruments were installed, but that the SAME instruments were installed but calibrated for the OPPOSITE rovers. So, the data have been slightly off in a predictable way. In the end, it's not too surprising nor is it devastating. The data is still valid and is being readjusted.
Why is it hard to support them when they're in the middle of a hugely successful Mars mission?
No one outside the community even noticed this until recently, and in the end it really made no difference. So where's the beef?
They're the same device on each machine, with the same function. The only problem has been that the data received has been interpreted with the wrong calibration adjustments. Swap the calibration adjustments and rerun the data, and it'll be correct.
It would have been far worse if, say, one had a spectroscope and the other had a *drill*, and they were swapped, and each rover couldn't use the other's tool. And in that kind of a switch, it would be really bad, because the two devices would be visually distinct. But the swapping of two devices that are 99.99% identical, on two rovers that are identical, is no big thing.
Compared to the fact that the rovers are still running long after they were expected to die, this is a tiny, tiny thing.
September 2011: Looking for Cocoa/iOS work in Boston area Cocoa Programmer Quincy, MA
At least they landed 'em on the right planet.
OK, now what?
While the lead scientist says that it wasn't a big deal and no investigation will be held, I think he isn't analyzing the significance of this event. While scientists are more focused on the validity of data, engineers have to analyze not just events that occur (like loss of a rover), but also events that could occur. Putting the wrong instrument into a rover is due to "failure to follow procedure". This is a big deal. Failure to follow procedures could have been caught by a better QA system, better monitoring of the installation, and better training (including walkthroughs on the installation of the instruments).
Even though this minor event that has had no impact on the mission, it has shown that there are holes in JPL's QA system, their monitoring system, and their training program for building these rovers. If you want to dig further you might find that all of these problems were caused by an unnecessary sense of urgency which may have been caused by poor project planning. These exact problems have caused the loss of spacecraft before (and many of them were cited for the loss of Challenger and Columbia).
No investigation? The lead scientist really needs to take a look at his project management priorities. Having experience working in nuclear power I have learned and have been trained that small problems are many times the only symptoms of much larger problems. The lead scientist's attitude on the problem gives me no confidence in his ability to run a more complicated mission. Like in gambling, one or two successes doesn't mean that you are going to win on the next roll.
Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
Let the New Scientist criticize from the cheap seats. It is hard to argue that the rovers have been anything other than a resounding success for over 400 days. I would have hoped /. would
instead print the recent story of the Spirit Rover
discovering
salty soil.
an ill wind that blows no good
The sensors are built and configured the same, and the raw data they collect is sent back to earth.
However, nothing is perfect, and each sensor has slight imperfections. Before they were sent up, each sensor was measured so that those imperfections could be accounted for. This calibration data is unique to each sensor. They used the calibration data for Spirit on the data from Opportunity, and vice versa. Luckily, since they still have the original(un-corrected, raw) data, it is easy to correct.
one word: Tolerances.
when you have 100 resistors at 0.5% tolerance, you are gonna have drift, and you will have to calibrate them to the right parts, you can make 100's of those cars of the same make and model, but none will be exactly the same, especially when you have sensitive equipment. (Think odometer)
>once lost a Mars spacecraft because engineers mixed up metric and imperial units.
I'm getting pretty tired of this sound (text?) bite the media throws out. It wasn't mixed up units; it was error accumulation from switching back and forth between the units.
Identical parts are not exact duplicates. Go to Radio Shack and buy some resistors, all marked for the same value. Take them home and measure their actual values with a multi-meter. You will find that the measured values are scattered over a range that is centered on the marked value of the resistors. That's why each resistor has a tolerance specification. For example, a resistor may be marked 47 ohms, plus or minus 5%. The value of the resistor is guaranteed to lie within that range. It isn't guaranteed to be 47 ohms. The same thing applies to capacitors, transistors, and other parts. Circuits built from these parts inherit some of the variability of their component parts.
Mea navis aericumbens anguillis abundat
So, let me get this straight: NASA has managed to successfully send two completely functional rovers to the planet Mars 45 million miles away. Since they have arrived, the two rovers have expanded our understanding of the planet greatly and have had few and mostly correctable errors. They are now way, way past their expected mission time and are still running, and a few people have the nerve around to here to bash NASA for their horrible, numerous mistakes?
This stuff isn't easy. Just because you reap the benefits of the entire space program from your living room couch via the TV without actually contributing one bit does not mean you have any understanding of how complex and spectacular these great accomplishments are.
To the NASA / JPL engineers and scientists: Thanks.
They don't need to upload the files to the rovers because they're getting raw data here on Earth. They simply need to switch the calibration data in the software.
This is not a major thing. Yes, all the data collected from these sensors will have to be re-analyzed, but that should be a simple thing.
Damn your Dumb! Read before posting dumb things. No two things are exactly the same even if designed as such. And yes you can get in two identical cars and one will perform different.
It's a calibration, the whole concept is no two things are the same. Any piece of instrumentation needs to be calibrated and that calibration is set for that device. No manufacturing can produce 2 identical things, just not possible since the two items could not share the same time and space in the universe together thus both are going to be slightly different.
Anytime you get a piece of gear, you get it setup, then you take some means of calibrating it and test it with something that has deemed to be as accurate as possible. Maybe some source such as a rock. But basically anything that will provide a common test basis for the device.
So say you are measuring temperatures on something in a lab. You set up a big system for collecting data off a bunch of thermocouples. Each one has it's own channel through it's own voltage modules and thermocouples and so forth. So you take a calibrator and have it feed a signal through the system to mimic a thermocouple. you get a calibration curve for a channel, then you go to the next channel with the same device and do the same for the next channel, this will be a different calibration. and then you work through them all. I have system set up with 16 channels at work and all use the same parts, but there is about 4 closely similar calibrations across the channels, but no two channels follow the same calibration.
NASA did the same, they built the devices, then calibrated them with the same rocks, and developed a calibration curve for each system, and that was to be kept with each rover, they swapped the instruments, so now they switch the calibrations and everything is fine.
This all goes back to simple accuracy and how close you can get things, but bottom line no two things are the same. Look at computers, you can have 100 computers, exactly the same built right in a row, with the exact same software and so forth. Turn them on and let the run under exact same conditions, some will have hardware failures, some will have software get wacky on them and so forth. It's just the way it works.
RTFA. NASA swapped the instruments inadvertently but since both the rovers and the detectors are manufactured identically the only consequence was that we were using the calibration data file from one instrument to analyze the data coming from the other. Swap the calibration files so they are coupled to the correct instruments again, reanalyze the raw data, and the problem is solved without having to privatize NASA.
How can i possibly advocate for a mars mission when they can't even get this shit right?
The Mars mission is stupid but not for the reason you give.
A hermit writes: "The Church is reporting that the two human genders, male and female, have instruments installed in the wrong genders. From the article: 'While the bungle does not undermine the main reproductive conclusions from the reproductive activities between genders, it is an embarrassing slip-up for a supreme being that once lost a world of worshippers to a flood because the first prototypes mixed up good and evil.'"
Sometimes I wish I was a plumber, then I'd know how to deal with other people's shit.
In the old days, if you bought a carbon +/-10% resistor, you could be assured that it was either -10% to -5% or +5% to +10%, and almost never in-between. The reason? They'd mark the ones that fell between as +/-5% and sell them for more moeny.
This doesn't work for +/-5% and the next grade (+/-1%) because the parts are built differently.
What I don't understand is why this is a big deal.
It isn't a big deal. Instead of "Mars Rovers Have Incorrect Instruments Installed", a better headline would have been "Mars Rover Data Analyzed With Incorrect Calibration Data Files". But the editors would have rejected a headline like that.
It's true that the swap occurred when the instruments were installed. But it's really just a matter of semantics whether you consider the instruments to be swapped in the rovers on Mars, or their calibration files to be swapped in a computer's filesystem on Earth. Once the swap is discovered, it's over.
As a science teacher, I weep. For any instrument, it's important to perform calibration: to check the instrument against known samples, values, whatever, so that you can take the unique response of the instrument and convert it into a believable interpretation of the data. Every instrument has its own peculiarities, resulting from the (essentially unknowable) history of the construction of the instrument. Most of these features are entirely unimportant, if you know about them. So you run calibrations and figure out how to correct for the individual features.
NASA did its job here, in that the instruments were calibrated. Yay. Then they mixed up the instruments and installed package A into rover B, meaning the calibrations were in fact wrong. Luckily they keep all the raw data, so they can simply run it through the correct calibration filter now. Double yay.
But for all those saying "This is a small thing.": Wrong. They mixed up an entire package. Didn't it occur to anyone to actually, you know, label the two? Or to in fact make sure they weren't in the same lab at the same time? Or if that proved impossible, to keep track of which was which? Or to -- oh, I don't know -- check which package they were installing?
Excusing this as "just a minor thing" is akin to minimizng a case where you fall asleep while driving and are awakened by the rumble strips on the side of the road. Sure, you fell asleep. But you woke up and no one was hurt. No harm, no foul, right?
A minor screw up on its own, it still speaks volumes about NASA's continuing inability to cross all the t's and dot all the i's. And it's a pretty close relative to the error that cost us Mars Observer.
The Mongrel Dogs Who Teach
Small problems lead to medium sized problems which lead to big problems. Example: In the 1970's the NRC was similar to the Department of Transportation or FAA (pre 9/11) in that their job was to help facilitate the nuclear economy, not to beat down offenders. In the early 70's plant managers at a nuclear power plant in Alabamba, Browns Ferry Nuclear Power Plant, received reports that their insulation connecting to a cable room was not in accordance with fire specifications (small problem). Since this was not a significant problem, managers ignored it. Later workers testing the air-tightness of the room failed to follow the correct procedures by using candles to check the air tightness (if the flame is deflected, air is moving in that direction--small problem). Managers were aware but dismissed the problem. During testing for air leaks the flame of a candle was sucked into insulation and a fire erupted. The cable run that caught on fire was non-redundant and carried all of the control features for two nuclear reactors. Control of the reactors was lost and reactor safety was severly compromised. Problems that occured included that the operators of the reactors did not know how to properly respond to this casuality (including attempts to put out a large class A fire with portable CO2 extinguishers). Over $100 million in damages occured, but the reactors narrowly escaped tragedy (medium sized problem). This occured in 1975 and the NRC mostly covered up the problem. No congressional hearing were held. No significant corrective actions were issued and review of the ability of the operators to fight a casuality at a nuclear power plant was not reviewed. Fast forward four years and we arrive at Three Mile Island (big problem), where many of the shortcomings of the Brown's Ferry Plant and of the NRC being able to regulate and control the nuclear industry were exposed.
The lesson to learn here: if small problems exist, dig at them to see how far you can get and then fix *all* of the problems that you uncover. There are many other examples (including the 9/11 incident) but I think the point is obvious: there are problems at JPL that are not being looked at because *nothing* happened. They should be examined and corrected prior to a medium or large problem occuring.
Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
This was a follow up mission to the Viking landers which found no signs of life on Earth.
Had either of the Mars Rovers crashed or broken in some way, this mistake would never have been discovered. With only 1 rover's data, there would be no mysterious discrepency to solve and this mistake would have never been resolved.
So scientists would have spent the next 10 years developing their theories of martian geology based on incorrect data if either one of those rovers hadn't deployed and you call this a minor issue?!
This kind of error is inexcusable. But of course, it'll get brushed over because NASA was lucky enough to be in a position to fix it.
Okay, I was in the meeting where the difference in calibration was discussed, and I was the one that suggested that the instrument packages should be marked so that the right package would be installed in the right lander.
I recommended that one package should be marked with an "O" for "Spirit" and the other with an "S" for "Opportunity". I even donated the Sharpie marker and masking tape for this purpose.
It's not my fault that the implementation was screwed up. It's those numbnuts in the Vehicle Assembly Department who can't read a bloody memo.
Fortunately, I've left NASA for a position at the Department of Defense. My team is tasked with identifying sites related to the constructon of weapons of mass destrucion in South Korea.
k.
"In spite of everything, I still believe that people are really good at heart." - Anne Frank
No, small problems lead to no serious consequences. That's why they're called small problems. If they can lead to serious consequences then by definition they are not small problems. The magnitude of the problem is determined by the worst case scenario (Murphy's Law being what it is and all). Let's look at your example:
In the early 70's plant managers at a nuclear power plant in Alabamba, Browns Ferry Nuclear Power Plant, received reports that their insulation connecting to a cable room was not in accordance with fire specifications (small problem).
What is the worst case scenario if there should be a fire and the cables fail? If this is the cabling to the coffee pot, not much (small problem). If this is the cabling to the non-redundant control features of the nuclear reactor then this is a BIG problem and should have been treated as such.
Later workers testing the air-tightness of the room failed to follow the correct procedures by using candles to check the air tightness
What is the worst case of using this alternate procedure? In this case, there is an increased likelihood of fire. Even if the cabling was not faulty ANY fire is bad, so this should have been flagged as a BIG problem as well.
Both of these should have been recognized as big problems and not ignored. The fault is not that small problems were ignored, it was that they were not properly classified and prioritized. It sounds like there may have been many other problems as well, but they are not related to your main point.
The lesson to learn here: if small problems exist, dig at them to see how far you can get and then fix *all* of the problems that you uncover.
This sounds very profound but it is a fallacy. The lesson to learn from your example is to properly classify and prioritize potential problems. It is a major waste of time and effort to address every single tiny problem which creeps up, especially in highly complex systems it is close to impossible. There are only a limited amount of resources available. You must prioritize the truly important vs the trivial or you will never accomplish anything. BTW, way to pull out the nuclear bogeyman to help make your case.
Of course, this really has nothing to do with the NASA screw up since it really is a small problem. I doubt that the sensors were really that far off to begin with, and now that the problem has been discovered it can be 100% fixed with no loss of data. No harm no foul. Problems like this will continue to happen because everything NASA builds is a prototype! These are not mass produced items. When you build something (or write code) for the first time, is it perfect? I am also suspect of your conclusion that this problem indicates that "there are problems at JPL that are not being looked at." There may very well be problems in the bureaucracy, however this problem is indicative of nothing more than "shit happens." Of course, don't let this get in the way of a good NASA/JPL bashing.
When you lose something irreplaceable, you don't mourn for the thing you lost, you mourn for yourself. - Harpo Marx
"On September 27, 1999, the operations navigation team consulted with the spacecraft engineers to discuss navigation discrepancies regarding velocity change (V) modeling issues. On September 29, 1999, it was discovered that the small forces V's reported by the spacecraft engineers for use in orbit determination solutions was low by a factor of 4.45 (1 pound force=4.45 Newtons) because the impulse bit data contained in the AMD file was delivered in lb-sec instead of the specified and expected units of Newton-sec."
My journal. Mainly about freedom.