Slashdot Mirror


Mars Rovers Have Incorrect Instruments Installed

Christopher Reimer writes "The New Scientist is reporting that the twin Mars rovers, Opportunity and Spirit, has instruments installed in the wrong rovers. From the article: 'While the bungle does not undermine the main scientific conclusions drawn from the data collected by the rovers, it is an embarrassing slip-up for a space agency that once lost a Mars spacecraft because engineers mixed up metric and imperial units.'"

69 of 294 comments (clear)

  1. Man by Anonymous Coward · · Score: 5, Funny

    Who knew being a rocket scientist was so tough.

    1. Re:Man by Mr.Progressive · · Score: 5, Funny

      More proof that NASA is filled with these guys.

      --
      Okay, so a philosopher, a philologist, and a philatelist walk into a bar...
  2. In Other News.... by TimeTraveler1884 · · Score: 5, Funny

    Twin Mars rovers, Opportunity and Spirit landed on the Moon.

  3. Puddnhead Wilson Goes to Mars by rkmath · · Score: 2, Funny

    Subject says it all - mod me down if you have never read Mark Twain :)

    1. Re:Puddnhead Wilson Goes to Mars by Kennric · · Score: 3, Informative

      Except, as you recall, Puddnhead Wilson was quite intelligent, and used the nascent science of fingerprinting to solve a complicated crime. His sardonic wit and odd intellectual habits led the uneducated people of the little town he moved to to assume he was a bit slow. Eventually, they concluded he wasn't just slow, he was a Puddnhead.

      From The Tragedy of Puddnhead Wilson:

      He was a homely, freckled, sandy-haired young fellow, with an intelligent blue eye that had frankness and comradeship in it and a covert twinkle of a pleasant sort. But for an unfortunate remark of his, he would no doubt have entered at once upon a successful career at Dawson's Landing. But he made his fatal remark the first day he spent in the village, and it "gaged" him. He had just made the acquaintance of a group of citizens when an invisible dog began to yelp and snarl and howl and make himself very comprehensively disagreeable, whereupon young Wilson said, much as one who is thinking aloud:

      "I wish I owned half of that dog."

      "Why?" somebody asked.

      "Because I would kill my half."

      The group searched his face with curiosity, with anxiety even, but found no light there, no expression that they could read. They fell away from him as from something uncanny, and went into privacy to discuss him. One said:

      "'Pears to be a fool."

      "'Pears?" said another. "Is, I reckon you better say."

      "Said he wished he owned half of the dog, the idiot," said a third. "What did he reckon would become of the other half if he killed his half? Do you reckon he thought it would live?"

      "Why, he must have thought it, unless he IS the downrightest fool in the world; because if he hadn't thought it, he would have wanted to own the whole dog, knowing that if he killed his half and the other half died, he would be responsible for that half just the same as if he had killed that half instead of his own. Don't it look that way to you, gents?"

      "Yes, it does. If he owned one half of the general dog, it would be so; if he owned one end of the dog and another person owned the other end, it would be so, just the same; particularly in the first case, because if you kill one half of a general dog, there ain't any man that can tell whose half it was; but if he owned one end of the dog, maybe he could kill his end of it and -- "

      "No, he couldn't either; he couldn't and not be responsible if the other end died, which it would. In my opinion that man ain't in his right mind."

      "In my opinion he hain't got any mind."

      No. 3 said: "Well, he's a lummox, anyway."

      That's what he is;" said No. 4. "He's a labrick -- just a Simon-pure labrick, if there was one."

      "Yes, sir, he's a dam fool. That's the way I put him up," said No. 5. "Anybody can think different that wants to, but those are my sentiments."

      "I'm with you, gentlemen," said No. 6. "Perfect jackass -- yes, and it ain't going too far to say he is a pudd'nhead. If he ain't a pudd'nhead, I ain't no judge, that's all."

  4. Confusion...Why differing configurations? by Alpha_Traveller · · Score: 3, Interesting
    Creeping errors
    Although their designs are identical, each instrument is unique because of quirks in the materials they are made from. So before the rovers were launched, each instrument was calibrated using known rock samples. The measurements from each rover are then processed using the calibration files, but because of the mix-up, researchers were using the wrong ones. As a result, small errors have crept into the APXS results, affecting measurements of sodium, magnesium and aluminium abundance.
    Perhaps someone can clarify this statement? (Since I am not an engineer, eh?)... I'm wondering why it's so important to have differing configurations for the sensors in the first place. Wouldn't it be wise to collect exactly the same kind of information regardless of how complex if it's all being sent via transmission back to us anyway? Wouldn't it be extremely important to have the exact same configuration on BOTH sensors? I would think the end result would be useful when comparing the chemical composition of any particular area. It would be like hacking off a sensor "at the knees" when you had no significant reason to do so wouldn't it?
    --
    "Love is like pi - natural, irrational, and very important." (Lisa Hoffman)
    1. Re:Confusion...Why differing configurations? by theparanoidcynic · · Score: 4, Informative

      It's not intentional. Building something exactly to spec is impossible. The sensors are not identical for this reason. Not really a problem when you do a proper calibration.

      Of course, it becomes a problem when you use the wrong calibration curve for the sensor.

      --
      Only in a Slashdot fantasy can a Slackware install turn into several hours of sex . . . . .
    2. Re:Confusion...Why differing configurations? by araemo · · Score: 5, Informative

      The sensors are built and configured the same, and the raw data they collect is sent back to earth.

      However, nothing is perfect, and each sensor has slight imperfections. Before they were sent up, each sensor was measured so that those imperfections could be accounted for. This calibration data is unique to each sensor. They used the calibration data for Spirit on the data from Opportunity, and vice versa. Luckily, since they still have the original(un-corrected, raw) data, it is easy to correct.

    3. Re:Confusion...Why differing configurations? by Detritus · · Score: 5, Informative

      Identical parts are not exact duplicates. Go to Radio Shack and buy some resistors, all marked for the same value. Take them home and measure their actual values with a multi-meter. You will find that the measured values are scattered over a range that is centered on the marked value of the resistors. That's why each resistor has a tolerance specification. For example, a resistor may be marked 47 ohms, plus or minus 5%. The value of the resistor is guaranteed to lie within that range. It isn't guaranteed to be 47 ohms. The same thing applies to capacitors, transistors, and other parts. Circuits built from these parts inherit some of the variability of their component parts.

      --
      Mea navis aericumbens anguillis abundat
    4. Re:Confusion...Why differing configurations? by Anonymous Coward · · Score: 5, Informative

      They don't need to upload the files to the rovers because they're getting raw data here on Earth. They simply need to switch the calibration data in the software.

      This is not a major thing. Yes, all the data collected from these sensors will have to be re-analyzed, but that should be a simple thing.

    5. Re:Confusion...Why differing configurations? by Anonymous Coward · · Score: 5, Interesting

      In the old days, if you bought a carbon +/-10% resistor, you could be assured that it was either -10% to -5% or +5% to +10%, and almost never in-between. The reason? They'd mark the ones that fell between as +/-5% and sell them for more moeny.

      This doesn't work for +/-5% and the next grade (+/-1%) because the parts are built differently.

    6. Re:Confusion...Why differing configurations? by MillionthMonkey · · Score: 5, Insightful

      What I don't understand is why this is a big deal.

      It isn't a big deal. Instead of "Mars Rovers Have Incorrect Instruments Installed", a better headline would have been "Mars Rover Data Analyzed With Incorrect Calibration Data Files". But the editors would have rejected a headline like that.

      It's true that the swap occurred when the instruments were installed. But it's really just a matter of semantics whether you consider the instruments to be swapped in the rovers on Mars, or their calibration files to be swapped in a computer's filesystem on Earth. Once the swap is discovered, it's over.

    7. Re:Confusion...Why differing configurations? by gilroy · · Score: 4, Informative
      Blockquoth the poster:

      I'm wondering why it's so important to have differing configurations for the sensors in the first place.

      As a science teacher, I weep. For any instrument, it's important to perform calibration: to check the instrument against known samples, values, whatever, so that you can take the unique response of the instrument and convert it into a believable interpretation of the data. Every instrument has its own peculiarities, resulting from the (essentially unknowable) history of the construction of the instrument. Most of these features are entirely unimportant, if you know about them. So you run calibrations and figure out how to correct for the individual features.

      NASA did its job here, in that the instruments were calibrated. Yay. Then they mixed up the instruments and installed package A into rover B, meaning the calibrations were in fact wrong. Luckily they keep all the raw data, so they can simply run it through the correct calibration filter now. Double yay.

      But for all those saying "This is a small thing.": Wrong. They mixed up an entire package. Didn't it occur to anyone to actually, you know, label the two? Or to in fact make sure they weren't in the same lab at the same time? Or if that proved impossible, to keep track of which was which? Or to -- oh, I don't know -- check which package they were installing?

      Excusing this as "just a minor thing" is akin to minimizng a case where you fall asleep while driving and are awakened by the rumble strips on the side of the road. Sure, you fell asleep. But you woke up and no one was hurt. No harm, no foul, right?

      A minor screw up on its own, it still speaks volumes about NASA's continuing inability to cross all the t's and dot all the i's. And it's a pretty close relative to the error that cost us Mars Observer.
    8. Re:Confusion...Why differing configurations? by Aglassis · · Score: 5, Insightful

      Small problems lead to medium sized problems which lead to big problems. Example: In the 1970's the NRC was similar to the Department of Transportation or FAA (pre 9/11) in that their job was to help facilitate the nuclear economy, not to beat down offenders. In the early 70's plant managers at a nuclear power plant in Alabamba, Browns Ferry Nuclear Power Plant, received reports that their insulation connecting to a cable room was not in accordance with fire specifications (small problem). Since this was not a significant problem, managers ignored it. Later workers testing the air-tightness of the room failed to follow the correct procedures by using candles to check the air tightness (if the flame is deflected, air is moving in that direction--small problem). Managers were aware but dismissed the problem. During testing for air leaks the flame of a candle was sucked into insulation and a fire erupted. The cable run that caught on fire was non-redundant and carried all of the control features for two nuclear reactors. Control of the reactors was lost and reactor safety was severly compromised. Problems that occured included that the operators of the reactors did not know how to properly respond to this casuality (including attempts to put out a large class A fire with portable CO2 extinguishers). Over $100 million in damages occured, but the reactors narrowly escaped tragedy (medium sized problem). This occured in 1975 and the NRC mostly covered up the problem. No congressional hearing were held. No significant corrective actions were issued and review of the ability of the operators to fight a casuality at a nuclear power plant was not reviewed. Fast forward four years and we arrive at Three Mile Island (big problem), where many of the shortcomings of the Brown's Ferry Plant and of the NRC being able to regulate and control the nuclear industry were exposed.

      The lesson to learn here: if small problems exist, dig at them to see how far you can get and then fix *all* of the problems that you uncover. There are many other examples (including the 9/11 incident) but I think the point is obvious: there are problems at JPL that are not being looked at because *nothing* happened. They should be examined and corrected prior to a medium or large problem occuring.

      --
      Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
    9. Re:Confusion...Why differing configurations? by Laur · · Score: 4, Insightful
      Small problems lead to medium sized problems which lead to big problems.

      No, small problems lead to no serious consequences. That's why they're called small problems. If they can lead to serious consequences then by definition they are not small problems. The magnitude of the problem is determined by the worst case scenario (Murphy's Law being what it is and all). Let's look at your example:

      In the early 70's plant managers at a nuclear power plant in Alabamba, Browns Ferry Nuclear Power Plant, received reports that their insulation connecting to a cable room was not in accordance with fire specifications (small problem).

      What is the worst case scenario if there should be a fire and the cables fail? If this is the cabling to the coffee pot, not much (small problem). If this is the cabling to the non-redundant control features of the nuclear reactor then this is a BIG problem and should have been treated as such.

      Later workers testing the air-tightness of the room failed to follow the correct procedures by using candles to check the air tightness

      What is the worst case of using this alternate procedure? In this case, there is an increased likelihood of fire. Even if the cabling was not faulty ANY fire is bad, so this should have been flagged as a BIG problem as well.

      Both of these should have been recognized as big problems and not ignored. The fault is not that small problems were ignored, it was that they were not properly classified and prioritized. It sounds like there may have been many other problems as well, but they are not related to your main point.

      The lesson to learn here: if small problems exist, dig at them to see how far you can get and then fix *all* of the problems that you uncover.

      This sounds very profound but it is a fallacy. The lesson to learn from your example is to properly classify and prioritize potential problems. It is a major waste of time and effort to address every single tiny problem which creeps up, especially in highly complex systems it is close to impossible. There are only a limited amount of resources available. You must prioritize the truly important vs the trivial or you will never accomplish anything. BTW, way to pull out the nuclear bogeyman to help make your case.

      Of course, this really has nothing to do with the NASA screw up since it really is a small problem. I doubt that the sensors were really that far off to begin with, and now that the problem has been discovered it can be 100% fixed with no loss of data. No harm no foul. Problems like this will continue to happen because everything NASA builds is a prototype! These are not mass produced items. When you build something (or write code) for the first time, is it perfect? I am also suspect of your conclusion that this problem indicates that "there are problems at JPL that are not being looked at." There may very well be problems in the bureaucracy, however this problem is indicative of nothing more than "shit happens." Of course, don't let this get in the way of a good NASA/JPL bashing.

      --
      When you lose something irreplaceable, you don't mourn for the thing you lost, you mourn for yourself. - Harpo Marx
    10. Re:Confusion...Why differing configurations? by Anonymous Coward · · Score: 2, Insightful
      Excusing this as "just a minor thing" is akin to minimizng a case where you...

      Oops, you just blew up a spacecraft with that spelling error. See how easy that was?

      Unless you've tried this you have no idea how hard it is. Try designing a flight program to make sure all the i's are dotted and t's are crossed, then having the budget slashed over and over again until you can barely manufacture, test, and launch. Then try the same thing in a three-shift environment that goes on for a couple of years, and make sure that not a single mistake happens.

      Please understand that a lot of time and money is spent testing for mission failure scenarios, and there isn't much left for the two-identical-instruments-switched scenarios. For all you know the Scientists missed, misread, or forgot the memo telling them the switch occurred.

      Didn't it occur to anyone to actually, you know, label the two?

      Of course it did, and they are. Do you know how many pieces originally intended for Spirit were installed on Opportunity because of schedule issues, and vice-versa?

      I did not have the honor of serving with the people that worked that mission, but I surely respect the sacrifices they made to make it happen, and the level of success they have achieved. If you think you can do better, feel free to send in a resume.
    11. Re:Confusion...Why differing configurations? by DavidTC · · Score: 2, Insightful
      They'd want them to be in the lab at the same time, along with the baseline measurer, to make sure they were measuring the same rock under the same conditions. They'd build a rig, put baseline sensor in rig, measure, put Spirit sensor in rig, measure, etc.

      What got me is that surely you'd calibrate it after putting it in the rover. You don't calibrate something, install it, and then test it, you install it, test it, and then calibrate it. (Then test the calibration.)

      So maybe they're confused, and the problem isn't that they swapped the instruments, it's that they saved the data under the wrong filenames or whatever. Someone tested the two, and then sat down and filed Spirit's test results under Opportunity, and vis versa.

      --
      If corporations are people, aren't stockholders guilty of slavery?
    12. Re:Confusion...Why differing configurations? by Laur · · Score: 3, Insightful
      While your logic works well for software development projects where noone can be killed if it fails, high-risk or high-value technologies cannot follow the same procedure, especially when they are of an integrated design (where many items can affect the operations of remote items).

      When did I say this primarily applies to software developement? I don't even work in software developement. Your point was that every problem, large or small, needs to be addressed with the same diligence. This is ridiculous and impossible. Problems must be categorized and prioritized. This applies to everything, including software, hardware, as well as high-risk high-value technologies. Actually more so in this case since the system is more complex. If you do not evaluate and prioritize nothing will get done.

      By your definition this isn't a problem because a reactor meltdown didn't occur.

      Nice strawman, when did I ever say anything like that? My exact words were "Both of these should have been recognized as big problems and not ignored." That seems to be the exact opposite of claiming that there wasn't a problem. My point was not that there was no problem (obviously there was), my point was that the problem was not what you said it was (a small problem being ignored leading to an almost catastrophe).

      I work as a reactor operator.

      And I assume you spend your days tracking down and solving every problem, no matter how trivial? If someone forgets to change the water in the coffee pot you track it down and fix it, because any small problem could lead to catastrophe, right?

      Wrong attitude.

      Sorry, that's reality, where things do not always go according to plan, no matter how carefully you plan or test.

      When you build something, you build it to specification, and you write procedures for it.... And then you test your product to verify that it meets the standards.

      Of course you do, however sometimes you don't foresee everything when you write the spec or procedures, or sometimes you have a new assembler or mechanic come on board and something is not done quite correctly and the tests don't catch it, or any of a hundred other things. I am a mechanical engineer working in aerospace and I can tell you that SHIT HAPPENS, no matter how much we wish it didn't, and no matter how many tests you plan or steps you take in order to make sure it doesn't. Engineers are not supermen, and the people who actually put it together and test it are human too.

      If it ever deviates, you carefully analyze the problem and fix it. If something breaks, you determine why it broke, because you might have a bigger problem.

      Of course you do, I never said you just ignore the problem. However, my point was that this is a prototype, some problems are expected. What if your tests don't catch the problem and it isn't discovered until it is out in the field? In my field we fix the part and sometimes retrofit the fix back to units in the field, however NASA only has one shot at this. If NASA manufactured hundreds of these rovers you can bet that they would become super reliable and all of these issues would be caught and fixed. However, it is simply impossible to catch all of the potential issues in the lab, and a one off prototype is going to has a few mistakes.

      You never say that "shit happens". "Shit happens" is just a codeword for "I'm too lazy to determine the real cause".

      I'm getting pretty sick of your strawman arguments and misrepresentation of my position. Once again, I never said the problem should be ignored, I just said that this is properly classified as a small problem. It does not impact the mission success, and in fact it can be corrected 100%. Obviously NASA should still investigate the cause and take steps to prevent it in the future, however to expect an experimental prototype to be perfect is ridiculous, and taking NASA to task for this error is equally ridiculous.

      --
      When you lose something irreplaceable, you don't mourn for the thing you lost, you mourn for yourself. - Harpo Marx
    13. Re:Confusion...Why differing configurations? by wjsteele · · Score: 3, Insightful

      I disagree - it is a minor thing. As this project had a finite development budget. A Risk Analysis (RA) was performed. RA would tell you that this instance, because the "data processing" was done back here on Terra, that it made no difference which instrument was installed in which rover. Risk Mitigation would also have pointed out that it's an easily correctable problem and therefore time/money shouldn't be spent verifying it. Time would be better spent on making sure that the instruments worked. There were many, many, many more important thing to make sure were right - like trajectories, timings, Radar altimeters, etc. That's where you focus on the "major things."

      Bill

      --
      It's my Sig and you can't have it. Mine! All Mine!
  5. nah... by Arctic+Dragon · · Score: 5, Funny

    They instruments were installed correctly on Earth. It's the Martians that switched them as a prank. :-)

  6. A very minor issue... by Roached · · Score: 4, Insightful

    It annoys me that so much is made of this problem. This in no way compares to the lost spacecraft error, it's simply a calibration adjustment to a sensor. I think the fact that they have two rovers that have performed extremely well under harsh conditions 4x over their rated life is an incredible accomplishment. This just sounds like someone looking for sensationalism in a non-issue.

    1. Re:A very minor issue... by supabeast! · · Score: 4, Insightful

      The reason people constantly point out problems like this is that they just keep happening in US Aerospace programs. Regardless of government agency, if you want to do anything with space in the US government, you're going to have to deal with once of a few contractors. These contractors have screwed up stuff like this dozens of times in the history of our space program, and the government keeps giving them more contracts, and never demands recompense for screwups that range from miscalibrations that can be dealt with to screwed up launch vehicles blowing up on the launchpad and taking payloads out in the process; in the former case it was an NRO satellite and the NRO hired the guy who had been running the program at Lockheed after Lockheed threw him out.

      When companies like Lockheed Martin finally have to start paying for all of their multi-billion dollar screwups in space, then this stuff will stop happening. Until then, people will continue to make a fuss because we're sick of a corrupt system allowing this crap to continue.

  7. Not wrong, but swapped by dcclark · · Score: 5, Informative

    To clarify the summary: it's not that the WRONG instruments were installed, but that the SAME instruments were installed but calibrated for the OPPOSITE rovers. So, the data have been slightly off in a predictable way. In the end, it's not too surprising nor is it devastating. The data is still valid and is being readjusted.

    1. Re:Not wrong, but swapped by ScrewMaster · · Score: 3, Funny

      They are like The Register for computers and The National Inquirer for, uhm. WTF is The Inquirer about?

      For that matter, WTF is Slashdot about?

      --
      The higher the technology, the sharper that two-edged sword.
  8. Re:I want to fight for NASA but come on... by kaiser423 · · Score: 4, Insightful

    Why is it hard to support them when they're in the middle of a hugely successful Mars mission?

    No one outside the community even noticed this until recently, and in the end it really made no difference. So where's the beef?

  9. Re:If... by Tethys_was_taken · · Score: 2, Interesting

    Even though the designs of the rovers are identical, the instruments themselves are not. The article says so. Each instrument was calibrated to behave properly in one rover. When the instruments were swapped, the readings from them were incorrect.

  10. Re:I want to fight for NASA but come on... by zulux · · Score: 3, Funny

    How can i possibly advocate for a mars mission when they can't even get this shit right?


    How can I possibly listen to you when you cant even used the shift key properly.

    --

    Moneyed corporations, non-working 'poor' and criminal prisoners are turning productive citizens into tax-slaves.

  11. No big deal... by SteeldrivingJon · · Score: 4, Informative


    They're the same device on each machine, with the same function. The only problem has been that the data received has been interpreted with the wrong calibration adjustments. Swap the calibration adjustments and rerun the data, and it'll be correct.

    It would have been far worse if, say, one had a spectroscope and the other had a *drill*, and they were swapped, and each rover couldn't use the other's tool. And in that kind of a switch, it would be really bad, because the two devices would be visually distinct. But the swapping of two devices that are 99.99% identical, on two rovers that are identical, is no big thing.

    Compared to the fact that the rovers are still running long after they were expected to die, this is a tiny, tiny thing.

    --
    September 2011: Looking for Cocoa/iOS work in Boston area Cocoa Programmer Quincy, MA
    1. Re:No big deal... by MBCook · · Score: 3, Insightful
      It would have been far worse if...

      See, that's what I thought happened at first. I assumed it was something like one had an X-Ray detector while the other had a mass-spectrometer or something (I would think NASA could tell the difference between a drill and a spectroscope). It was nothing of the sort, they got the calibration files mixed up between the rovers (technically the rovers mixed up between the calibration files, but it's the same end result).

      This isn't journalism, this is headline mongering. Especially throwing in that metric/imperal thing. This would be journalism if it was "NASA discovers error in rover calibrations, corrects data". Since they have all the raw data they just stick it back through the computer and it's like it never happened.

      Instead they try to make the public think NASA screwed up big again, like where one rover was supposed to have a camera and the other some kind of gas meter and they swapped 'em.

      You can argue about whether there is bias in the media (and whether it's liberal or conservative), but the BIGGEST problem is crap like this. Why report the good stuff ("US troops build new school in Iraq despite RPG fire") when you can report just the bad ("US troops attached by RPG fire"). The former spokesman (he was temporary, can't remember name or title) for the Bush administration recently said that this was what he thought was wrong with the media in this country first and foremost, and I agree. I just wish whoever submitted this to /. had found a less sensationalistic source to link to rather than promoting this kind of crap.

      --
      Comment forecast: Bits of genius surrounded by a sea of mediocrity.
    2. Re:No big deal... by Aglassis · · Score: 5, Interesting

      "Compared to the fact that the rovers are still running long after they were expected to die, this is a tiny, tiny thing."

      Except for the fact that the same organization that made this error is designing other spacecraft. If they don't get to the root causes of the problem, like the failure of the technicians to properly follow the correct procedure to install the instrument and the failure of any other engineer or management to catch their failure to follow procedure, much larger problems could occur. Lets examine a couple of JPL's problem's in the last couple of years:

      Galileo: High power antenna failed to deploy resulting in a much lower data transfer rate. This was due to technical specifications in the lubrication of the antenna not being reviewed when the project was delayed.
      Mars Climate Orbiter: Burned up because the technical requirements were not met (converting from BES to metric).
      Mars Polar Lander: Lost on landing. Cause is not known. Project team was rushed in accordance with faster, better, cheaper plan.
      Genesis: Failed to deploy parachute and crashed on landing due to technical requirements not being met (backwards specification for G-force meters).
      Mars Exploration Rovers: Software glitch early in mission due to failure to test software for its entire expected lifespan. Instruments swapped due to failure to follow procedure.

      Some things we can get out of this analysis are that the QA was unsatisfactory. Procedures were not followed. Technical specifications were not verified. The culture was rushed (go-fever or product push environment). None of these are small problems, but they also point to much bigger problems: failure of the leadership to properly plan the project so that rushed timelines would not occur. This same culture is building new spacecraft. While JPL is a great agency and they do tremendous and incredible feats, they are not perfect and have lost several spacecraft and have had severe faults in others. These problems did not have to occur and more importantly these problems do not have to occur again in the future.

      --
      Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
    3. Re:No big deal... by Anonymous Coward · · Score: 2, Interesting
      Mars Exploration Rovers: Software glitch early in mission due to failure to test software for its entire expected lifespan. Instruments swapped due to failure to follow procedure.

      The Trashy Fucking File System (TFFS) has bitten many projects, and the response from WindRiver is "We can't fix it." Which they can't. They bought TFFS from someone else. And they can't fix it.. for some reason. It's really a pain in the ass and the "loss of a flash file system" happens infrequently enough that it is possible for it to pass QA.

  12. Cain't never tell them kids apart... by mactov · · Score: 5, Funny

    At least they landed 'em on the right planet.

    --
    OK, now what?
    1. Re:Cain't never tell them kids apart... by flyingsquid · · Score: 4, Funny
      At least they landed 'em on the right planet.

      Actually, one of the rovers just crested a hill and beamed back this photo of a sign that says "Death Valley 7-11"...

  13. Root Cause by Aglassis · · Score: 5, Insightful

    While the lead scientist says that it wasn't a big deal and no investigation will be held, I think he isn't analyzing the significance of this event. While scientists are more focused on the validity of data, engineers have to analyze not just events that occur (like loss of a rover), but also events that could occur. Putting the wrong instrument into a rover is due to "failure to follow procedure". This is a big deal. Failure to follow procedures could have been caught by a better QA system, better monitoring of the installation, and better training (including walkthroughs on the installation of the instruments).

    Even though this minor event that has had no impact on the mission, it has shown that there are holes in JPL's QA system, their monitoring system, and their training program for building these rovers. If you want to dig further you might find that all of these problems were caused by an unnecessary sense of urgency which may have been caused by poor project planning. These exact problems have caused the loss of spacecraft before (and many of them were cited for the loss of Challenger and Columbia).

    No investigation? The lead scientist really needs to take a look at his project management priorities. Having experience working in nuclear power I have learned and have been trained that small problems are many times the only symptoms of much larger problems. The lead scientist's attitude on the problem gives me no confidence in his ability to run a more complicated mission. Like in gambling, one or two successes doesn't mean that you are going to win on the next roll.

    --
    Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
    1. Re:Root Cause by Detritus · · Score: 4, Insightful

      Get over it. The Apollo era NASA had a lot more money and people. They could afford to do things the right way, with multiple backup systems and extensive testing and QA. They still made mistakes, just not as many. You want cheap space exploration? You've got cheap space exploration. Don't bitch that they didn't deliver a Ferrari when you only paid for a Chevy.

      --
      Mea navis aericumbens anguillis abundat
    2. Re:Root Cause by mrchaotica · · Score: 2, Insightful

      Fair enough. But can I bitch that I paid for a Corvette, and ended up with a Chevette instead?

      --

      "[Regarding the 'cloud,'] ownership was what made America different than Russia." -- Woz

    3. Re:Root Cause by notmuchtosay · · Score: 2, Interesting

      Having taken a class from Squires as these rovers were being built, I know he was very aware of small problems being important. He explained how "small" problems caused failed missions in the past. Such as the previous mention of SI to imperial additionally not "testing as you fly."

      A more complicated mission? Landing two rovers utilizing air bags on another planet isn't complicated enough for you?

      He was the PI but he cannot be expected to observe everyone's work personally. This sort of thing should not need supervising.

      I agree this should have been caught with better testing of full rovers, but the launch windows cannot be merely pushed back. It was a time crunch the rovers and instruments performed well, the analysis was wrong.

    4. Re:Root Cause by Aglassis · · Score: 3, Insightful
      Squyres is quoted in the article as saying:

      "[He is] not embarrassed at all".

      "It was an easy mistake to make. It happened during some very busy and stressful times."

      The article says that he also says it is not fair to compare it to past mishaps because the spacecraft suffered no damage.

      "There isn't going to be an investigation. We know when it happened."

      He doesn't get it. The big problem here isn't that a technician goofed. The big problem is that noone caught it. The purpose of the investigation is not just to assign blame. It will also find the root causes like I previously described. While JPL will probably never have instruments swapped again, there will probably be another case where procedures aren't followed and noone notices.

      Squyres statements from the article state that he isn't embarrased that his QA program and monitoring programs were inadequate, that his team did not take adequate precautions during stressful times to make sure that procedures were followed, that it's not an incident because no damage occured (sort of like it's not speeding unless you get pulled over), and that he doesn't believe there are any larger problems from this incident.

      I'm not saying that I expect the work of JPL to be perfect. There will always be problems. Thats the nature of engineering work. But once problems occur they need to be investigated to ensure that they aren't repeated and to find larger underlying problems. One of the worst types of problems that you can find are repeat problems (of the same nature). They indicate that you knew a problem existed but your corrective actions were inadequate. Without performing an investigation Squyres does not even have the chance to assign corrective actions! He is taking no action whatsoever to try to prevent future problems. Once the repeat problem occurs (and it will occur), heads will fly!

      --
      Suddenly, the hairy finger of a familiar monkey tapped me on the shoulder. It was time.--G. T.
  14. Cat calls from the cheap seats by amightywind · · Score: 4, Interesting

    Let the New Scientist criticize from the cheap seats. It is hard to argue that the rovers have been anything other than a resounding success for over 400 days. I would have hoped /. would instead print the recent story of the Spirit Rover discovering salty soil.

    --
    an ill wind that blows no good
  15. Something fishy here... by Anonymous Coward · · Score: 2, Insightful

    "There was a point when both of them were sitting on the same bench, and that has to have been it."

    Wouldn't they have been labeled, what does this have to do with anything?

  16. This doesn't effect results by Monkey_Genius · · Score: 3, Informative

    From the JPL website: "Meanwhile, scientists are re-calibrating data from both rovers' alpha particle X-ray spectrometers. These instruments are used to assess targets' elemental composition. The sensor heads for the two instruments were switched before launch. Therefore, data that Opportunity's spectrometer has collected have been analyzed using calibration files for Spirit's, and vice-versa. Fortunately, because the sensor heads are nearly identical, the effect on the elemental abundances determined by the instruments was very small. The scientists have taken this opportunity to go back and review the results for the mission so far and re-compute using correct calibration files. "The effect in all cases was less than the uncertainties in results, so none of our science conclusions are affected," Squyres said." It would have been more serious if they had lost the calibrations on the instruments.

    --
    I've got your sig, right here.
  17. Re:If... by Tolookah · · Score: 5, Informative

    one word: Tolerances.

    when you have 100 resistors at 0.5% tolerance, you are gonna have drift, and you will have to calibrate them to the right parts, you can make 100's of those cars of the same make and model, but none will be exactly the same, especially when you have sensitive equipment. (Think odometer)

  18. Mixed up units by eikonoklastes · · Score: 4, Informative

    >once lost a Mars spacecraft because engineers mixed up metric and imperial units.

    I'm getting pretty tired of this sound (text?) bite the media throws out. It wasn't mixed up units; it was error accumulation from switching back and forth between the units.

    1. Re:Mixed up units by DerekLyons · · Score: 4, Informative
      >once lost a Mars spacecraft because engineers mixed up metric and imperial units.

      I'm getting pretty tired of this sound (text?) bite the media throws out. It wasn't mixed up units; it was error accumulation from switching back and forth between the units.

      It's even worse when the person making the correction himself gets it wrong.

      The cause of the loss wasn't mixed up units, though they contributed. The loss was caused by ignoring a growing discrepancy between the precalculated navigation values and the actual navigation values. The errors were well within the correctable range, but for a variety of reasons the subtly different but incorrect values were ignored until it was too late to correct for them.

    2. Re:Mixed up units by GileadGreene · · Score: 2, Insightful
      ...it was error accumulation from switching back and forth between the units.

      No, it wasn't. Yes, there was error accumulation. But the accumulation was due to a metric-english conversion factor that had been dropped during the port of the flight software from a previous program. The lack of decent documentation for the software meant that the folks assigned to do the port were unaware of the significance of the conversion factor. without the conversion factor thruster burns were executed incorrectly, resulting in a deviation from the designed trajectory. Every burn resulted in worse errors. The mission still could have been saved, but the mission managers elected to ignore the growing deviations, and just "hope that things get better".

    3. Re:Mixed up units by InfiniteWisdom · · Score: 2, Informative

      Can you cite any source that confirms this? Everything I could find says that the ground software used pound-force instead of Newtons, so the thrust was off by a factor of ~4.5. Nobody says anything about conversions taking place

    4. Re:Mixed up units by FleaPlus · · Score: 2, Informative

      It wasn't mixed up units; it was error accumulation from switching back and forth between the units.

      Do you happen to have a source for that? Wikipedia says the following:

      http://en.wikipedia.org/wiki/Mars_Climate_Orbiter# The_metric_mixup

      The Mars Climate Orbiter's reaction wheels were kept within their linear (unsaturated) range through thruster firings in a procedure called Angular Momentum Desaturation (AMD). When an AMD event occurred, relevant spacecraft data was telemetered to the ground, processed, and placed into a file called the AMD file. The JPL operations navigation team used data derived from the AMD file to model the forces on the spacecraft resulting from these specific thruster firings. Modeling of these small forces is critical for accurately determining the spacecraft's trajectory. Immediately after the thruster firing, the velocity change (DeltaV) is computed using the firing time for each of the thrusters, and an impulse bit, which models each thruster's performance. The calculation of the thruster performance is carried out both on-board the spacecraft and on ground support computers. The AMD software installed on the spacecraft used metric units, newton seconds (Ns), for the impulse and was correct. The ground software reported the impulse bit to the AMD file in English units of pounds (force) seconds (lbfs), rather than the metric units required by the project's Software Interface Specification. Subsequent processing of the impulse bit values from the AMD file by the navigation software underestimated the effect of the thruster firings on the spacecraft trajectory by a factor of 4.45 (1 pound force = 4.45 newtons).

  19. Hats Off to NASA by Orphaze · · Score: 5, Insightful

    So, let me get this straight: NASA has managed to successfully send two completely functional rovers to the planet Mars 45 million miles away. Since they have arrived, the two rovers have expanded our understanding of the planet greatly and have had few and mostly correctable errors. They are now way, way past their expected mission time and are still running, and a few people have the nerve around to here to bash NASA for their horrible, numerous mistakes?

    This stuff isn't easy. Just because you reap the benefits of the entire space program from your living room couch via the TV without actually contributing one bit does not mean you have any understanding of how complex and spectacular these great accomplishments are.

    To the NASA / JPL engineers and scientists: Thanks.

    1. Re:Hats Off to NASA by jimmyCarter · · Score: 3, Insightful

      Amen. When's the last time any of us can say we've written a bug-free app? NASA's effort is significantly harder than connecting your website to a database.

      --

      -- jimmycarter
  20. Re:If... by UniverseIsADoughnut · · Score: 4, Informative

    Damn your Dumb! Read before posting dumb things. No two things are exactly the same even if designed as such. And yes you can get in two identical cars and one will perform different.

    It's a calibration, the whole concept is no two things are the same. Any piece of instrumentation needs to be calibrated and that calibration is set for that device. No manufacturing can produce 2 identical things, just not possible since the two items could not share the same time and space in the universe together thus both are going to be slightly different.

    Anytime you get a piece of gear, you get it setup, then you take some means of calibrating it and test it with something that has deemed to be as accurate as possible. Maybe some source such as a rock. But basically anything that will provide a common test basis for the device.

    So say you are measuring temperatures on something in a lab. You set up a big system for collecting data off a bunch of thermocouples. Each one has it's own channel through it's own voltage modules and thermocouples and so forth. So you take a calibrator and have it feed a signal through the system to mimic a thermocouple. you get a calibration curve for a channel, then you go to the next channel with the same device and do the same for the next channel, this will be a different calibration. and then you work through them all. I have system set up with 16 channels at work and all use the same parts, but there is about 4 closely similar calibrations across the channels, but no two channels follow the same calibration.

    NASA did the same, they built the devices, then calibrated them with the same rocks, and developed a calibration curve for each system, and that was to be kept with each rover, they swapped the instruments, so now they switch the calibrations and everything is fine.

    This all goes back to simple accuracy and how close you can get things, but bottom line no two things are the same. Look at computers, you can have 100 computers, exactly the same built right in a row, with the exact same software and so forth. Turn them on and let the run under exact same conditions, some will have hardware failures, some will have software get wacky on them and so forth. It's just the way it works.

  21. Re:I want to fight for NASA but come on... by MillionthMonkey · · Score: 4, Informative

    RTFA. NASA swapped the instruments inadvertently but since both the rovers and the detectors are manufactured identically the only consequence was that we were using the calibration data file from one instrument to analyze the data coming from the other. Swap the calibration files so they are coupled to the correct instruments again, reanalyze the raw data, and the problem is solved without having to privatize NASA.

    How can i possibly advocate for a mars mission when they can't even get this shit right?

    The Mars mission is stupid but not for the reason you give.

  22. In other news: by Viceice · · Score: 4, Funny

    A hermit writes: "The Church is reporting that the two human genders, male and female, have instruments installed in the wrong genders. From the article: 'While the bungle does not undermine the main reproductive conclusions from the reproductive activities between genders, it is an embarrassing slip-up for a supreme being that once lost a world of worshippers to a flood because the first prototypes mixed up good and evil.'"

    --
    Sometimes I wish I was a plumber, then I'd know how to deal with other people's shit.
  23. Re:If... by Fallingcow · · Score: 3, Funny

    No manufacturing can produce 2 identical things, just not possible since the two items could not share the same time and space in the universe together thus both are going to be slightly different.

    Oh, come on! It's easy!

    ln rover_1 rover_2

    See?

  24. Finish giggling about poor, dumb NASA... by The+Master+Control+P · · Score: 2, Insightful

    Then ask yourself how many times identical twins that you've known managed to play some trick on you.

    And can we tone down the headline sensationalism a bit? You'd think the rovers have a core drill where there should be a camera or something. They somehow managed to switch two spectrometers, as identical as modern metallurgy can make them, destined for two similarly identical rovers - and now the error's been uncovered and the data recomputed. Jeesh...

  25. Re:No, it's still stupid by cetialphav · · Score: 3, Insightful

    Right, a Post-It. On a spacecraft to Mars? These are highly sensitive one of a kind instruments. You don't just go sticking paper and glue all over it.

    Post-Its are not static dissipative. You could have a static discharge damage components and you wouldn't even know until the rover had landed on Mars. You could accidently leave a Post-It on the spacecraft and cause damage. How do you know residue from the glue on the Post-It won't cause damage? Now you have to test for that. It is amazing how one stupid thing like a Post-It note could add more complexity and make things even worse.

    Now what would have been smart is to have devices like this keyed so that they can't possible be installed in the wrong place. But that tends to add complexity to the design and when you are only building a handful of rovers in highly controlled conditions, it can be hard to justify.

    What is stupid is that there is no investigation of what happened. Sure, in this case the mixup was relatively harmless, but the next one might not be. NASA needs to be more proactive and not wait until things blow up to have an investigation. I don't expect perfection, but they at least have to understand their flaws.

  26. Hit it! by dexter+riley · · Score: 3, Funny

    Because they're rovers,
    Identical rovers, you will find...
    They look alike, they rove alike,
    They even calibrate alike!
    (Should I put this alpha-particle X-ray spectrometer in you...or you? Whoooaaaa!)
    You will lose your mind!
    When rovers...are two of a kind!

    Identical Rovers! Tuesdays at 8 on SCTV!

  27. 'Bungle' is a bit too harsh... by Frennzy · · Score: 2, Informative
    Calling this oversight a bungle is a bit too harsh...for those who didn't read the article, it also says:

    their mission has been considered an unqualified success. Spirit and Opportunity provided the first irrefutable evidence that there was once liquid water on the surface of the Red Planet and are still roaming long after their scheduled 90-day mission.


    Once the mistake was realized, they could easily accomodate it through other calibration techniques. I think the parent article is trying to raise a sandstorm in an otherwise rarefied atmosphere.

  28. Speaking of slipups... by fm6 · · Score: 3, Informative
    ... because engineers mixed up metric and imperial units.
    The U.S. does not use imperial units. We use customary units. Many unit names are the same, but not the units themselves.
  29. They should have... by chud67 · · Score: 2, Interesting

    ...put Burt Rutan in charge of the mission.

  30. Two for two by Anonymous Coward · · Score: 5, Funny

    This was a follow up mission to the Viking landers which found no signs of life on Earth.

  31. Re:If... by Anonymous Coward · · Score: 2, Funny

    Damn your Dumb

    "you're".

    you get it setup

    "set up". ("setup" is a noun.)

    something that has deemed to be

    "has been deemed".

    Each one has it's own channel through it's own voltage modules

    "its" (both places).

    but there is about 4 closely similar calibrations

    "there are about".

  32. Only a minor issue because of luck by Illserve · · Score: 4, Insightful

    Had either of the Mars Rovers crashed or broken in some way, this mistake would never have been discovered. With only 1 rover's data, there would be no mysterious discrepency to solve and this mistake would have never been resolved.

    So scientists would have spent the next 10 years developing their theories of martian geology based on incorrect data if either one of those rovers hadn't deployed and you call this a minor issue?!

    This kind of error is inexcusable. But of course, it'll get brushed over because NASA was lucky enough to be in a position to fix it.

    1. Re:Only a minor issue because of luck by deglr6328 · · Score: 3, Insightful

      Jeez how clueless can you get? Are people really this anal here? If you even took 5 minutes to look into this issue you'd see that it is NOT a big deal. Steve Squires himself said: "The effect in all cases was less than the uncertainties in results, so none of our science conclusions are affected,". Conclusion: it wouldn't have even affected the science appreciably if they never found out they were switched!! Also you're totally wrong when you say "Had either of the Mars Rovers crashed or broken in some way, this mistake would never have been discovered.". Hello? do you not think they took the calibration data ON EARTH? It wouldn't have mattered if either one were lost. We'd still have all the calibration data.

      --
      - "Hear that?! The percolations are imminent! Cease your ingress!"
  33. Beagle 2 had the correct instruments installed by tinrobot · · Score: 2, Insightful

    ...and it proceeded to install those instuments all over the surface of mars.

    Spirit and Opportunity have performed incredibly well. These guys deserve nothing but respect.

  34. Could be worse by Cliff.Braun · · Score: 2, Funny

    You know, Compared to mixing up metric and imperial units, this seems downright intelligent of them.

  35. Oops. by ktakki · · Score: 4, Funny

    Okay, I was in the meeting where the difference in calibration was discussed, and I was the one that suggested that the instrument packages should be marked so that the right package would be installed in the right lander.

    I recommended that one package should be marked with an "O" for "Spirit" and the other with an "S" for "Opportunity". I even donated the Sharpie marker and masking tape for this purpose.

    It's not my fault that the implementation was screwed up. It's those numbnuts in the Vehicle Assembly Department who can't read a bloody memo.

    Fortunately, I've left NASA for a position at the Department of Defense. My team is tasked with identifying sites related to the constructon of weapons of mass destrucion in South Korea.

    k.

    --
    "In spite of everything, I still believe that people are really good at heart." - Anne Frank
    1. Re:Oops. by Dominic_Mazzoni · · Score: 3, Informative

      I recommended that one package should be marked with an "O" for "Spirit" and the other with an "S" for "Opportunity". I even donated the Sharpie marker and masking tape for this purpose.

      I know you were joking, but keep in mind that the names "Spirit" and "Opportunity" were chosen very late into the mission, as the result of a contest. Within JPL, the probes were known as MER-A and MER-B, and the rovers were known as MER-2 and MER-1. To make things even more confusing, for various sensible reasons they ended up putting MER-2 inside MER-A, and MER-1 inside MER-B, even though that made things more confusing.

      So, considering that they were otherwise identical, can't you see how easy it would have been to get otherwise identical parts mixed up...was it supposed to go in MER-B? Or MER-2? I just remember it was the second one of something...

  36. Not informative, but wrong by 4lex · · Score: 4, Informative
    Please look page 13 of this report of the NASA (las paragraph of the page):

    "On September 27, 1999, the operations navigation team consulted with the spacecraft engineers to discuss navigation discrepancies regarding velocity change (V) modeling issues. On September 29, 1999, it was discovered that the small forces V's reported by the spacecraft engineers for use in orbit determination solutions was low by a factor of 4.45 (1 pound force=4.45 Newtons) because the impulse bit data contained in the AMD file was delivered in lb-sec instead of the specified and expected units of Newton-sec."

    --
    My journal. Mainly about freedom.
  37. I guess being... by Anita+Coney · · Score: 2, Funny

    ... a rocket scientist isn't what it used to be.

    --
    If someone says he and his monkey have nothing to hide, they almost certainly do.