Mars Phoenix Lander's Ovens Were Destined To Fail

RobertB-DC writes "The Phoenix mission to Mars' frigid polar regions was going to be tricky from the start, with only a few weeks to perform as much science as possible. Success depended on everything working right. But one of the mission's most frustrating glitches — the stuck doors on the TEGA ovens — could have been prevented with basic quality control on Earth. Nature is reporting that bad brackets were replaced by the manufacturer ... with identically bad brackets. The Planetary Society blog sums it up succinctly: 'Ouch. Ouch ouch ouch.'"

Comments

Design by commitee

[linzeal]

This is what happens when too many people have their hands up the engineers and by extension the technicians' asses.

Re:Design by commitee

[causality]

This is what happens when too many people have their hands up the engineers and by extension the technicians' asses.

Sounds like a bunch of smelly hands.

Re:Design by commitee

[Anonymous Coward]

Sounds like they want to stink-palm somebody... mmh, chocolate pretzels...

Re:Design by commitee

[Squeeonline]

This is what happens when the pieces are made by the lowest bidder.

Re:Design by commitee

[lysergic.acid]

i'm pretty sure all NASA and DoD contractors are paid via cost-plus-award-fee contracts.

so the problem isn't that they're being paid too little. if anything, they're being paid too much for too little work (and too little quality). if NASA contracts are handed out the same way that military contracts get handed out, then it is probably done through a corruption-filled old boy network negotiated by kickbacks and bribery. that kind of cronyism breeds incompetence as it destroys any hint of meritocracy or accountability.

Re:Design by commitee

[Squeeonline]

So whats the solution?

Re:Design by commitee

[reilwin]

Sounds like a bunch of smelly hands.

Smells like a bunch of noisy hands.

Re:Design by commitee

[Anonymous Coward]

That was lame. You fail.

Re:Design by commitee

[Anonymous Coward]

liability and success bounded payment

So ?

[Fred_A]

Aren't they covered by warranty ? Get them to replace them.

Re:So ?

[TheLink]

I think the main problem is when you ask for on site support.

They'll look at you as if you came from another planet or something.

Re:So ?

[peragrin]

hey i can see the tech support bill for that one.

think call center's in India are annoying?The martian centers refuse to speak any earth language.

Re:So ?

[confused one]

But they want you to pay the return shipping.

Re:So ?

[Sockatume]

Seriously, they should do this. Hold them to the same standards as a washing machine company. If a contractor screws up, they're going to pay for sending an engineer out there to fix the product. (And if they want him back, they can pay or that too.) If they don't want to do that, well, they can pay for a whole new mission. Then they're less likely to do things like skip diagnostics and fuck up multi-million dollar missions.

Re:So ?

[Enter the Shoggoth]

This reminds me of the the Apple iBook I have rotting in a draw somewhere, Apple acknowledged that the product had a known design fault, but all they did was replace the logicboard with an identical one, which of course would also fail, in my case I went through _six_ logicboards, two of them in the one go (the tech replaced it and it failed during testing so had to be replaced again before it was returned to me)

What really amazes me about this is that it is legal. This is due (in my country at least) to corrupt politicians taking too many brown paper bags full of cash in return for winding back consumer protection laws... if a manufacturer acknowledges that there is a known _design_ fault and then continues to provide the faulty product they aught at the very least be told to replace the faulty product with a _redesigned_ one without someone having to go to the trouble of filing suit. Personally, in addition to this I think the executives should also be sent to pound-me-in-the-ass prison.

Re:So ?

[sjames]

Absolutely, insist on that in writing.

You'll get one of two possible responses. The first response is to decline to bid on the contract at all. The second is to form a subsidiary to bid with the explicit intention of declaring bankruptcy if you try to collect on the warranty.

Re:So ?

[Anonymous Coward]

Seriously, they should do this. Hold them to the same standards as a washing machine company. If a contractor screws up, they're going to pay for sending an engineer out there to fix the product. (And if they want him back, they can pay or that too.) If they don't want to do that, well, they can pay for a whole new mission. Then they're less likely to do things like skip diagnostics and fuck up multi-million dollar missions.

...and this will raise the price of the mission exponentially, as every contractor will have to carry billions in insurance, even if they supply no more than a single bolt used on the spacecraft. Good plan.

Re:So ?

[cide1]

I played that game with my iBook G3. Eventually I got fed up with it and bought an iMac G5 20". I paid $2200 for it. My iMac G5 failed after two years, Apple replaced the logic board + power supply due to bad caps. Another two years later, the exact same failures. Apple washed their hands of it, as it was past the 3 year warranty. They openly acknowledge this is a systemic problem, but have no plans to correct it. I was a very loyal customer, with the iBook G3, the iMac G5, a Macbook, 3 iPods and an AppleTV. Having gotten the run around several times from them, I have vowed to not buy Apple again after this.

Re:So ?

[adolf]

And this, boys and girls, is why we don't buy Apple products.

Or anything else proprietary, for that matter. Especially for desktops. (Laptops get a bit of exception here, on the basis that they're all finicky little pricks by default.)

Were it a parts-built desktop PC with after-warranty bad caps (raise your hand if you haven't seen an Athlon XP box with bad caps), it'd have been fairly easy and inexpensive to replace the motherboard with something different of newer design.

Alas, the 20" iMac G5 (which sounds like a rather nice computer) rots. And since all available boards for it seem to have the same design fault, nobody will ever fix it.

Fun.

But...But

[AkaKaryuu]

How will we know exactly how extremely high altitudes affect baking times now?

Proverbial problem

[owlnation]

Ovens?

Sounds like too many cooks were involved.

Without reading the article...

[3seas]

...isn't this what happens when you gotta have it yesterday?

Human space exploration.

[slmouradian]

One more thing to add to my list why humans should be involved in space exploration, not just robots.. Perhaps this could be fixed if there was a human there?!

Re:Human space exploration.

[Anonymous Coward]

Face it - humans (at least, American humans representing NASA) aren't going to Mars any time soon. The bad PR from a fatal mission failure would be too much for them to overcome, so they will never try. The West is too pussy whipped by the safety police to attempt manned interplanetary exploration.

Plus, this problem could have been averted by simply checking the oven bracket design again after it came back from Honeybee. Honeybee should be sued for returning a flawed design a second time and ruining the mission. Either sued, or banned from NASA's list of approved suppliers.

Re:Human space exploration.

[sjames]

One more thing to add to my list why humans should be involved in space exploration, not just robots.. Perhaps this could be fixed if there was a human there?!

That's not a very good reason to send humans to Mars. For the difference in cost, we can send a dozen or so replacement probes before we even approach the cost of a manned mission.

We would do well to expand our orbital presence first. We need better than chemical propulsion and we need life support systems that can run as a closed system. It's much better to test that in orbit where a failure means we evacuate and try again rather than on a Mars mission where failure means transmit your last words.

Once we have a significant orbital presence, that also gives us the ability to build and launch the Mars vehicle in orbit. That is, only the lander portion need be designed to operate in an atmosphere at all and only needs to handle landing and takeoff in Mars' gravity. The Earth-mars transport vehicle can be entirely un-aerodynamic and need only support it's own thrust.

Re:Human space exploration.

[Anonymous Coward]

why not start the test in Antarctida first ?

Re:Human space exploration.

[sjames]

why not start the test in Antarctida first ?

No need for Antarctica for that, just as long as the environment is sealed. Such tests are underway already (plus we have expeditions in Antarctica already gaining other valuable experience).

A next step will be to build something like the equipment that will actually be used on the mission and test that on Earth, then in the ISS.

haha

[someone1234]

I wonder who lost their humour by marking this a troll.

Re:haha

[Anonymous Coward]

Probably some turd-eating GNAA moron with an inflated scrotum or maybe it was someone from Honeybee Robotics.

Re:haha

[S.O.B.]

Maybe it was the Martian government trying to protect local jobs from foreign workers.

Hm...

[home-electro.com]

"were just a hair's width too big"

and is obstructing the door??? that's some horrible engineering.

Re:Hm...

[Anonymous Coward]

Or poor tolerances on the manufacturing. Or bad drafting. It *might* not be the engineer's fault. Maybe.

Re:Hm...

[tftp]

Yes, it is a horrible engineering. However these are one-off designs that never existed before and will never exist after. There is no legacy to build upon, and there is no "Release 1" to learn from. The very first release flies the mission, and if there are bugz ... too bad. To confound the problem, much of this work is probably done by scientists and not by engineers; that's why if the gap between doors is above zero it's all good to go. An experienced mechanical engineer would consider thermal expansion, free play in all pivot points, and other things - but first she'd try to increase the gap to some reasonable size, so that none of those secondary effects could affect the mission.

NASA might be tampering with photos

[Anonymous Coward]

http://www.libertythink.com/totalinformation/BlueMars.htm

Dunno what to think.

Re:NASA might be tampering with photos

[Anonymous Coward]

The horror! NASA is tampering with colors in their pictures to enhance scientifically interesting features!?! How shocking!!!!

Re:NASA might be tampering with photos

[Anonymous Coward]

Enhance? Did you even read that webpage? According to it, they are making it harder to see details.
It claims they are making it hazier and redder.

Re:NASA might be tampering with photos

[Anonymous Coward]

I'll just leave this here.

Re:NASA might be tampering with photos

[Fastolfe]

Off-topic, and the author is an idiot. The rovers' cameras do not necessarily take pictures using the standard red-green-blue colors that we perceive. Depending on what filters were used (for scientific reasons), if you want a "full color" image for humans to appreciate, you have to choose or synthesize non-RGB channels to form an RGB image. The blue tab, for example, on the color calibration target is also very bright in the infrared, so if you use an infrared image as your red channel, what should be blue appears to be pink. All of this perfectly normal and completely expected by everyone that knows how this stuff works. Stop being a silly conspiracy theorist and apply some rational thought and a tiny bit of research.

http://www.badastronomy.com/bad/misc/hoagland/mars_colors.html
http://marsrovers.jpl.nasa.gov/spotlight/spirit/a12_20040128.html

Re:NASA might be tampering with photos

[ElliotLee]

From your Google search, I clicked the #1 result and read the following:

If I sound a little testy here, it's partly because I've found those astronomer/geologists involved to be quite arrogant. I listened to Linda Howe's interview with Steve Squyers, principal investigator on the Mars Rover Missions, during which, when the subject of organic life on Mars was brought up, he became agitated and downright rude to Howe - as if she asked a forbidden question. The reaction was appalling, certainly not something you'd expect from real scientists. (You can read the transcript of the interview here, near the bottom.) The thing is, they know what they want to discover - or what they think they will see on Mars - and their big egos (or brainwashed minds) won't leave room for anything else. It's quite comparable to how the Bush administration went after the Iraqi WMDs, really.

http://www.goroadachi.com/etemenanki/mars-hiddencolors.htm

Echoes my thoughts. You could very well be right, but at least be open to other possibilities.

Re:NASA might be tampering with photos

[cheekyboy]

"Each image is not really color, but greyscale (what some people erroneously call "black and white"),"

Well, for decades greyscale TV was called B/W tv, walk into any photographers shop and they talk about 'black n white' photos.

Re:NASA might be tampering with photos

[Fastolfe]

Other possibilities?? Maybe a thought experiment will re-activate the logic centers of your brain:

Let's say that I have three monochromatic images, each measuring the intensity of light at 430nm, 550nm, and 700nm. Your task is to turn these three images into a full-color image on a computer screen. These happen to be approximately the primary colors in the CIE RGB color space, so this should be an easy task. You just map the 430nm image to the blue layer, 550 to the green, and 700nm to the red.

Now I hand you a set of images at 480nm, 530nm, and 680nm. These are still reasonably close to the CIE RGB primaries, so you can probably do the same thing. But will the resulting image look the same? Or will there be subtle differences in color? (Hint: the second one.)

Let's replace the 680nm image with one at 980nm. What do you do now? The red channel is now way out in the infrared. People can't see in that wavelength, so the intensity of light at that wavelength is going to be completely foreign and strange. If you have a chip of paint in this photograph that reflects brightly at ~430nm, and again at ~900nm, what color is this chip going to appear when the red channel is 680nm versus 980nm?

But let's take this one step further. What if I handed you a set of images at 670nm, 800nm, and 980nm? What do you do then? How do you build an RGB images out of wavelengths that are effectively just shades of red and infrared?

There is no need to examine "other possibilities" because this is already well-understood. Everyone that's taken a basic college photography class understands how light and color work. These people are not surprised by this because they understand it. It's like expressing shock and awe about how a house holds itself together, and when a builder goes on about how nails work, you scoff and say, "you need to consider other possibilities!" Just because you don't understand something doesn't mean there's something magic or secret about it.

Re:NASA might be tampering with photos

[Fastolfe]

Here are the filters available to the Mars rover camera:

http://www.ominous-valve.com/pancam.html

There's also a picture of the color calibration target through each of these filters. Fire up Gimp and do some mixing and matching.

Re:NASA might be tampering with photos

[adolf]

Depends on the film.

Photos taken with infrared film won't be described in a camera shop as being "black and white," but simply IR. The shades presented are related to the object's reflectivity at infrared, not with visible (white) light. X-ray photos aren't black and white, either - the shades grey in an x-ray photo have nothing at all to do with the color of the objects being x-rayed.

Black and white film, along with black and white TV, are both obviously different from these -- they both work with white light.

My Mother, NASA oven test engineer . . .

[PolygamousRanchKid]

From the blog:

Boynton and his team had noticed, on a test version of TEGA, that the brackets at the bottom of this cover were just a hair's width too big, and as a result obstructed the doors. They sent revised designs for the cover to the manufacturer, Honeybee Robotics of New York. New parts were delivered and installed. But Honeybee had made the new parts using the original flawed designs -- and nobody in Tucson checked them. "They should've caught it and we should've caught it, but neither of us did," says Boynton, ruefully.

. . . which is why NASA needs to hire my mother as oven test engineer. Not only would she have noticed "hair's width" difference, she would have taken every opportunity she had to complain to everyone she knows, and even total strangers about it.

On the other hand, once the door problem got fixed, she would find something else wrong with it, and the damn thing would probably never get off the ground.

Re:My Mother, NASA oven test engineer . . .

[isaac338]

From the blog:

Boynton and his team had noticed, on a test version of TEGA, that the brackets at the bottom of this cover were just a hair's width too big, and as a result obstructed the doors. They sent revised designs for the cover to the manufacturer, Honeybee Robotics of New York. New parts were delivered and installed. But Honeybee had made the new parts using the original flawed designs -- and nobody in Tucson checked them. "They should've caught it and we should've caught it, but neither of us did," says Boynton, ruefully.

Doesn't NASA of all institutions have the capacity in-house to machine off a couple thou from whichever dimension was oversize?

Re:My Mother, NASA oven test engineer . . .

[theralfinator]

Yeah seriously I work at a factory that makes formed insulation for electrical transformers, and I'm willing to say that just about any of our CNC machines could probly have planed down a couple thousandths from a few sides. O NASA, how we love you.

Re:My Mother, NASA oven test engineer . . .

[slmouradian]

yes, but it's very hard to ACCURATELY estimate the size of the oven in the quite large temperature variations on Mars.. Don't forget the oven does expand and contract every day/night, and every time it warms itself up.

Re:My Mother, NASA oven test engineer . . .

[pavon]

The lander wasn't made by NASA, JPL, or anyone like that. It was designed and assembled by the University of Arizona, who naturally had to get most of the parts fabbed by other folks.

Re:My Mother, NASA oven test engineer . . .

[Anonymous Coward]

A good shagging would put an end to all that complaining.

Re:My Mother, NASA oven test engineer . . .

[lytir]

The university of Arizona defined the scientific objectives. The spacecraft was designed and built by Lockheed.

Re:My Mother, NASA oven test engineer . . .

[Tablizer]

Different parts were built by different organizations. Lockheed may have been the main integrator.
     

Human Mars mission

[wfstanle]

You are forgetting something ...

Some plans for a manned Mars mission were based on there not being a return trip to Earth. Anyone who went on such a mission would be marooned there on purpose. It's not a kind of trip I would like to take.

Re:Human Mars mission

[Toonol]

I think there would be no shortage of volunteers. And by that, I mean, millions of volunteers, including all astronauts, and everybody that wants to be one.

I'm kind of surprised to read a poster on slashdot write they wouldn't volunteer for a one-way mission.

Re:Human Mars mission

[chis101]

I'm kind of surprised to read a poster on slashdot write they wouldn't volunteer for a one-way mission.

As cool as going to Mars would be, I for one don't want to die for it.

Re:Human Mars mission

[KORfan]

Just run an ad in the paper saying something like this

"Men wanted for hazardous journey. Low wages, bitter cold, long hours of complete darkness. Safe return doubtful. Honour and recognition in event of success."

Evidence suggests you'll get more volunteers than you can use.

Re:Human Mars mission

[Tubal-Cain]

Honour and recognition in event of success.

Technically, they would get that anyways. They just get to enjoy it more if they survive.

So, private enterprise is always better than gov.?

[Anonymous Coward]

"Nature is reporting that bad brackets were replaced by the manufacturer... with identically bad brackets."

Yeah, chalk one up for private enterprise and down with government!

(if you did not get it, that was sarcasm.)

Sorry for the political rant, but after seeing so many right-wing brainwashed drones spouting that mindless sound-bite that government is inefficient, can't do anything right, and so on, this failure shows that incompetence, negligence and ineffiency (sp?) is not the exclusive domain of 'government'. You will have lazy idiots everywhere, including private enterprise.

Like you can get some really bright & dynamic people working for 'government'.

This being said, too bad the Phoenix Lander had a problem. I'm really sorry for all those engineers and scientists that poured their heart and soul into this only to see it fail.

Amazing

[RAMMS EIN]

``Nature is reporting that bad brackets were replaced by the manufacturer... with identically bad brackets.''

Isn't that just purely amazing? A manufacturer who _knows_ the component is bad (because it needs replacement), and then replaces it with ... the same thing with the same faults. That's just unethical. I hope they are suitably punished.

Also, you would have thought that, after sending a component back for replacement, the replacement would be tested to see if the problem had been fixed.

I just don't have words anymore.

Re:Amazing

[NekSnappa]

As the saying goes. "Never attribute to malice what can be explained with incompetence."
This could well have been an issue of poor configuration management. Since the article says they used the same drawings. I imagine that even if their models were updated if those changes weren't propagated up through the drawings and the machining files used on the fabrication floor.
So the net result would be an identical part being fabbed.

Re:Amazing

[jimmydevice]

This sounds like the classic, if apocryphal, ISO 9000 process written by a noticed to termination and pissed employee.
"Floppy disks are to be soaked in warm, 96 deg water before shipping"
Same shit, developed without reviewing the processes and executing same without thought.
Idiots!

Re:Amazing

[instarx]

Not unethical - just sloppy and incompetent. On any job there is usually one and only person who is able to anticipate problems and do "what ifs" to head off problems. Everyone else is just doing their 9-5. Clearly the spacecraft people didn't have one of those.

i am increasingly depressed by the sheer incompetence of most people in this country, from assembling hamburgers to Mars landers.

documentation

[wkk2]

Everything needs a version number and serial number.

Re:documentation

[Ig0r]

I am not a number; I am a free man!

not all are cost plus

[Anonymous Coward]

The ODIN contract comes to mind.

Sounds like me.

[antdude]

I am very picky and stubborn with things. And that' why I have a job in software quality assurance (SQA). I always finds things that bother me even it is a pixel size problem. :D

manglement

[sohp]

Most of the comments so far are focusing on the oven door problems. Naturally, because that's what's mentioned in the summary and no one RTFAs.

Anyway, the *much* more interesting revelation is that after the problems came up, the directive came all the way down from the top of NASA directing the mission scientists to change their plans. "At the end of June, word came down that the Phoenix team was to treat its next TEGA sample as its last, and to go after a sample of rock-hard ice before it did anything else. The Tucson team had lost its autonomy." After that, the team blew at least a month trying to meet this directive, and missed out on doing some of the basic science they wanted to do, just so NASA heads could trumpet feel-good publicity about having detected ice with Phoenix.

Re:manglement

[MichaelSmith]

It was clear to management that the mission scientists had stuffed up and were not to be trusted.

Re:manglement

[sohp]

That *might* be true, but if it were, you'd think they'd have been asked to conduct some real science, not focus on the golly-gee-whiz-we-found-ice aspect.

Re:manglement

[MichaelSmith]

No bucks, no buck rogers. NASA management need headlines to justify budgets.

Re:So, private enterprise is always better than go

[Keys1337]

It seems that private industry has figured out that when dealing with the gov't that they can regularly provide substandard products, go over budget and behind schedule with no negative consequences and usually benefit from these shenanigans. You blame private industry, others blame government for not providing the correct positive and negative feedback to private industry like any nonretarded entity would.

Wrong units of measurement strike again.

[techno-vampire]

Clearly what happened is that the design was made specifying a tolerances of a blonde hair, and it was built to within a red one. When will these engineers learn to be more specific!

NASA quality - the state of US science

[Anonymous Coward]

Maybe I'm the only one, but I think NASA quality control has really sucked in recent years. They've been plagued by problems that just shouldn't have happened. Problems that seem both childish and stupid.

And as a solution, we get things like "banging the craft with robot arms," which is on par with hitting the TV with your fists to fix it. -- Come on, nobody ever really thought that this oven thing might happen? If this is really so, what else won't these people think of when they send real people out there?

I know I may be expecting too much from american science, and I know the americans' education budget has been cut and the system encourages anything but smart kids to perform, and maybe that's to blame for the current state of things, but my point is that I wouldn't want to be aboard any Mars mission craft that is either designed or tested by these guys. I'm sorry, but I just had to say this.

Next time, maybe NASA should consider including a remote-operated leg that can give a proper kick to the Mars robots in case there's any problems.

I think there ought to be a Simpsons or South Park gag about this if there isn't one already. I mean, NASA _resorting to violence to fix things_ is about as primitive as "the state of the art" in US science can get (at least I sincerely hope so... How low can they go?), and I think it really speaks volumes of how things are today, and makes me feel rather sad. I also no longer wonder why there's so much serious talk of making the Mars mission just a one-way trip there.

If only the Moon mission teams were here.

Marvin

[HTH NE1]

Nature is reporting that bad brackets were replaced by the manufacturer ... with identically bad brackets. The Planetary Society blog sums it up succinctly: 'Ouch. Ouch ouch ouch.'

Not to mention the diodes down its left side.