Slashdot Mirror


Saving Huygens

TazMainiac writes "This months IEEE Spectrum is running an article on how a Swedish scientist discovered that the Titan probe Cassini had a communications flaw that would cause it to lose all data sent back from the Huygens lander as it plunges into Titan's atmosphere. The problem - Doppler effect. The fix: go read the article."

42 of 267 comments (clear)

  1. RTFM is the fix? by Kenja · · Score: 5, Funny

    So wait, reading the article will fix the Cassini probe?

    --

    "Have you ever thought about just turning off the TV, sitting down with your kids, and hitting them?"
    1. Re:RTFM is the fix? by Random_Goblin · · Score: 3, Interesting
      from TFM
      Alenia Spazio's insistence on confidentiality may have played a role in this oversight. NASA reviewers were never given the specs of the receiver..."Alenia Spazio considered JPL to be a competitor and treated the radio design as proprietary data."
      ...NASA probably could have insisted on seeing the design if it had agreed to sign standard nondisclosure agreements, but NASA didn't consider the effort worthwhile, automatically assuming Alenia Spazio would compensate for the changing data rate.

      so in this case the problem was indeed caused by the fact they couldn't RTFM to check the supplier had done the job correctly.

      I think it is all summed up with the line "An Alenia Spazio spokeswoman said that none of the company's officials were available to comment because of a company-wide summer vacation period."
    2. Re:RTFM is the fix? by drew · · Score: 4, Insightful

      In many European countries there is a month long period where everyone goes on vacation. As I understand it, pretty much the entire country except for basic service industries shuts down for a month. I don't understand the specifics, as I've never been to Europe during a vaction, but I did work on a project once with SwedenPost (the Swedish Post Office) that ended up being pushed back quite a bit because the original project schedule had us going into client QA right as the entire company took a month and a half off for vacation. And this was the Post Office!

      So I doubt that the fact that all of the company's officials were on summer vaction at the same time reflects on their abilities to design complicated hardware. It's just business as usual over there.

      And as the article points out, NASA probably could have gotten the specs if they had signed an NDA but they didn't believe they were necessary. Given that statement, it's quite possible that no one would have looked at the specs close enough to notice the problem, even if they had them.

      --
      If I don't put anything here, will anyone recognize me anymore?
    3. Re:RTFM is the fix? by twiddlingbits · · Score: 5, Informative

      NASA Has an Independant Verification and Validation Center to provide technical oversight. I worked there (it's in West F'ing Virgina of all places). The folks there do a great job with what data they are given. Often to save costs, this IV&V team is not even allowed to participate in the Design Reviews, and when they are and discover issue the Project Offices sweep them under the rug. No use admitting to problems that might show someone is not thinking correctly or is not managing the project well. The theory is "Let's avoid the problems by witholding information and communication from anyone who might find a problem". Solid testing? Thats a joke too, it costs money to test things well, and who knows they might break!. Contractors will make mistakes, after all they employ humans, but the mistakes can be corrected BEFORE flight if they are found. Having an extra set of eyes, and doing extensive testing is valuable but costly. In FACT having IV&V on manned systems is the LAW since the Challenger disaster, it's just commonly disregarded at NASA for anything but ISS. Even STS has no IV&V, after all it's a "mature" system and there are no bugs left. Regardless of what you hear about NASA "changing" after Columbia it really isn't. I fully expect another STS disaster, and several more mini-disasters in unmanned systems in the near future.

  2. Not quite by palad1 · · Score: 4, Funny

    RTFA will fix the probe.

    1. Re:Not quite by eingram · · Score: 3, Funny

      So you're telling us the probe is doomed?

  3. Save yourself from RTA, the fix is: by Tibor+the+Hun · · Score: 5, Funny

    reversing the polarity of the transponder...
    duh...

    --
    If you don't know what AltaVista is (was), get off my lawn.
  4. Lots of amazing stuff by erick99 · · Score: 5, Interesting

    It is amazing that the problem with the reciever was detected. It was more amazing to read what they went through to document and present the problem. It also says something about the relationship between NASA and it's subcontractors when they can accept a receiver design and not sign a standard non-disclosure agreement so that they can see the specific design elements. If they had done so, they would have been able to see the problem before launch. However, having read the article, the complexity of the mission is such that I am possibly more amazed that more didn't go wrong.

    --
    http://www.busyweather.com/
    1. Re:Lots of amazing stuff by erick99 · · Score: 4, Informative

      They were supposed to run a simulation, as one of three safety nets to catch such problems, but decided not to because of the cost.

      --
      http://www.busyweather.com/
    2. Re:Lots of amazing stuff by cellocgw · · Score: 3, Insightful

      They were supposed to run a simulation, as one of three safety nets to catch such problems, but decided not to because of the cost.
      Which doesn't make sense: did nobody at NASA have the brainpower to conceive of sending an emulated signal just like the one they actually ended up using? How much could it have cost to run a few hours' testing of Cassini's commlink prior to assembly of the craft? It's *always* a good thing to check system components in a full emulation environment.
      I think there were many problems, and one of them was that the system (or system test) engineers didn't stop to think of the Q&D way to get some proper failsafe testing done.

      --
      https://app.box.com/WitthoftResume Code: https://github.com/cellocgw
    3. Re:Lots of amazing stuff by erick99 · · Score: 4, Insightful

      To run the test they would have had to dismantle part of the craft and then go through an expensive recertification process to put it back together. Apparently the cost for all of this was very, very high. Probably not as high as the fix for this problem, though.

      --
      http://www.busyweather.com/
    4. Re:Lots of amazing stuff by cft_128 · · Score: 3, Informative
      Which doesn't make sense: did nobody at NASA have the brainpower to conceive of sending an emulated signal just like the one they actually ended up using? How much could it have cost to run a few hours' testing of Cassini's commlink prior to assembly of the craft? It's *always* a good thing to check system components in a full emulation environment.

      s/NASA/ESA/g

      NASA was only observing - this part of the project was pretty much run by ESA. Still a "cock-up" all around.

      --

      Underloved Movies and Pub Quiz: donotquestionme.org

  5. farsighted by Doc+Ruby · · Score: 4, Funny

    Installing the reentry sensor upside down, ignoring the Doppler effect - this rocket science stuff is so hard, they're missing all the easy stuff.

    --

    --
    make install -not war

    1. Re:farsighted by hopemafia · · Score: 4, Interesting

      It's actually fairly common...when you have a bunch of smart people working on difficult problems they tend to breeze through the easy parts, and don't necessarily double check each other's work because of "Jim has 2 PhD's...he'll get that right" syndrome.

      Reminds me of taking caculus exams...it was always something dumb like switching +/- or "1+1=3" that I got wrong...not the partial differentials.

      It just shows that no matter how smart you are, if you hurry and don't pay attention to every trivial detail you'll make mistakes.

      --
      If God had had a computer it would have taken him 7 months to create the earth...if he even bothered to do it at all.
  6. Old news by Timesprout · · Score: 3, Informative

    The problem was discovered years ago, took 6 months to investigate and nearly 2 years to resolve. The BBC told us all about it with nice graphics the other night.

    --
    Do not try to read the dupe, thats impossible. Instead, only try to realize the truth
    What truth?
    There is no dupe
    1. Re:Old news by orac2 · · Score: 4, Insightful

      I'm uptight because its annoying when you work on an article and people comment without bothering to read it -- yes, I know that's endemic to /., but it's still a pain in the ass.

      The point is that we dug up an aspect of the story you're not going to see any where else, let alone a general overview program, but a really cool story of a guy who deserves a lot of credit, Boris Smeds. I would hate for anyone to not bother to find out about him because a related program on the telly happened to be braodcast the night before /. decided to post the story.

      --
      "Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
    2. Re:Old news by EccentricAnomaly · · Score: 4, Interesting

      Boris was key to finding the problem, but the solution was mostly a trajectory one. The article has a rather mangled explanation of the trajectory changes, and it treats the people who worked on it rather anonymously.... just calling them 'navigators'. The fact is that the trajectory changes done to save the probe mission were far from trivial.

      Boris deserves a lot of the credit for saving Huygens, but several other people deserve credit but have been rather anonymous outside of ESA/JPL. 'Saving' Huygens was a team effort, and a lot of people played a part. There are a handful of other key people that the article doesn't mention at all.

      Also there are a few factual errors in the article.... NASA couldn't simply sign a NDA to get the specs for the receiver, and there was a lot of effort by NASA to get the specs. Even after the problem was detected, Alenia resisted sharing information for many months.

      --
      There are 10 types of people in this world, those who can count in binary and those who can't.
    3. Re:Old news by orac2 · · Score: 4, Interesting

      I agree the recovery was a team effort, but the fact remains that what Smeds did was a rarity: a singular individual effort that, if it hadn't occured, would have resulted in disaster. Thus we felt he deserved some serious kudos and so the article focused on him.

      I'm in no way deingrating the amazing and creative work that the trajectory guys did. But think of it like this: If any one of those guys were absent from the project, because of a sabbatical, or, God forbid, an accident, chances are that the mission still would have been salvaged.

      The same cannot be said about Smeds during the period between being told to do a test and coming back to ESA with the results -- it's fair to say that many, if not most, engineers would have just developed a carrier wave only test as originally planned, or wouldn't have bothered to persist with the more complex test after being turned down (after all, who's looking to get into trouble to do extra work?), or might not have had the insight required to modify the test on-the-fly when the downlink started showing problems.

      The situation is analogous to Apollo in some ways: a lot of people helped design and build the LEM (Tom Kelly is one of my personal engineering heroes), but John Houbolt deserves his place in history for pushing NASA onto the LOR architecture path in the first place.

      As for what we said about NASA and the NDA, I'll just have to say we stand by Oberg's reporting. But if you have something that shows we really did get it wrong, I'd be more than happy to look at it and print a correction if warranted.

      --
      "Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
  7. classic by theMerovingian · · Score: 4, Funny


    "We have a technical term for what went wrong here," one of Huygens's principal investigators, John Zarnecki of Britain's Open University, would later explain to reporters: "It's called a cock-up."

    We Americans speak English, but this is proof positive that the British have had much more time to master the use of it :)

    --
    "If you think you have things under control, you're not going fast enough." --Mario Andretti
  8. We have a technical term for it by RealAlaskan · · Score: 5, Informative
    "We have a technical term for what went wrong here," one of Huygens's principal investigators, John Zarnecki of Britain's Open University, would later explain to reporters: "It's called a cock-up."

    Oooooh! I love that technical jargon.

    Spoiler Warning:

    ... the Cassini team crafted a response plan that centered on reducing the Doppler shift sufficiently to keep the data signal within the recognition range of the receiver. They accomplished this trick by altering the planned trajectory of Cassini.

    Now you know how they fixed it, so no need to read the article.

  9. Re:Obvious by erick99 · · Score: 4, Informative
    To be fair, it was not just NASA, according to the article this was

    a collaboration with the European Space Agency, Cassini, in addition to its own suite of scientific instruments designed to scan Saturn and its moons, carries a hitchhiker--a lander probe called Huygens.

    --
    http://www.busyweather.com/
  10. Re:Obvious by TykeClone · · Score: 3, Insightful
    What is wrong with NASA?

    As you answered in the previous paragraph - it's the non-scientist administrators.

    --
    A fine is a tax you pay for doing wrong and a tax is a fine you pay for doing all right.
  11. Re:what esa makes to people by erick99 · · Score: 3, Insightful

    The article says he is a 26-year EASA veteran, it does not say that he is 26 years old. Though, I thought the same thing on my first pass and had to re-read it.

    --
    http://www.busyweather.com/
  12. I could have helped out with this by kalpol · · Score: 5, Funny

    I drove a Fiat for years. I could have told them an Italian radio wasn't gonna work. :)

    --
    12:50 - press return.
  13. Re:NOOO!!! by Anonymous Coward · · Score: 3, Funny

    Given this is Slashdot I'd bet there are some that view Star Trek as absolute reality. Aparently some of them have mod points as well. *shudder*

  14. May not be that simple... by Smilodon · · Score: 4, Insightful

    This is not a simple subcontractor arrangement. It is cooperation between government agencies in different governments, each of which has private contractors working for them.

    Besides the obvious contractual nightmare this represents, there is also the issue of Export control between governments, which cannot be countermanded with a simple non-disclosure.

    IMNAL, but I work on a similar project and you need to learn some of this stuff, sadly, to get your work done. I'm hopeful this incident will help to clear up these sort of cooperation issues in the future.

    Good work in resolving this all involved! Remember Slashdotters, we explore to learn...

  15. Long vacation. by RobertB-DC · · Score: 3, Funny

    [NASA's] Horttor never got an explanation of why Alenia Spazio's telemetry system was built with a timing system that couldn't accommodate the Doppler shift in Huygens's telemetry. "It is a design feature of another application in Earth orbit, and they just reused it," he told Spectrum, adding, "I don't know why anyone would ever want to build it that way." (An Alenia Spazio spokeswoman said that none of the company's officials were available to comment because of a company-wide summer vacation period.)

    Anyone think that the "company-wide summer vacation" may extend a little longer than originally expected?

    "Hey, Tony! Glad to hear you ready for work. But why don't you go ahead and stay in Verona another month or two? Check out this web site while you're there. Ciao!"

    --
    Stressed? Me? Of course not. Stress is what a rubber band feels before it breaks, silly.
  16. Re:Doppler shifting radio waves? by dougmc · · Score: 4, Informative
    but don't you have to me going *really fucking fast* if you want to make any noticeable doppler shift in light?
    Yes and no. Ultimately, it depends on how fast you consider `really fucking fast' to be.

    Certainly, satellites around the Earth qualify -- if you want to be truly successful working with amateur (ham) satellites, you need to adjust for the doppler shifts., especially at the higher frequencies. If you don't, you'll only be successful when the doppler shift is small -- basically, when the satellite is as high in the sky as it's going to get in this pass.

    Suppose your signal is at 441 mHz, and the signal is only 20 kHz wide. It only takes a 0.005% shift in the frequency to move that signal 20 kHz so you can't detect it at all, and doppler shifts seen by objects in low Earth orbit satellites can be a good deal larger than that.

  17. Clever Solution by SparksMcGee · · Score: 5, Informative

    It looks like the relative velocity of Cassini to Huygens actually *was* high enough to lend a singificant Doppler shift, so correspondingly the data rate was massively compressed--like the frequency of a racecar coming towards you getting higher and higher, except in this case its bitrate instead of sound. The antenna was only designed to "listen" for a fairly static bitrate --like if once the car got close enough and the sound frequency high enough you just stopped hearing it. So instead they're altering the flightpaths so that Cassini is now far enough away from Huygens that the broadcast vector is mostly perpendicular, with minimal Doppler shift -- think about standing very far away from the racetrack instead of right in front of the car. The total distance between you and the car changes by much less, so you hear more of a constant hum than a higher and higher frequency, analogous to the drone of a jet plane passing far overhead. Because the Doppler shift is minimal, the antenna can now receive data at a nearly constant bitrate it can handle. Very nicely done.

  18. SDR by wowbagger · · Score: 4, Informative

    What I find hard to beleive is that the data slicer for the radio was not a chunk of code running on a processor, rather than a hardwired circuit.

    I do SDR (Software Defined Radio) for a living - doing a data slicer like this isn't very hard at all. Why they couldn't just reprogram the slicer to take into account the bit timing shift - or better still, why weren't they resyncing on the zero crossings of the signal so they could deal with bit timing errors automatically?

    Hell, for that matter why don't they have an option to route the recovered signal verbatim to the main transmitter and send that to earth - and do the signal processing here? NASA *used* to have the philosophy of "all the bits to earth" - the wouldn't even use lossless data compression lest the signal be corrupted and unrecoverable.

    1. Re:SDR by Anonymous Coward · · Score: 3, Informative

      Why they couldn't just reprogram the slicer to take into account the bit timing

      because as the article said, the firmware was not designed for being reflashed remotely.

    2. Re:SDR by Anonymous Coward · · Score: 5, Informative
      The signal strength is very weak. We'll be using the 100 meter Greenbanks telescope, as well as the VLA, Parkes and Mauna Kea radio telescopes to record the signal on the ground, but the primary plan is still the Cassini orbiter.

      This isn't the only screw-up for Alenia this mission. Look for articles involving the Ka Band Translator if you're interested. You may not find many, it hasn't been covered very publicly. Basically, we can no longer send a Ka band uplink to the spacecraft becasue the Alenia built receiver broke. See Paragraph 10 here

    3. Re:SDR by twiddlingbits · · Score: 3, Informative

      DSN is saturated and time slices are VERY hard to get. It needs to be upgraded, but there is not any money. NASA will spend 10's of millions on a probe but won't spend any on the data network to get the probe's data to the ground. In part, due to the saturation of DSN, most missions now have to have an on-board data recorder that holds anywhere from 24 hours to 7 days of data for compressed delivery when a slot is open. That adds costs and weight to every mission.

  19. Proprietary by Eryximachus · · Score: 3, Insightful

    A shining example of the promblems with proprietary design. No one can see what's wrong with it without expending a huge amount of effort. I'm just glad someone did decide to spend the effort.

  20. Slashdot needs a 'HERO' tag by mcrbids · · Score: 4, Interesting

    While slashdot has category icons, it really REALLY needs a "hero" tag, like you'd see on Fark.

    This engineer that found the problem and rallied against opposition to see that this gets fixed is, in my opinion, a total hero. The world would be a much better place if more people like him were around!

    --
    I have no problem with your religion until you decide it's reason to deprive others of the truth.
  21. Re:Doppler shifting radio waves? by Ruie · · Score: 4, Informative
    The key is "noticeable". Our hardware is very precise nowadays and the relative shift is approximately proportional to v/c for small v.

    The relative velocity was quoted as 5.5km/sec which means v/c=0.000018. This is not such a small number.

    Furthermore, even though the frequency changes little, the phase can shift a lot. The change in phase is proportional to v/c times the number of cycles in the segment you are examining - and there are a lot of cycles in 1/8192 second chunk of the microwave signal they are using to communicate.

    Lastly, the length of the transmission also matters - 2 hour transmission of 1/8192 sec chunks amounts to approximately 60 million chunks. If you multiply the doppler shift above by the number of chunks you get approx 1000 - i.e. the chunk timing will shift through completely 1000 times during transmission. (In other words you will be drifting in and out of sync with transmission rate 1000 times during descent.. A sure way to get most data scrambled)

  22. Re:the solution is so anticlimactic. by Fortran+IV · · Score: 3, Funny
    I know what you mean. I was expecting something about reprogramming Cassini or Huygens or both. But then I read:
    ...the firmware could not be altered after launch.
    What? A $300M mission, and there's no provision for firmware upgrades? Even my $40 wireless hub can get firmware upgrades. Oh, wait: "Do NOT upgrade firmware on any D-Link product over a wireless connection. Failure of the device may result. Use only hard-wired network connections."

    So I guess they'd have had to run a billion-mile cable first. Line noise would be a bitch, wouldn't it?
    --
    I figure by 2030 or so my 6-digit UID will be something to brag about.
  23. Re:Dont Bother Reading Long Article by orac2 · · Score: 5, Insightful

    Sure, everyone knows that a)something Doppler related went wrong with Huygens and b) they fixed it with "fancy flying", but that's like saying don't bother to read a history of World War II because everyone knows a) Hitler started it and b) the Allies won.

    The point of the story was to explain the problem with a level of accuracy and detail that was simply missing from most report and to tell the story of some stone-cold great work by an engineer, something of interest to most engineers, and I would hazard, to most slashdotters.

    As far as I am aware, no-one else has told the story of how Boris Smeds pushed through the comms test that showed something was wrong, despite intial rejection and then later, modified it on the fly to reveal the problem was Doppler related, saving months of delay. Learning about his example of how to be a great engineers is the article's real utility, not teaching Spectrum readers how to fix Titan landers.

    Disclaimer -- I edited this story for IEEE Spectrum

    --
    "Just once, I'd like to meet an alien menace that wasn't immune to bullets." -- The Brigadier, Dr. Who
  24. Whats wrong with Proprietary by Qzukk · · Score: 3, Insightful

    You mean "What is wrong with proprietary?" right? After all, the entirety of this problem was because NASA bought a black box proprietary technology, and without access to its specs could only pray that it performed as advertised.

    In this case, the black box didn't meet the required standards, but there was no way NASA could have known that this company built the black box out of off-the-shelf terrestrial design principles unfit for cosmic use.

    --
    If I have been able to see further than others, it is because I bought a pair of binoculars.
  25. RTFA by scribblej · · Score: 3, Informative

    Is *IS* a "software" radio and not hardware. It's implemented in "firmware" and they say they could have changed it easily, except there's apparently no way to do so after launch.

    The problem is they didn't find this problem until AFTER launch. good timing, right?

  26. Re:Doppler shifting radio waves? by mikael · · Score: 4, Informative

    The amount of doppler shift is proportional to frequency and velocity. But it wasn't the change in signal frequency alone, it was the change in length of data timing as well.


    The general equation is:

    fdoppler = (frest * velocity )/ c

    where:
    fdop = frequency after doppler shift
    frest = frequency before doppler shift
    velocity = speed of object relative to oberver
    c = speed of light


    Although radio waves have a longer wavelength (kilohertz/megahertz) than light (terahertz+), the
    effect is less noticable, but still significant.

    According to the article, the doppler shift was +/-38 Kilohertz. Given the fact that data was being transmitted on an 8/16 Kilohertz carrier wave, that's a rather significant change.

    This is enough difference to allow police speed radar traps to work, and for researchers to measure the wind speeds inside tornado's.

    --
    Vintage computer adverts: http://www.vintageadbrowser.com/computers-and-software-ads
  27. Great work by Smeds. by francisew · · Score: 3, Interesting

    Boris Smeds did a great job in replacing lots of expensive tests with a series of trivial, yet critical tests.

    Why weren't simple tests like these used while the spacecraft was on the ground?

    These are obvious problems. When you take a transmitter and throw it into a planetary descent, this is what should be expected.

    It is shocking to me that a transceiver pair isn't tested by the team assembling the spacecraft before launch!

    If it can be tested in 2 days when it's in space, 48 light-speed minutes away, why can't it be tested on the ground, fully assembled?

    Engineering isn't a science, but I expect that engineers desigining projects like this should be using thorough unbiased scientific testing, not only thorough design.

    If they slip up like this in non-destructive tests, one has to wonder about how tests on the resistance to physical damage are carried out?

    Do they simply make assumptions that all nuts & bolts are manufactured to spec? Do they assume that all parts will withstand the forces that they are requesting in spec sheets?

    How can a cost of hundreds of millions of dollars be justified in creating such craft, when basic, inexpensive testing isn't being carried out? If the test would cost 30,000$ (a few days of well-paid outside experts time, plus expenses and travel), as a critical portion of a 300,000,000$ mission, how is it not done?

    The only agreement that should be accepted by an agency purchasing a part is that they won't use the specifications of that part to replicate exactly the same device. I'm sure that they paid a high premium for the transceiver. Why wouldn't they have access to the documentation and spec sheets? This use of NDA's is dangerous.