Slashdot Mirror


Spirit Sends Debug Information to Earth

gfilion writes "NASA has released a press release that says: 'Shortly before noon, controllers were surprised to receive a relay of data from Spirit via the Mars Odyssey orbiter. Spirit sent 73 megabits at a rate of 128 kilobits per second.'" They've been having communications troubles with Spirit since Wednesday, so it's good to hear from it again, even if the data is just filler.

26 of 477 comments (clear)

  1. Spirit rebooting 60 times a day by G4from128k · · Score: 5, Informative

    CNN is reporting that spirit is self-rebooting 60 times a day. NASA suspects a hardware fault that is causing the processor to detect trouble and automatically reboot.

    --
    Two wrongs don't make a right, but three lefts do.
    1. Re:Spirit rebooting 60 times a day by cosmo7 · · Score: 4, Informative

      Mars is still Microsoft-free.

      Spirit uses Wind River's RTOS, VxWorks. The main computer (the Rover Electronics Module) uses a 20MHz 32-bit Rad 6000 CPU, a radiation-hardened PowerPC variant manufactured by BAe in England. The computer has 128M of ECC RAM and a 3M EEPROM. It connects with hardware via a Versa Module Europa (VME) bus.

      The software was compiled with a compiler from Green Hills' MULTI development environment, but the developers coded using the Wind River Tornado IDE.

    2. Re:Spirit rebooting 60 times a day by spaceyhackerlady · · Score: 5, Informative
      Something like 2/3 of NASA's recent missions have failed in some way or another. Is it quite possible that NASA engineers simply have not mastered the art and science of designing hardware and software operable in the harshest of environments?

      Maybe they have. That's how they know how difficult a task it is to get it right.

      I am something of an aerospace engineer, and work professionally with real-time systems (based on VxWorks - fancy that!). Let me illustrate the kind of bizarre bug that can happen on a spacecraft, and how it was fixed from the ground.

      Consider a satellite with a simple on-board computer. To guard against the OS locking up (no matter how good the software is, you can't protect against radiation-induced bit flips in memory), it has a hardware watchdog timer. The software resets the timer periodically, before the hardware can reboot the system. Things run well for a while.

      Then the on-board system starts resetting for no apparent reason. No suggestion of memory problems, no apparent hardware problems. The problem is traced to a radiation-induced change in component values in the watchdog timer, causing the timer to go off sooner than expected. Until the satellite is finally turned down a few years later, an important task of the ground stations was checking for watchdog resets and adjusting the software watchdog task accordingly. When the software eventually spent all its time resetting the watchdog timer, the satellite could no longer function and was turned down.

      The moral of the story: space is weird and hostile. Things happen. No matter how hard you try, you cannot always get it right.

      ...laura

  2. It wasn't exactly 'filler' by Eevee · · Score: 5, Informative

    Only a couple of frames were fillers of random values. Most of the frames were engineering data. No actual scientific data came down, though.

    Still, it's a good sign that it's still able to talk.

  3. Re:Linux Cost Tax Payers at least $410M...nothing by TheGrayArea · · Score: 5, Informative

    You might want to check your facts before you spew. While the ground system is heavy on Linux according to the article you referenced, the actual OS on the rover itself is VxWorks from Wind River.
    http://www.windriver.com/news/press/20040105.html

    --

    This space for rent.
  4. Re:Linux Cost Tax Payers at least $410M...nothing by Kumochisonan · · Score: 2, Informative

    IIRC... The 'Spirit' rover runs VxWorks.

    Why is it that you do not know this? mmm?

    --
    kill elrond
    take elrond
    put elrond in cupboard
  5. Re:Wow. by skillio · · Score: 2, Informative

    its actually 128 kbps to mars odyssey (its max throughput, incidentally)...the MO's high gainer tops out at 110 kbps. still, not too shabby, too bad it seems to be 95% crapola.

  6. Re:Tell the rover by Bill+Quayle · · Score: 2, Informative
    Actually, rebooting seems to be part of the problem (or at least one of the symptoms). According to an article on the BBC:

    It appears that every time Spirit tries to load the software it encounters a problem and then tries to re-boot

  7. vxWorks... by codepunk · · Score: 2, Informative

    From the windriver site....

    Power and versatility was delivered via the advanced applications developed for each of the robotic functions of the Rover devices, plus their communications links with the landing craft. VxWorks not only served as the ideal development platform for the engineers, it also had to be sufficiently robust itself to ensure it would perform according to plan under the extreme conditions on Mars and during the journey from Earth.

    To bad it never makes it to run level three sounds like init is dying..

    --


    Got Code?
  8. Re:Linux Cost Tax Payers at least $410M...nothing by sir_cello · · Score: 4, Informative


    Some of us Engineers work with RTOS all the time, not just for fun-and-dandy projects, for for multi-million dollar outcomes. Consensus is that Linux is not good enough. QNX, VRTX, VxWorks etc are still the preferred choices, but everyone admits that Linux is getting there. Most of us don't hang out on slashdot, yet many Linux zealots do: you don't get a good opinion here.

  9. Re:Can low-power corrupt memory? by funkhauser · · Score: 2, Informative
    IANANE

    I am not a NASA Engineer?

    Come on people, these stupid IANA* acronyms are getting out of hand. "IANA NASA Engineer" and such would be okay, because we know what IANA means. But using IANANE like it's an established acronym just makes you look stupid.

  10. Re:Mars Orbiter pictures of Spirit by AndroidCat · · Score: 2, Informative

    Argh! I thought I added the link (Preview is for whmips!)

    --
    One line blog. I hear that they're called Twitters now.
  11. Re:400 million and only one CPU by SagSaw · · Score: 2, Informative

    I saw over at the windriver site that this thing has a proprietary os

    I'm not sure how this is a disadvantage. The people at NASA can't be experts at everything, and in this case, it looks like they decided to hire an outside company to write the rover software. Just becasue it is a proprietary OS doesn't meant that the code is any buggier, that NASA can't review the software, or that there is any less ability to debug the thing when problems occur.

    and only on cpu and only one set of code

    A second CPU (or an extra anything for that matter) would add to the weight and energy consumption of the device. I think one reliable system beats out two redundant (but necessarily more complex) systems in this case.

    --
    Come test your mettle in the world of Alter Aeon!
  12. Re:Can low-power corrupt memory? by Alioth · · Score: 2, Informative

    Analogue circuits are hardly a silver bullet.

    On the subject of streetlights, I was travelling down a major highway the other day. Usually there's a light or two that's stuck on during the day wherever you go. I decided to count how many there were, so I counted 100 streelights out. Out of those hundred, 19 *were stuck on in broad daylight*. The waste of electricity must be phenomenal, since these are all bright high-pressure Na lamps.

  13. Fill Data by Detritus · · Score: 4, Informative

    Fill data is typically transmitted when the telemetry multiplexer does not have any engineering or science data to send. Due to the way synchronous communications links work, something is always being transmitted, even if there is no "real data" available.

    --
    Mea navis aericumbens anguillis abundat
    1. Re:Fill Data by Detritus · · Score: 2, Informative

      Normally, the bit rate of the telemetry transmitter is set to a value equal to the peak bit rate of the active telemetry sources. Depending on the design of the spacecraft, some sources may be constant bit rate and some sources may be variable bit rate. In traditional spacecraft designs, everything is constant bit rate and there is no need for fill data. In many newer spacecraft designs, various spacecraft subsystems produce telemetry packets which are multiplexed into a single stream that modulates the transmitter. Some science experiments are designed to vary their bit rate depending on external conditions. If you are looking for transient phenomena, like x-ray bursts, you may want to generate telemetry at a low rate when nothing interesting is happening. If a x-ray burst is detected, you can shift to a higher rate to capture the event in greater detail. Some spacecraft have many independent experiments, generating telemetry in a mix of constant and variable rates. If the sum total of the inputs is less than the output telemetry rate, the telemetry multiplexer must insert fill data to keep its output at a constant rate. The telemetry system's output bit rate can be set to a number of values so that it can adapt to current RF link conditions. Most spacecraft have a wide selection of telemetry modes to allow for varying requirements and conditions. These may range from very low bit rate engineering data via the omnidirectional antenna if the spacecraft is in trouble, to high bit rates via the high-gain antenna if everything is working properly and important science data is being transmitted. The engineers on the ground send commands to the spacecraft to set the appropriate mode for current conditions. The spacecraft itself may change modes if it detects a problem or if it hasn't received any commands in a specified period of time.

      --
      Mea navis aericumbens anguillis abundat
  14. Re:Wind river by AaronW · · Score: 5, Informative

    I wouldn't brag. I've been programming VxWorks for several years now and all I can say is it's a piece of crap for a complex system.

    VxWorks does not provide any memory protection (well, AE does, but it's so buggy nobody uses it).

    If a task dies, it does not clean up after it. All memory is global, i.e. any task can overwrite memory for any other task.

    Wind River couldn't even implement a decent malloc implementation. I had to replace it with Doug Lea's DLMalloc code (which glibc's malloc is based off of). It fragments horribly, and becomes increasingly slower the more free blocks exist.

    Just by replacing malloc, I brought the time down on our box from 50 minutes to under 3 minutes and went from tens of thousands of fragments to a couple of dozen.

    If you want a reliable embedded system with a lot of complexity, go with QNX or perhapse a good embedded Linux (I like Timesys Linux myself - good realtime support).

    At least with QNX if there's a problem in a task, it's much easier to isolate it and not kill the entire system. As it is on the product I'm working on, if a task dies about the only way to recover is to reboot. Also, VxWorks has piss-poor built-in debugging support. Sometimes you can get a stack trace. Tracing the heap is virtually impossible (and because it's a global memory pool, you don't even know what blocks were allocated by what task or even how much memory each task has allocated). In the product I'm working on I added such support to find memory leaks and detect memory corruption.

    VxWorks AE does provide memory protection. We tried to use it, but it was so buggy and slow we had to drop it and go back to standard VxWorks.

    VxWorks hasn't really changed in the last few years and Wind River is losing customers like crazy to the better alternatives. They're hemmoraging money at an astronomical rate and quickly losing market share to the likes of QNX and Linux.

    Even the realtime performance of VxWorks isn't that great. The finest granularity for a reliable timer is 1/2 the system tick rate (often no more than 20ms resolution).

    VxWorks doesn't have a shell as such either. The commands you type in are functions with parameters to those functions. You can do things like my_global = global_a + 7

    or

    my_func(&my_global, 3)

    on the command line, but it's not at all like a traditional command line.

    Most real-time Linux implementations arn't all that great either from my research into it. Most don't deal with priority inversion, or require a completely separate set of APIs for RT tasks (i.e. RT Linux). I found Timesys Linux to solve most of these issues and it looks like our next generation will be based off of either Timesys Linux or QNX.

    -Aaron

    --
    This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
  15. Re:Can low-power corrupt memory? by Jeremi · · Score: 2, Informative
    But using IANANE like it's an established acronym just makes you look stupid.


    OTOH, if nobody ever used an acronym unless other people were already using it, we'd never have any new acronyms at all. Slashdot culture would suffer terribly.

    --


    I don't care if it's 90,000 hectares. That lake was not my doing.
  16. (AXCH) MER2004 More Info by dekashizl · · Score: 2, Informative

    For news, status, updates, scientific info, images, video, and more, check out:
    (AXCH) 2004 Mars Exploration Rovers - News, Status, Technical Info, History.

  17. Re:Can low-power corrupt memory? by techiemac · · Score: 3, Informative

    There is a simple reason here...
    It's cheaper to send a space craft to Mars that it is to send it to the Moon believe it or not. When it comes to sending manned missions, that's another story. I would rather not get into all of the nitty gritty here since I have somewhere to be right now but if you are interested, hunt a little on the web about "gravitational slingshots" for lack of a better term.

  18. Re:got some useful data by SnowZero · · Score: 4, Informative

    On Earth at least, picking bits off of radio links usually involves an adaptive threshold and a clock that syncs to the clock of the sender. Sending too many 1's or 0's in a row can interfere with that because there aren't any "bit edges" on the signal. Sending random data ensures all patterns are equally likely and your adaptive filter stays happy for when you have real data to send. Otherwise you'll miss the first part while you re-establish the threshold and sync to the signal.

    My guess is the NASA rover's link follows a similar principle, though its probably using some pretty damn fancy techniques to get the data from that far. Oh and missing the first part of the data would really suck for them since a retransmit would take 20 minutes.

  19. 3pm EST JPL news conference by Anonymous Coward · · Score: 2, Informative

    -downgrade from critial to serious
    -3 types of memory: random(lost every night), flash(science data, etc), double eprom(program data. harder to write to).
    -cripple mode- run without the flash ram
    -Sent commands to put spirit into cripple mode. No more resets so far. It can also sleep now.
    -Will relay entire contents of flash ram to mars observor over the next day or so. Hopefully that'll give clues as to what happened.
    -Since cripple mode means no permenant storage, spirit forgets it's in cripple mode everytime it goes to sleep.
    -proabably about 3 weeks until it's back up running to any significant degree.

  20. Re:mars dvd message by 222 · · Score: 2, Informative

    Yea, I actually asked ( http://www.elonka.com )Elonka of Kryptos fame about this yesterday, it seems someone has indeed cracked it, although im not sure if any of them have come forward.
    Although i havent personally worked on this, it really seems like something that nasa put together for middle school students looking for something fun to do, not something any experienced codebreaker would have trouble poking through.

  21. Re:Wow by Anonymous Coward · · Score: 1, Informative
  22. Re:sure would be nice.. by dekashizl · · Score: 2, Informative

    Go to your Slashdot Preferences, click on Comments tab. Or just click here.

    You can adjust the modifiers for all adjectives. The default is 0 (which is +1 for positives and -1 for negatives), but you can, e.g., make Funny == -4.

    I don't recommend it, though, because then you might miss funny Mars cartoons like this one.

    --
    For news, status, updates, scientific info, images, video, and more, check out:
    (AXCH) 2004 Mars Exploration Rovers - News, Status, Technical Info, History.

  23. Re:ISDN to mars by chiph · · Score: 2, Informative