Slashdot Mirror

← Back to Stories (view on slashdot.org)

NASA To Repair Hubble By Remote Control

Posted by CmdrTaco on from the in-my-day-i-had-to-walk-over-and-do-it-manually dept.

Matt_dk writes "NASA says it plans to fix the Hubble Space Telescope by remote control this week. The Hubble stopped beaming information to Earth about two weeks ago, when a data unit on the telescope completely failed. Scientists on Tuesday said they will bypass the failed unit and switch to a back-up system to restart the flow of information. The computer glitch forced NASA to postpone a shuttle mission this month to repair the Hubble. That shuttle mission has been postponed until next year." Update - 10/15, 17:45 by SS: Readers have pointed out further details from Spaceflight Now and the NASA press release.

53 comments

  1. They're gonna need by krog · · Score: 4, Funny

    Some really fresh AA batteries to make this work.

    --
    Cretin - a powerful and flexible CD reencoder
    1. Re:They're gonna need by rtboyce · · Score: 1

      ... or a really hot cup of tea.

    2. Re:They're gonna need by Anonymous Coward · · Score: 0

      My remote control takes AAA batteries, you insensitive clod!

  2. MST3K by staryc · · Score: 3, Funny

    It's about time someone fixed the Hubble after Mike Nelson crashed the Satellite of Love into it.

    --
    The most perfidious way of harming a cause consists of defending it deliberately with faulty arguments. - Nietzche
  3. So, a better summary... by txoof · · Score: 4, Insightful

    NASA will flip a switch and kick in the backup system.

    The story is pretty light on details. It reads like a 6th grader wrote it.

    --
    This one's tricky. You have to use imaginary numbers, like eleventeen... --Hobbes
    1. Re:So, a better summary... by afidel · · Score: 1, Insightful

      Yeah most likely it will result in degraded performance and/or loss of functionality, just like the initial 'fix' for the bad mirror was to avoid using the most sensitive sensors. In a way it's very fortunate that this part broke before the service mission since now they know they will need to fly the ground spare and replace it along with the other scheduled repairs.

      --
      There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
    2. Re:So, a better summary... by $RANDOMLUSER · · Score: 1, Interesting

      I don't get the whole thing. I mean, why does it take two weeks to say "Main system offline, switching to backup."?

      --
      No folly is more costly than the folly of intolerant idealism. - Winston Churchill
    3. Re:So, a better summary... by CrackerJackz · · Score: 1, Insightful

      I think it has to do with the level of cost of a failure, its one thing to have a system fail, but its more complicated then the trip alone costs 10+ million dollars

      my guess is 5min to send the commands, 1.9weeks to diagnose what exactly failed, and how to repair it...

    4. Re:So, a better summary... by Anonymous Coward · · Score: 0

      SOAP over a 300 baud acoustic modem?

    5. Re:So, a better summary... by Anonymous Coward · · Score: 5, Informative

      Actually, the part that failed has a backup, it's almost like a two-sided cassette tape, and that second side needs to be loaded into the device that is responsible for transmitting data.

      The broken part right now stops any and all data transmission from Hubble. Performing the fix (flipping the tape) will restore all performance, with no degradation, other than no backup.

      When the next shuttle mission launches to repair the Hubble, they will be replacing this part so that there is, again, redundancy.

      It's pretty amazing that these parts are still working after...how many years?

    6. Re:So, a better summary... by Anonymous Coward · · Score: 0

      The backup system has been unused since launch, it has never been used.

    7. Re:So, a better summary... by Anonymous Coward · · Score: 0

      It's Slashdot, probably was!

    8. Re:So, a better summary... by Kjella · · Score: 4, Insightful

      Because the point is to avoid catastrophic failure. Imagine it was something like say a solar flare, backup systems online and poof goes the backups too. Or there is some form of short circuit that'll fry the backups too. Or just figuring out what the failure state is and how to best handle it. To take an example, say a Mars rover wheel is busted. How? Is it stuck? Has it lost the drive? Can it turn? Is the wheel itself torn? Is it just a sensor malfunction? You need to be 100% sure what state Hubble was in when it failed in order to be sure to recover properly. On earth it's really easy to throw a lot of real redundancy into things, in space it's still one device more or less and you try to figure out if the right side is safe when the left side is on fire. Most anywhere else it's either functioning correctly or you'd kill it and replace it with something that does. Trying to salvage half-borked systems only happens when they're really expensive or really hard to reach, and I think the Hubble qualifies on both.

      --
      Live today, because you never know what tomorrow brings
    9. Re:So, a better summary... by davolfman · · Score: 1

      Because they're NASA?

    10. Re:So, a better summary... by hairyfeet · · Score: 1

      Because this is one of those "Cross your fingers and hope this works" kind of deal? NASA doesn't really know if they switch to the backup it will work(after all,it has been out there for a LONG time) and without a ready to go replacement(I read that the replacement which has been in storage since '91 is screwing up) this is pretty much it. But at this point they don't really have a choice. So either they try it and MAYBE get a working sat back,or they don't and get squat. So I can understand the guys at NASA being nervous. After all we are talking about a bird so old that it uses core memory.

      --
      ACs don't waste your time replying, your posts are never seen by me.
    11. Re:So, a better summary... by PolygamousRanchKid+ · · Score: 4, Funny

      It's pretty amazing that these parts are still working after...how many years?

      Maybe NVIDIA needs to have a chat with NASA about quality assurance.

      Well, that and Tang, of course.

      --
      Schroedinger's Brexit: The UK is both in and out of the EU at the same time!
    12. Re:So, a better summary... by saigon_from_europe · · Score: 1

      Maybe NVIDIA needs to have a chat with NASA about quality assurance.

      I hope NVIDIA won't start producing O-rings any time soon.

      --
      No sig today.
  4. Fixed! by PearsSoap · · Score: 2, Funny

    Hubble

    There, fixed that for you, NASA.

  5. Doesn't make any sense... by gfxguy · · Score: 0

    ... except when it's the government running things.

    So... a mission to fix something is delayed because that thing is broken?

    --
    Stupid sexy Flanders.
    1. Re:Doesn't make any sense... by camperdave · · Score: 2, Insightful

      Boss: Okay team. You have all you need to fix the Hubble. How are you feeling?
      Shuttle pilot: Good to go, sir
      Mission specialist: Hoo-ah!
      Boss: Right! You're scheduled to launch...
      [Underling comes rushing in.]
      Underling: Mr Houston, we've had a problem.
      Boss: What sort of problem?
      Underling:The AE-35 unit on the Hubble just went to 100% failure.
      Boss: How long will it take to prep a replacement?
      Underling: Let's see... A week to order the part... three to five weeks of testing... decontamination and clean room testing... about two months, give or take a few days, sir.
      Boss: Damn. That puts us too close to the end of the year. Well, boys! It looks like the mission will have to be postponed until next year.
      Shuttle pilot: We'll be ready, sir.
      Mission specialist: Hoo-ah!

      --
      When our name is on the back of your car, we're behind you all the way!
  6. Finally! by TheNecromancer · · Score: 3, Funny

    They're going to get some use out of that old Atari joystick that's been sitting in the office!

    --
    Attention all planets of the Solar Federation! We have assumed control! - Neil Peart
  7. I'm curious... by apodyopsis · · Score: 4, Interesting

    I'm curious, I presume somebody knows this.

    A satellite is the ultimate inaccessible device running SW. Any task that goes wrong has the chance of bricking a device that cost many many millions, so they *must* practice and check all commands sent to it when things go wrong.

    Do they have several mock ups?

    A complete computer model of the whole thing, emulated right down to hardware and software?

    How are reboot/reprogram sequences like this handled/practiced/tested?

    Even at design stage I imagine failure modes are extensively analyzed and multiple redundancy built in.

    My company builds stuff that goes up masts and is generally quite inaccessible and we always attempt to prove these things first, but we had fast serial communication, low level boot loaders under all the SW and if the worst comes somebody can climb the mast.

    Anybody know how space tech is handled?

    On a kind of related note, google for "expensive software errors" - most of the top ten are space related...
    .. or just look at this http://en.wikipedia.org/wiki/List_of_notable_software_bugs

    1. Re:I'm curious... by Utini420 · · Score: 2, Interesting

      Back during the Space Race, yes, they had full mock-ups, right down to the last screw. Many (probably not all) were themselves space-worthy.

      Nowadays, I kinda doubt it. My hunch would be computer simulation including hardware, but its hard to cite "my gut" as a credible source.

      --
      A little inaccuracy sometimes saves tons of explanation.
    2. Re:I'm curious... by Anonymous Coward · · Score: 1, Informative

      A ton of IV&V is done on the hardware before it is sent into space, some is on the actual hardware or a mockup (utilizing either equivalents or actual orbital hardware) of the hardware. Much of what's done today is done using model based integration and test procedures. This can be done either via a complex computer program, which simulates all the hardware and software, as well as inputs, outputs, triggers, etc, or on a representative hardware suite with simulated inputs.

    3. Re:I'm curious... by stuckinarut · · Score: 2, Informative

      The following is shamelessly plagerised from the following two articles; NASA to reboot Hubble Space Telescope and Hubble replacement part has glitches of its own

      Engineers plan to send commands to the telescope switch over to a backup computer that has not even been turned on since before the telescope arrived in orbit 18 years ago. However there's very little ageing that goes on with an unpowered component in space. It's actually a very benign storage environment.

      The errors were found in Hubble's science data formatter, which relays data between Earth and the probe's science instruments. There is an identical formatter â" known as 'Side B' â" on the telescope, and NASA is planning to boot up that backup system. That entails switching not only the telescope's formatter but also six other units over to their B sides, a process that is expected to take 47 hours.

      Ground testing of the switchover was completed on Monday, but the Hubble team is still awaiting approval from top NASA officials to make the switch.

    4. Re:I'm curious... by redKrane · · Score: 1

      On a side note, I wish more people were as smart as you in realizing that one's gut is not a credible source. Well said sir.

      --
      that's my word, holla...
    5. Re:I'm curious... by pnewhook · · Score: 4, Informative

      A satellite is the ultimate inaccessible device running SW. Any task that goes wrong has the chance of bricking a device that cost many many millions, so they *must* practice and check all commands sent to it when things go wrong

      Yes they practice everything on a mockup, but there is no chance that you can 'brick' Hubble. Proper spacecraft design means you have a completely separate system able to intercept telemetry and reprogram the main computer if an undiscovered bug happens to lock it up.

      Do they have several mock ups?

      Yes, both for software and a full mechanical mockup.

      A complete computer model of the whole thing, emulated right down to hardware and software?

      Partially, yes. But you can never em ulate the real thing perfectly which is why they usually rely on physical mockups.

      How are reboot/reprogram sequences like this handled/practiced/tested?

      No simple answer to that. But check out NASAs system and software design guidelines, or better yet take a spacecraft design course at your local college or university if they offer one.

      Even at design stage I imagine failure modes are extensively analyzed and multiple redundancy built in.

      Yes - very extensively. Anything that goes into space is analyzed for years. You have no idea how expensive and time consuming this can be until you actually work on one of these projects.

      Anybody know how space tech is handled?

      Yes - expensively and with great rigor. There are no shortcuts.

      --
      Tesla was a genius. Edison however was a overrated hack who liked to torture puppies.
    6. Re:I'm curious... by Deadstick · · Score: 0, Redundant

      Do they have several mock ups?

      They have actual duplicate examples of onboard units, as well as "breadboard" versions built for easy access to the innards.

      A complete computer model of the whole thing, emulated right down to hardware and software?

      Betcher sweet ass.

      How are reboot/reprogram sequences like this handled/practiced/tested?

      Endlessly.

      Even at design stage I imagine failure modes are extensively analyzed and multiple redundancy built in.

      Yes they are. But before switching in a redundant unit, you want to be very sure you know exactly what happened where. The last thing you want to do is to "switch into a short".

      rj

    7. Re:I'm curious... by decsnake · · Score: 5, Informative

      I'm curious, I presume somebody knows this.

      I don't work on hubble but I know something about this stuff and I'll try and answer your questions directly

      Do they have several mock ups?

      Yes, for hubble there are several mock-ups, from ones that are fairly low fidelity that are used by the software developers (maintainers) for code/debug/test to a very high fidelity full scale electrical and mechanical mock-up of the aft end of the vehicle called the VEST, when the astronauts practice the repair tasks on dry land before they move to the pool at JSC to learn to do them in a simulated 0g environment.

      A complete computer model of the whole thing, emulated right down to hardware and software?

      When HST was built we were still doing spacecraft control simulations on hybrid analog/digital computers. For all missions in the last 20 years or so there are computer models of all the control modes built using products like matlab/simulink long before any metal is cut. I wouldn't be surprised if there is a matlab (or similar) model of hubble now as the hybrid computer is long gone. In fact, I'm almost positive, as I don't see any way they could have come up with the 2 gyro science controller otherwise.

      How are reboot/reprogram sequences like this handled/practiced/tested?

      Not to be a wise guy, but reboot and reprogramming are handled very carefully. This is why switching to a backup system is going to take 2 weeks, most of which were used for analysis and a formal review before the decision to swap to the B side was made. It probably takes 5 minutes to send the actual commands to switch to the B side of the data formatter, but they will double, triple or quadruple check everything as they go thru the process. Remember, they are switching to hardware that hasn't been used since 1990. They expect it to take about 40 hours to switch to the B side and plan to be done by Friday.

      Even at design stage I imagine failure modes are extensively analyzed and multiple redundancy built in.

      There are failure modes and effects analysis done at each design step. Before launch there would have been a peer review of the final failure detection and correction design.

      A lot of the NASA standards are available to the public. If you go to http://standards.nasa.gov/ and click on the public button it takes you to the listing of them.

    8. Re:I'm curious... by Anonymous Coward · · Score: 0

      I'm curious, I presume somebody knows this.

      You presume correctly, but those guys are busy now. I will answer for my borther-in-law and hopefully I won't garble it too much, I've been out of the space biz since Reagan.

      A satellite is the ultimate inaccessible device running SW. Any task that goes wrong has the chance of bricking a device that cost many many millions, so they *must* practice and check all commands sent to it when things go wrong.

      Do they have several mock ups?

      Yes.

      A complete computer model of the whole thing, emulated right down to hardware and software?

      Yes. At Goddard.

      How are reboot/reprogram sequences like this handled/practiced/tested?

      Carefully. There are multiple levels of deadmen and heartbeats involved - all systems will kick into a safe mode if they don't handshake properly. The ultimate safe mode on the Hubble actually shutters all the instruments and realigns the whole bird to present the smallest possible cross-section to the direction of travel.

      Even at design stage I imagine failure modes are extensively analyzed and multiple redundancy built in.

      You imagine reasonably correctly.

      My company builds stuff that goes up masts and is generally quite inaccessible and we always attempt to prove these things first, but we had fast serial communication, low level boot loaders under all the SW and if the worst comes somebody can climb the mast.

      Yeah, that's similar, but climbing the mast into orbit is really, really expensive.

      Anybody know how space tech is handled?

      Nowadays it's mostly massively computerized modeling, they don't build geographically separated physical models any more.

      On a kind of related note, google for "expensive software errors" - most of the top ten are space related... .. or just look at this http://en.wikipedia.org/wiki/List_of_notable_software_bugs

      One of the primary cost justifications for the Space Shuttle was the ability to get human repairmen access to satellites.

    9. Re:I'm curious... by recharged95 · · Score: 1
      MIL-STD-498

      Of course, that was the gold standard when the Hubble was built. That's how I remember it. It was a unexciting, but guranateed process development practice.

      Nowadays (since 1995 when DoD went to better/faster/cheaper mentalityand threw out the MIL-STD standards!), it a watered down version using RAD, XP, Agile, and SOP since software development has gone internet style.

      Hence why [space vehicle] failures are up almost 30% (in gov't, more in commercial) since they dropped the standards. Here's a great article on how the industry plans to correct the increasing failure rate. Now I know why I got out of the biz.

    10. Re:I'm curious... by Anonymous Coward · · Score: 0

      A lot of the NASA standards are available to the public.

      Oh. Well then I've got to ask what ones aren't, and why?

      Probably something obvious like work in conjunction with mil projects, but the answer could be more interesting so thought I'd ask. Presumably there's a standard or guideline describing what standards can and cannot be made public. Is it available?

    11. Re:I'm curious... by thaWhat · · Score: 1

      I'm not sure why you posted anonymously one of the more informative entries in this thread. Might I just add that if I were responsible for switching over to a redundant system especially if I were uncertain as to the exact failure, I would be going over the software and hardware until I was certain of what was going to happen when I did.

      For example: The primary system failed due a malfunction in the breakfast dispenser, the first thing that I would want to do is to ensure that the redundant system doesn't suffer the same fate, otherwise you're up the same creek and have just thrown your remaining paddle overboard. Subsequently, I'd be ensuring that the instruction sequence would be, for example:

      • instruction_1:switch_to_backup.
      • instruction_2:forget_the_breakfast_dispenser.

      After all, if the redundant system fails in such a way that the ultimate failsafes don't - or can't - do their job, you have just bricked $55 million in hardware and as mentioned a service call isn't a one hour turnaround here.
      Here's one I prepared earlier:

      I got up at 4am to be on a 7am interstate flight to arrive onsite at 9am only to discover that neither the tools nor the spares existed to effect the repair *crickets... crickets*. Some embarrassment and some $300 out of pocket later I flew home. In space, radio-shack is not a ten minute trip away. If you don't have it with you, then your trip is a very expensive exercise in futility. I don't blame NASA for playing things carefully. After all, if you can fix it over the 'phone, rather than having to be there, I'm in favour of the 'phone call.

      C:\>

      --
      If all you have is a hammer, everything looks like a thumb.
  8. But what they don't know .... by RoverDaddy · · Score: 2, Funny

    is that Hubble will burn out the transmitter relay before NASA can send the command sequence requesting all its science data through the backup system, thus requiring The Creator to appear in person to complete the sequence. Calling Story Musgrave!

    --
    RETURN without GOSUB in line 1050
  9. Stupid to call a car repair man to fix your stereo by Anonymous Coward · · Score: 2, Informative

    The mission was to repair some other equipment and to install some new instruments and gyros and such. However, if the onboard computer isn't working, it's not worth it to send up anyone to add the rest of the stuff. They might as well wait, retrain the people to install a new computer, so that when they do send up people to repair Hubble, they can do it all in one shot.

    The Shuttle program is ending. We can't just keep sending astronauts to Hubble like we could have in the 90's. Now we either do it all at once, or not at all.

  10. Re:More money down the drain by Anonymous Coward · · Score: 0

    Hey! Those dorks as you call them were really creative in the way they stuffed things up. They deserve some kind of reward, like $480 million and maybe a Spa holiday in the Caymans.
    I bet you couldn't do better!

  11. Programming in Hollywood by kungfugleek · · Score: 1

    I bet ya the command they send up is "Delete Virus".

  12. NASA press release by Anonymous Coward · · Score: 0

    http://www.nasa.gov/mission_pages/hubble/servicing/SM4/news/status_update_20081014.html

  13. A more detailed look at this process by Anonymous Coward · · Score: 0

    A better story is found at the following URL:

    SpaceFlightNow.com

    This one talks about how the craft is wired, and explains in more detail how this failure is going to affect the systems on-board during the switch-over, as well as some of the challenges they're facing.

  14. Cosmic rays? by wfstanle · · Score: 1

    You have to remember that the Hubble Space Telescope has been in orbit for at least 15 years. (I could look up the actual amount of time but it is unimportant.) The probably has been a lot of exposure to cosmic rays that can damage electronics. The damage can be mitigated somewhat by proper shielding. The only disadvantage is that this is heavy so you would want to keep this to a minimum.

  15. It's a workaround not a repair, and being careful by SleptThroughClass · · Score: 1

    Because there is a possibility that switching to the backup will permanently break something. They wanted to be sure they knew what was broken before trying to work around it. There is a possibility that when "flipping the switch" that the switch will break or something else will permanently fail. It would be more than awkward to later figure out that there was a simpler solution which was no longer available.

  16. does anyone else think.... by roc97007 · · Score: 1

    ...this summary was a little condescending?

    --
    Oliver's law of assumed responsibility: If you're seen fixing it, you will be blamed for breaking it.
  17. What about by tekproxy2 · · Score: 1

    What about if they divert power from the plasma relay decoupler and reconfigure their Heisenberg compensators to sync with Hubble's warp signature? Hell, any Ensign would think of that.

  18. VGERrrrrrr VGERrrrrr by Anonymous Coward · · Score: 0

    VGERrrrrrrrr

  19. By RC? THAT'S why my garage door is going nuts! by Trip6 · · Score: 1

    Obligitory.

    --
    I hate being bipolar; it's awesome!