Slashdot Mirror


Galileo Satellites Are Experiencing Multiple Clock Failures (bbc.com)

elgatozorbas writes: According to a BBC article, the onboard atomic clocks that drive the satellite-navigation signals on Europe's Galileo network have been failing at an alarming rate. From the report: "Across the 18 satellites now in orbit, nine clocks have stopped operating. Three are traditional rubidium devices; six are the more precise hydrogen maser instruments that were designed to give Galileo superior performance to the American GPS network. Each Galileo satellite carries two rubidium and two hydrogen maser clocks. The multiple installation enables a satellite to keep working after an initial failure. All 18 spacecraft currently in space continue to operate, but one of them is now down to just two clocks. Most of the maser failures (5) have occurred on the satellites that were originally sent into orbit to validate the system, whereas all three rubidium stoppages are on the spacecraft that were subsequently launched to fill out the network. Esa staff at its technical centre, ESTEC, in the Netherlands are trying to isolate the cause the of failures - with the assistance of the clock (Spectratime of Switzerland) and satellite manufacturers (Airbus and Thales Alenia Space; OHB and SSTL). It is understood engineers have managed to restart another hydrogen clock that had stopped. It appears the rubidium failures 'all seem to have a consistent signature, linked to probable short circuits, and possibly a particular test procedure performed on the ground.'"

37 of 195 comments (clear)

  1. CIA inside job by Anonymous Coward · · Score: 3, Interesting

    To make sure there is no concurrence from another global measuring system. CT ? Possibly. But is that far fetched ?

    1. Re:CIA inside job by anti-pop-frustration · · Score: 5, Insightful

      Well I couldn't help thinking about the Iranian civilian program and the 'alarming rate of failure' of their centrifuges.

      Are the contractors who worked on Galileo going to discover some kind of Stuxnet variant on their network?

      Technically, it's possible. Thanks to work of security researchers we know it happened before, Stuxnet is well-documented. And thanks to Edward Snowden and the journalists who reported on the documents he leaked, we know the NSA/TAO does in fact hack allies.

      I wish these kind of doubts could be instantly discarded as conspiracy theories, unfortunately, this is not the world we live in. The most technically capable nations (USA/China/Russia/Isreal) have made the choice of using hacking as a weapon rather than helping secure the systems used by their citizens.

    2. Re:CIA inside job by GuB-42 · · Score: 4, Informative

      GLONASS and Beidou work fine.
      I suspect that these would be the most likely targets should the CIA want to disable competing systems. Europe is supposedly an ally, Russia and China, not so much.

    3. Re:CIA inside job by enriquevagu · · Score: 5, Interesting

      I do not work for Galileo, but I know some people involved in the Project.

      Interestingly (and unfortunately) the entities most interested in a failure and subsequent delay are neither millitary enemies nor allies (for whatever "ally" means, when you consider hacking your allies). The largest interest in a project delay comes from the many corporations directly involved in its development. This project has been feeding many millios of taxpayer euros during many years to a lot of European tech companies, and the cost uprising has been actually benefitial for them because there were no substantial economic sanctions from these delays (probably an example of a wrongly managed project). Successfully completing the project would kill this guaranteed job & income. I do not claim that delays and increased costs have been intentional, but they have been definitely benefitial for them.

    4. Re:CIA inside job by Cochonou · · Score: 2

      ESA contracts are fixed price. Even if there are no penalties, you do not gain any benefit by being late: actually, you do lose money as you spend more for the same payout.

  2. Re:Just a guess.. by I'm+New+Around+Here · · Score: 5, Funny

    Quality Test #87: Can you bend the circuit card more than 10 degrees? If so, to what degree?

    --
    If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
  3. Re: Europeans are so cute! by Anonymous Coward · · Score: 3, Interesting

    Europeans are the big boys in satellites.

  4. Not alarming by hackwrench · · Score: 4, Insightful

    The satellites have backup clocks for the very reason that something like this would happen. It just wasn't expected to happen on this scale, but despite that all satellites apparently have at least one rubidium and one hydrogen clock running. They have successfully gotten one hydrogen clock that wasn't running to run again, and there is apparently no reason to expect either way that the other clocks will or will not get restarted. Somebody in earlier comments mentioned a test that was run that may have induced the failure, so it is a bad test. Add it to the things to not do in the future. But the article seemed a bit light on what the things that can be done are, or what got that one clock working.

    1. Re:Not alarming by Hognoxious · · Score: 5, Funny

      I blame systemd.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  5. Re:Tin Whiskers? by Brett+Buck · · Score: 2

    Solder with lead is *far* less prone to tin whiskers than lead-free solder - which is mostly tin. Most aerospace applications have special dispensation from the RoHS rules concerning lead because it's a critical component in eliminating tin whiskers.

  6. Wow by nospam007 · · Score: 2

    My local Mom and Pop newspaper told me that a couple of days ago, I guess it also has news for nerds, and faster than /.

    1. Re:Wow by phantomfive · · Score: 3, Insightful

      Sometimes it's good to get the news later to let the facts reveal themselves. Not everything needs to be tied to the '24 hour immediate news' cycle.

      --
      "First they came for the slanderers and i said nothing."
  7. Re:Just a guess.. by thinkwaitfast · · Score: 3, Insightful

    Long ago when my company looked into this, the problem was finding manufacturing facilities that had procedures for it.

  8. Re:No surprises by Anonymous Coward · · Score: 4, Insightful

    Where is your vaunted aryan science, naziboys? Hmmm?

    At NASA. They were very eager to have it.

  9. Zoze Zwiss by codeButcher · · Score: 3, Funny

    ... with the assistance of the clock (Spectratime of Switzerland) ... manufacturer

    Seems to be a big blow to the Swiss clock makers' reputation for accuracy and reliability. But rejoice. Maybe those Swiss watches will start to sell for more realistic prices...

    Disclaimer: This post is intended to produce whooshing.

    --
    Free, as in your money being freed from the confines of your account.
    1. Re:Zoze Zwiss by Applehu+Akbar · · Score: 2

      EU to the Swiss timekeeping sector: Folks, this is not what we meant by a grand complication!

  10. Re: by phantomfive · · Score: 2

    4 clocks can only tolerate 1 failure

    That's only true if they fail simultaneously. Since they know one is bad, they can eliminate it before the next one fails. Now if one more fails, they are fine (the other two are still in agreement).

    If they have a way of externally checking the correctness then they can have three fail and still be operational, because they know which one is correct.

    --
    "First they came for the slanderers and i said nothing."
  11. Re:Tin Whiskers? by AmiMoJo · · Score: 3, Informative

    Since the Indians are not experiencing the same failure rate on identical hardware, it seems likely that it's something about the environment in the EU's satellites that is causing the problem. Maybe power supply issues, temperature control issues, or vibration issues.

    As you say, lots of things (like test equipment and of course aerospace) are exempt from RoHS so it's unlikely to be that. They will have used the most suitable materials.

    --
    const int one = 65536; (Silvermoon, Texture.cs)
    SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
  12. Re:Just a guess.. by davester666 · · Score: 3, Funny

    Answer: Each time, I can bend it a little more. I ended the test when I reached 180 degrees. I was unable to go past 180 degrees due to a physical limitation.

    --
    Sleep your way to a whiter smile...date a dentist!
  13. Re:Just a guess.. by Anonymous Coward · · Score: 5, Informative

    "The only property we really wanted from the lead was the lower melting point"

    and better wetting
    and higher ductility so thermal cycling doesn't crack your joints
    and suppression of tin scavenging so your solder bath doesn't strip the gold off your boards
    and suppression of whisker growth
    and the inhibition of tin pest

    along with of course the lower process temperature

    it's almost as if a eutectic lead-tin alloy was the ideal material for making solder joints.....

  14. Re:Just a guess.. by arglebargle_xiv · · Score: 3, Funny

    These failures would never have happened with an H5 chronometer. Maybe they should launch with one of those as an additional backup.

  15. Re:Just a guess.. by serviscope_minor · · Score: 2

    Lots of stuff is exempt from RoHS. It mostly applies to things you're selling on the open market, i.e. products. You can still buy full-fat lead solder perfectly legally.

    I'm actual rather partial to the tin/lead/silver blend for rework. It's quite a bit more expensive than normal eutectic solder, but melts at a lower temperature, which is really handy.

    --
    SJW n. One who posts facts.
  16. Re: by Anonymous Coward · · Score: 2, Informative

    This is not true for frequency references, which are the function of the on-board clocks. And they are externally disciplined by the two ground stations, which have an array of Cesium and Active Hydrogen Maser clocks.

    The Galileo satellites can operate with only a single atomic clock, at reduced performance. They can operate with two clocks of different types at full performance.

  17. Re: Just a guess.. by Anonymous Coward · · Score: 5, Interesting

    No, no they don't

    Nothing improves ductility and wetting like lead and to knock tin scavenging on the head you need to do more than dope it with .5% copper or bismuth or whatever. Considering our current propensity for shoving high-pin count BGA packages on top of each other and then sticking them into hot little boxes and then shoving them in our pockets we couldn't have chosen a worse time to drop lead, the rate at which devices are failing due to thermal stress induced cracking is embarrassing.

    and it just so happens that lead doesn't leech out of landfill, so it's almost as if the whole exercise was pointless.

  18. Re: by Anonymous Coward · · Score: 5, Interesting

    You are wrong. Please refrain from commenting on things you have no clue about.

    And don't get "electrical engineering and physics" clocks (jargon for "frequency reference") with layman and computer science clocks (devices that output a time coordinate relative to some frame of reference). Atomic clocks are *always* clocks only in the electrical engineering/physics jargon sense. They are pure frequency references *only*.

    Also, "atomic clocks" are always disciplined. Active Hydrogen Masers (in the ground) are primary references, but you always use several of them together to reduce *jitter* (it is not to "majority vote"). Anything else is disciplined by such masers long-term, and often output far more precise short term frequency references (rubidium oscillators, for example) e.g. due to better phase error measurement characteristics, etc.

    In the case of a Galileo satellite down to one clock, it will have somewhat reduced performance, and it *will* still be disciplined by the ground telemetry signal.

  19. Re:Just a guess.. by tomhath · · Score: 4, Interesting

    NASA had a similar problem years ago. They were testing the component by exposing it to voltage spikes. Passing the test also damaged the component.

  20. Re:ENDED is not a verb by rl117 · · Score: 2

    It's the past tense, it's perfectly valid and normal English. "I ended; I am ending; I will end".

  21. Re:Just a guess.. by Anonymous Coward · · Score: 2, Interesting

    Actually, surprisingly enough. RoHS does not apply to things manufactured within the EU, just items imported into the EU.
    It is more a protectionist regulation than an environmental one.
    The Chinese now have a much more restrictive RoHS policy than the EU does on paper

  22. Proof it's NOT a CIA inside job by cmseagle · · Score: 3, Insightful

    The proof that these clock failures aren't the result of some CIA meddling is that they haven't had any actual impact. A foreign government would want to take enough of these clocks offline to make the constellation fail, and they would want to make this happen at a time of their choosing. Ho do a few sporadic clock failures, some of which have already been fixed, benefit anyone?

  23. Time will tell. by UnixUnix · · Score: 2

    Years ago working on a NASA spacecraft a hitch developed and we couldn't read the system clock. We informed the mission manager. "-- Great, I'll just go tell the Director we can't do this mission because we don't know what time it is". Never lose your sense of humor.

  24. According to someone who builds vacuum electronics by raymorris · · Score: 4, Insightful

    I was just reading some posts from a guy whose job is building electronics which operate in a vacuum. As in, that's what he does all day. His first #1 tip for building electronics to be used in a vacuum is ...

    1) Don't use lead-free solder. Vacuum promotes the growth of whiskers, so lead-free solder always ended up with whiskers for us.

    I'll take it from the person who does this for a living.

  25. Re: Europeans are so cute! by TWX · · Score: 3, Funny

    Just not the British though.

    Why don't the British build satellites?

    Because they haven't figured out how to make them leak oil yet!

    --
    Do not look into laser with remaining eye.
  26. Re:Just a guess.. by Thud457 · · Score: 2

    "Tin Whiskers" is Optimus Prime's cat's name.

    --

    the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff

  27. Re:Just a guess.. by Ol+Olsoc · · Score: 2

    Lots of stuff is exempt from RoHS. It mostly applies to things you're selling on the open market, i.e. products. You can still buy full-fat lead solder perfectly legally.

    I'm actual rather partial to the tin/lead/silver blend for rework. It's quite a bit more expensive than normal eutectic solder, but melts at a lower temperature, which is really handy.

    I have multiple types of solder, modern substandard lead free solder, and Tin lead which I use for most things, depending on what the original solder was. Plus a number of silver containing We don't want to mix the types for certain. It is amazing the number of different compositions we have to choose from. I even worked on a couple ancient Tektronix scopes that used special solder that they included inside the case - the components were soldered onto ceramic bars with metal plated notches. You were screwed if you used any other solder.

    But the worst solders in my opinion are the lead free, and the more tin, the more likely the whisker problem shows up.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  28. Re:Just a guess.. by Megol · · Score: 2

    Bullshit! RoHS isn't silly, it wouldn't apply to spacecraft and you "bible" doesn't negate the actual research that is done on solder alloys after it was printed!

    You obviously don't understand alloys and that proportions of different components plus the choice of the components makes a huge difference. You don't understand the reasoning (and research) behind the RoHS and you think an old book triumphs modern research...

    It is easy to list things that changes the properties of solders, one example is that bismuth shouldn't be used in standard leaded solder as it can drastically reduce melt temperatures, in a lead free formulation there are not such problems (though an excess of bismuth will make joints fragile).

  29. Re:Just a guess.. by Ol+Olsoc · · Score: 2

    Screwed how? Was it calibrated for the thermocouple effect with that solder or something?

    No, what happened was that the lead in regular solder would dissolve the metal plated onto the notch, and it would destroy the connection point. Here is the interior of one of those old scopes http://www.barrytech.com/tektr... You can see the ceramic strips around the middle and top of the scope with the components mounted in between. Seriously weird stuff.

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
  30. Re:Just a guess.. by Ol+Olsoc · · Score: 2

    Bullshit! RoHS isn't silly, it wouldn't apply to spacecraft and you "bible" doesn't negate the actual research that is done on solder alloys after it was printed!

    You obviously don't understand alloys and that proportions of different components plus the choice of the components makes a huge difference. You don't understand the reasoning (and research) behind the RoHS and you think an old book triumphs modern research...

    It is easy to list things that changes the properties of solders, one example is that bismuth shouldn't be used in standard leaded solder as it can drastically reduce melt temperatures, in a lead free formulation there are not such problems (though an excess of bismuth will make joints fragile).

    Bullshit! RoHS isn't silly, it wouldn't apply to spacecraft and you "bible" doesn't negate the actual research that is done on solder alloys after it was printed!

    Chill out anger person. The pamphlet showed photograps and photomicrographs of various solders. Lead free solder is not something just thought up then designed by teams of scientists after tin/lead was declared bad. Analysis has gone on for many decades, as pointed out in a cite below. This thing had dozens of images, and the lead free solders of the time looked remarkably like lead free solder of today.

    You obviously don't understand alloys and that proportions of different components plus the choice of the components makes a huge difference. You don't understand the reasoning (and research) behind the RoHS and you think an old book triumphs modern research...

    Oh anger person, I think maybe less coffee or working out whatever has caused your rage might be in order, because there are one of two choices, you either have anger management issues because going wild over my post is uncalled for, or whatever that is bothering you that causes you to snap at an innocent post needs addressed.

    Anyhow, yes, they did have a lot of research back in the day. Especially with solder, due both to it's importance in electronics, but in the different melting points and strength characteristics, as well as the final intended use. A lot of interesting work was going on, in aerospace and space itself, where problems could be pretty critical and life ending. In addition, Mil-Spec had it's own strict requirements due to extreme reliability requirements. In general, Lead free solders were considered inadequate, and many consider them inadequate for the same reasons today. In space, whiskers are a problem aside for their obvious shorting tendencies, but rework isn't likely at all.

    If you need a more recent report, here's one from Auburn University: http://aaq.auburn.edu/node/88

    Especially important is the following:

    "Tin whiskers" were a problem with early electronic solders, and lead was initially added to the alloy in part to eliminate them (http://en.wikipedia.org/wiki/Solder). On July 1, 2006 the European Union Waste Electrical and Electronic Equipment Directive (WEEE) and Restriction of Hazardous Substances Directive (RoHS) came into effect prohibiting the intentional addition of lead to most consumer electronics produced in the EU. Manufacturers in the U.S. may receive tax benefits by reducing the use of lead-based solder. Lead free solders in commercial use may contain tin, copper, silver, bismuth, indium, zinc, antimony, and traces of other metals (http://en.wikipedia.org/wiki/Solder). There is no U.S. legislation presently on the horizon pertaining to lead containing solders. Most of the focus presently on eliminating lead in electronic solders is from European community countries and Japan (www.indium.com/_dynamo/download.php?docid=21). There are many lead free solders exists, but the group comprising tin-silver-copper alloys is considered the mainstream alloy system that will replace tin-lead (www.indium.com/_dynamo/download.php?docid=21).

    Is lead free

    --
    The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.