Slashdot Mirror


Galileo Satellites Are Experiencing Multiple Clock Failures (bbc.com)

elgatozorbas writes: According to a BBC article, the onboard atomic clocks that drive the satellite-navigation signals on Europe's Galileo network have been failing at an alarming rate. From the report: "Across the 18 satellites now in orbit, nine clocks have stopped operating. Three are traditional rubidium devices; six are the more precise hydrogen maser instruments that were designed to give Galileo superior performance to the American GPS network. Each Galileo satellite carries two rubidium and two hydrogen maser clocks. The multiple installation enables a satellite to keep working after an initial failure. All 18 spacecraft currently in space continue to operate, but one of them is now down to just two clocks. Most of the maser failures (5) have occurred on the satellites that were originally sent into orbit to validate the system, whereas all three rubidium stoppages are on the spacecraft that were subsequently launched to fill out the network. Esa staff at its technical centre, ESTEC, in the Netherlands are trying to isolate the cause the of failures - with the assistance of the clock (Spectratime of Switzerland) and satellite manufacturers (Airbus and Thales Alenia Space; OHB and SSTL). It is understood engineers have managed to restart another hydrogen clock that had stopped. It appears the rubidium failures 'all seem to have a consistent signature, linked to probable short circuits, and possibly a particular test procedure performed on the ground.'"

118 of 195 comments (clear)

  1. Just a guess.. by fred911 · · Score: 1

    " It appears the rubidium failures 'all seem to have a consistent signature, linked to probable short circuits, and possibly a particular test procedure performed on the ground."

    Dependent upon what the what the test was designed to do, it either passed or failed. It's probably best not to test it again.

    --
    09 F9 11 02 9D 74 E3 5B - D8 41 56 C5 63 56 88 C0 45 5F E1 04 22 CA 29 C4 93 3F 95 05 2B 79 2A B2
    1. Re:Just a guess.. by I'm+New+Around+Here · · Score: 5, Funny

      Quality Test #87: Can you bend the circuit card more than 10 degrees? If so, to what degree?

      --
      If you think I voted for Trump because of this post, you're wrong. I voted for Dr. Jill Stein of the Green Party. Again.
    2. Re:Just a guess.. by klui · · Score: 1

      My swag is solder whiskers from RoHS directives. Anyone familiar with these devices that know if they're on the list of exemptions?

    3. Re:Just a guess.. by K.+S.+Kyosuke · · Score: 1, Informative

      Why would RoHS directives apply to spaceflight? That would be silly.

      --
      Ezekiel 23:20
    4. Re:Just a guess.. by K.+S.+Kyosuke · · Score: 1

      Even if it were, it would still make no sense to mandate it for non-EU jurisdiction such as Earth's orbit.

      --
      Ezekiel 23:20
    5. Re:Just a guess.. by thinkwaitfast · · Score: 3, Insightful

      Long ago when my company looked into this, the problem was finding manufacturing facilities that had procedures for it.

    6. Re:Just a guess.. by davester666 · · Score: 3, Funny

      Answer: Each time, I can bend it a little more. I ended the test when I reached 180 degrees. I was unable to go past 180 degrees due to a physical limitation.

      --
      Sleep your way to a whiter smile...date a dentist!
    7. Re:Just a guess.. by Anonymous Coward · · Score: 5, Informative

      "The only property we really wanted from the lead was the lower melting point"

      and better wetting
      and higher ductility so thermal cycling doesn't crack your joints
      and suppression of tin scavenging so your solder bath doesn't strip the gold off your boards
      and suppression of whisker growth
      and the inhibition of tin pest

      along with of course the lower process temperature

      it's almost as if a eutectic lead-tin alloy was the ideal material for making solder joints.....

    8. Re:Just a guess.. by arglebargle_xiv · · Score: 3, Funny

      These failures would never have happened with an H5 chronometer. Maybe they should launch with one of those as an additional backup.

    9. Re:Just a guess.. by serviscope_minor · · Score: 2

      Lots of stuff is exempt from RoHS. It mostly applies to things you're selling on the open market, i.e. products. You can still buy full-fat lead solder perfectly legally.

      I'm actual rather partial to the tin/lead/silver blend for rework. It's quite a bit more expensive than normal eutectic solder, but melts at a lower temperature, which is really handy.

      --
      SJW n. One who posts facts.
    10. Re: Just a guess.. by Anonymous Coward · · Score: 5, Interesting

      No, no they don't

      Nothing improves ductility and wetting like lead and to knock tin scavenging on the head you need to do more than dope it with .5% copper or bismuth or whatever. Considering our current propensity for shoving high-pin count BGA packages on top of each other and then sticking them into hot little boxes and then shoving them in our pockets we couldn't have chosen a worse time to drop lead, the rate at which devices are failing due to thermal stress induced cracking is embarrassing.

      and it just so happens that lead doesn't leech out of landfill, so it's almost as if the whole exercise was pointless.

    11. Re:Just a guess.. by oobayly · · Score: 1

      Had to search for what that was - the Board of Longitude really did screw him over, didn't they.

    12. Re:Just a guess.. by sugar+and+acid · · Score: 1

      Their is a specific exemption Spacecraft in the RoHS directive.

    13. Re:Just a guess.. by Anonymous Coward · · Score: 1

      " (Silver used to be popular even before RoHS.)"

      Errr, silver solder wan't lead-free, it contained 2-3% silver, the balance was lead and tin.

    14. Re: Just a guess.. by Anonymous Coward · · Score: 1

      "In most cases the alternatives to leaded solder is a lot better in pretty much every regard, we just needed components specified for higher temperatures."

      You have obviously never worked with lead free solder. It sucks.

    15. Re:Just a guess.. by Anonymous Coward · · Score: 1

      Are you seriously asking? Because someone had to build/produce the components, dispose of the waste, etc.

    16. Re:Just a guess.. by tomhath · · Score: 4, Interesting

      NASA had a similar problem years ago. They were testing the component by exposing it to voltage spikes. Passing the test also damaged the component.

    17. Re:Just a guess.. by Anonymous Coward · · Score: 2, Interesting

      Actually, surprisingly enough. RoHS does not apply to things manufactured within the EU, just items imported into the EU.
      It is more a protectionist regulation than an environmental one.
      The Chinese now have a much more restrictive RoHS policy than the EU does on paper

    18. Re:Just a guess.. by petermgreen · · Score: 1

      IIRC Aerospace gets an exemption.

      The problem is that having both leaded and lead-free inventory is a PITA. A small ammount of lead contamination leaking into stuff that is supposed to be lead-free can lead to expensive problems both legally and technically.

      The result is lots of stuff that doesn't legally need to be lead-free nevertheless gets made on lead-free processes. Even if the final assembly is done with leaded solder it is very likely that things like component surface platings will be done with lead-free materials.

      --
      note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
    19. Re: Just a guess.. by stealth_finger · · Score: 1, Offtopic

      They're you go.

      --
      Wanna buy a shirt?
      https://www.redbubble.com/people/stealthfinger/shop?asc=u
    20. Re:Just a guess.. by ctilsie242 · · Score: 1

      I sort of doubt this is the cause. Things like annealing, epoxy potting, solder dip, and component geometry can be used to mitigate this. However, if tin whiskers did form, it could be a show-stopper.

    21. Re:Just a guess.. by TWX · · Score: 1

      The H4 was a partial failure?

      Last time I was at the Royal Observatory, the H4 was not only on display, it was still running.

      --
      Do not look into laser with remaining eye.
    22. Re:Just a guess.. by Thud457 · · Score: 2

      "Tin Whiskers" is Optimus Prime's cat's name.

      --

      the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff

    23. Re: Just a guess.. by Thud457 · · Score: 1, Offtopic

      Theirs your problem...

      --

      the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff

    24. Re:Just a guess.. by Ol+Olsoc · · Score: 1

      Why would RoHS directives apply to spaceflight? That would be silly.

      Yes it would. Of course, RoHS is silly to begin with. Even more so with more and more electronics being recycled.

      Years ago, I came across an old soldering booklet put out by IIRC Kester. Of interest was the contaminants section, with failures and defects. Many of the defects shown were by the presence of the components of modern day lead free solder.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    25. Re:Just a guess.. by Ol+Olsoc · · Score: 2

      Lots of stuff is exempt from RoHS. It mostly applies to things you're selling on the open market, i.e. products. You can still buy full-fat lead solder perfectly legally.

      I'm actual rather partial to the tin/lead/silver blend for rework. It's quite a bit more expensive than normal eutectic solder, but melts at a lower temperature, which is really handy.

      I have multiple types of solder, modern substandard lead free solder, and Tin lead which I use for most things, depending on what the original solder was. Plus a number of silver containing We don't want to mix the types for certain. It is amazing the number of different compositions we have to choose from. I even worked on a couple ancient Tektronix scopes that used special solder that they included inside the case - the components were soldered onto ceramic bars with metal plated notches. You were screwed if you used any other solder.

      But the worst solders in my opinion are the lead free, and the more tin, the more likely the whisker problem shows up.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    26. Re:Just a guess.. by serviscope_minor · · Score: 1

      We don't want to mix the types for certain.

      A cretain amount of mixing is OK. Mixing standard RoHS and Sn/Pb is fine, since tin and lead are completely miscable. Not sure about the silver one, but I've reworked SnPb and RoHS with it (it's designed for that) no problem.

      I wouldn't go any further and mix more exotic types though. There are some awfully exotic ones there.

      You were screwed if you used any other solder.

      Screwed how? Was it calibrated for the thermocouple effect with that solder or something?

      But the worst solders in my opinion are the lead free, and the more tin, the more likely the whisker problem shows up.

      Yeah nasty sticky things. To be honest though, I have done reflow with RoHS, and it worked fine. Whiskers are fascinating. The RoHS blends have improved beyond the initial pure tin ones, but ther're not as good as leaded ones.

      --
      SJW n. One who posts facts.
    27. Re:Just a guess.. by Megol · · Score: 1

      You can't be serious?!? First whiskers is much less of a problem than those ranting about non-leaded solder, second it is possible to use techniques to reduce* the risk of whiskers etc. that can be done for some extra expense and third critical infrastructure etc. like spacecraft are exempt from the RoHs demands!

      (* using leaded solder also only reduce risks of whisker formation)

    28. Re:Just a guess.. by Megol · · Score: 2

      Bullshit! RoHS isn't silly, it wouldn't apply to spacecraft and you "bible" doesn't negate the actual research that is done on solder alloys after it was printed!

      You obviously don't understand alloys and that proportions of different components plus the choice of the components makes a huge difference. You don't understand the reasoning (and research) behind the RoHS and you think an old book triumphs modern research...

      It is easy to list things that changes the properties of solders, one example is that bismuth shouldn't be used in standard leaded solder as it can drastically reduce melt temperatures, in a lead free formulation there are not such problems (though an excess of bismuth will make joints fragile).

    29. Re:Just a guess.. by Bob+the+Super+Hamste · · Score: 1
      Like others I had to look that up but the makers name sounded familar and then I got the this text on wiki:

      One of the controversial claims of his last years was that of being able to build a land clock more accurate than any competing design. Specifically, he claimed to have designed a clock capable of keeping accurate to within one second over a span of 100 days.

      And I remembered reading about it when someone did finally build his clock and it turned out he was right.

      --
      Time to offend someone
    30. Re:Just a guess.. by Ol+Olsoc · · Score: 2

      Screwed how? Was it calibrated for the thermocouple effect with that solder or something?

      No, what happened was that the lead in regular solder would dissolve the metal plated onto the notch, and it would destroy the connection point. Here is the interior of one of those old scopes http://www.barrytech.com/tektr... You can see the ceramic strips around the middle and top of the scope with the components mounted in between. Seriously weird stuff.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    31. Re:Just a guess.. by ChrisMaple · · Score: 1

      So much for Nietzsche.

      --
      Contribute to civilization: ari.aynrand.org/donate
    32. Re:Just a guess.. by Ol+Olsoc · · Score: 2

      Bullshit! RoHS isn't silly, it wouldn't apply to spacecraft and you "bible" doesn't negate the actual research that is done on solder alloys after it was printed!

      You obviously don't understand alloys and that proportions of different components plus the choice of the components makes a huge difference. You don't understand the reasoning (and research) behind the RoHS and you think an old book triumphs modern research...

      It is easy to list things that changes the properties of solders, one example is that bismuth shouldn't be used in standard leaded solder as it can drastically reduce melt temperatures, in a lead free formulation there are not such problems (though an excess of bismuth will make joints fragile).

      Bullshit! RoHS isn't silly, it wouldn't apply to spacecraft and you "bible" doesn't negate the actual research that is done on solder alloys after it was printed!

      Chill out anger person. The pamphlet showed photograps and photomicrographs of various solders. Lead free solder is not something just thought up then designed by teams of scientists after tin/lead was declared bad. Analysis has gone on for many decades, as pointed out in a cite below. This thing had dozens of images, and the lead free solders of the time looked remarkably like lead free solder of today.

      You obviously don't understand alloys and that proportions of different components plus the choice of the components makes a huge difference. You don't understand the reasoning (and research) behind the RoHS and you think an old book triumphs modern research...

      Oh anger person, I think maybe less coffee or working out whatever has caused your rage might be in order, because there are one of two choices, you either have anger management issues because going wild over my post is uncalled for, or whatever that is bothering you that causes you to snap at an innocent post needs addressed.

      Anyhow, yes, they did have a lot of research back in the day. Especially with solder, due both to it's importance in electronics, but in the different melting points and strength characteristics, as well as the final intended use. A lot of interesting work was going on, in aerospace and space itself, where problems could be pretty critical and life ending. In addition, Mil-Spec had it's own strict requirements due to extreme reliability requirements. In general, Lead free solders were considered inadequate, and many consider them inadequate for the same reasons today. In space, whiskers are a problem aside for their obvious shorting tendencies, but rework isn't likely at all.

      If you need a more recent report, here's one from Auburn University: http://aaq.auburn.edu/node/88

      Especially important is the following:

      "Tin whiskers" were a problem with early electronic solders, and lead was initially added to the alloy in part to eliminate them (http://en.wikipedia.org/wiki/Solder). On July 1, 2006 the European Union Waste Electrical and Electronic Equipment Directive (WEEE) and Restriction of Hazardous Substances Directive (RoHS) came into effect prohibiting the intentional addition of lead to most consumer electronics produced in the EU. Manufacturers in the U.S. may receive tax benefits by reducing the use of lead-based solder. Lead free solders in commercial use may contain tin, copper, silver, bismuth, indium, zinc, antimony, and traces of other metals (http://en.wikipedia.org/wiki/Solder). There is no U.S. legislation presently on the horizon pertaining to lead containing solders. Most of the focus presently on eliminating lead in electronic solders is from European community countries and Japan (www.indium.com/_dynamo/download.php?docid=21). There are many lead free solders exists, but the group comprising tin-silver-copper alloys is considered the mainstream alloy system that will replace tin-lead (www.indium.com/_dynamo/download.php?docid=21).

      Is lead free

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    33. Re: Just a guess.. by Agripa · · Score: 1

      There are applications obviously exempt from RoHS and space flight is one of them.

      Unfortunately the exemption is moot when availability of non-RoHS parts vanishes or even worse, non-RoHS parts get substituted without warning. The result at best is having to re-plate leads before assembly which is not always feasible.

      If the problem ends having been caused by the RoHS regulations, I will laugh. They deserved it.

    34. Re:Just a guess.. by Agripa · · Score: 1

      I even worked on a couple ancient Tektronix scopes that used special solder that they included inside the case - the components were soldered onto ceramic bars with metal plated notches. You were screwed if you used any other solder.

      The situation was not quite that bad. Tektronix included a small spool of the silver solder they used for assembly because normal tin/lead solder can dissolve enough of the silver off of the ceramic strips to cause the solder joint to separate.

    35. Re:Just a guess.. by serviscope_minor · · Score: 1

      Oh I see. How amazing! Any idea what the composition of the solder or contact point was? I've never worked on anything remotely like that.

      --
      SJW n. One who posts facts.
    36. Re:Just a guess.. by Ol+Olsoc · · Score: 1

      Oh I see. How amazing! Any idea what the composition of the solder or contact point was? I've never worked on anything remotely like that.

      They wanted us to use a solder with 3 percent silver. Sn62Pb36Ag2 The silver was to help against dissolution of the silver from the contact, and had a little bit lower of a melting point.

      The process to make the strips was probably similar to that used for plated through holes. There must have been some reason that they used silver instead of copper however, I suspect it is mechanical, since the plated area isn't annular, and the stresses will be different. To my surprise - silver ceramic terminals are still manufactured and sold! http://www.partsconnexion.com/... .

      I had a little experience with plated through hole construction, but we used copper and solder plating. So anyone knowing about the silver process is welcome to chime in.

      --
      The shepherds did so well protecting the flock that the sheep no longer believed that wolves existed.
    37. Re:Just a guess.. by serviscope_minor · · Score: 1

      How remarkable. I'm currently in electronics professionally a bit, but it's all modern, mainstream stuff. It's amazing to see something totally off the wall. The silver could be a conductivity thing? It's about 10% better than copper I think, though I'd be surprised if there was enough in that to make it worthwhile. I guess it might be adhesion to the ceramic too, or possibly they didn't develop copper plating techniques as well back when the scope was made.

      I'd never heard of the problem of Sn/Pb dissolving silver before today.

      I had a little experience with plated through hole construction, but we used copper and solder plating. So anyone knowing about the silver process is welcome to chime in.

      Holes on ceramic like those ones?

      Anyway, fascinating info, thanks! Always good to learn something new.

      --
      SJW n. One who posts facts.
    38. Re:Just a guess.. by RockDoctor · · Score: 1

      It's amazing to see something totally off the wall. The silver could be a conductivity thing? It's about 10% better than copper I think,

      You think that you can get better than 10% cross-sectional area consistency when soldering something by hand? Without having to do an individual test on every component made, and re-work on ... well, rework on any of them would probably destroy the cost saving from whatever solution your peculiar solder was trying to achieve. There's a reason that, for example, you build voltage divider networks from k-Ohm or M-Ohm components linked by connections with joint resistances of fractions of an Ohm - it reduces the effect of soldering errors.

      --
      Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"
    39. Re:Just a guess.. by AutodidactLabrat · · Score: 1

      Silver carrying microwaves prevents surface oxidation "bubbling" or isolation of sub-millimeter standing wave stubs, since silver oxide is enough conductive to dissipate, rather than propagate, surface isolated stubs caused by corrosion.
      The other reason has been mentioned, the mount points of many surface mount components, particularly capacitors, are silver/palladium, and leaching of silver into the liquid solder can crack or even dissolve the contact.
      Both causes together make saturating the tin/copper/gold or Tin/gallium/copper alloy necessary to successful PCB construction

  2. CIA inside job by Anonymous Coward · · Score: 3, Interesting

    To make sure there is no concurrence from another global measuring system. CT ? Possibly. But is that far fetched ?

    1. Re:CIA inside job by anti-pop-frustration · · Score: 5, Insightful

      Well I couldn't help thinking about the Iranian civilian program and the 'alarming rate of failure' of their centrifuges.

      Are the contractors who worked on Galileo going to discover some kind of Stuxnet variant on their network?

      Technically, it's possible. Thanks to work of security researchers we know it happened before, Stuxnet is well-documented. And thanks to Edward Snowden and the journalists who reported on the documents he leaked, we know the NSA/TAO does in fact hack allies.

      I wish these kind of doubts could be instantly discarded as conspiracy theories, unfortunately, this is not the world we live in. The most technically capable nations (USA/China/Russia/Isreal) have made the choice of using hacking as a weapon rather than helping secure the systems used by their citizens.

    2. Re:CIA inside job by anti-pop-frustration · · Score: 1

      Replying to myself but yes, 'Israel' not 'Isreal'.

      Time for that second cup of coffee....

    3. Re:CIA inside job by GeekWithAKnife · · Score: 1


      As soon as I read the subject I thought this may be a stuxnet-like derivative. -which may turn out to be wrong...

      The reality is that the hacking tools available are far scarier than we have yet found proof of if you believe the reactions of some agency leakers about Stuxnet. (how they address the notion as a minor part of a far more sophisticate project)

      If it didn't happen already, electricity itself can be used as an attack vector to infiltrate systems. I don't mean shutting the power down I mean using the power connection as a medium to initiate malicious activity.

      Assuming power lines are not yet an attack vector that's no consolation as it's practically not needed for most situations. Any core network or infrastructure that is even remotely associated with vulnerable common systems (windows, wifi, humans etc) are susceptible because they are in contact with the weakest link in the security chain. They can introduce code into secure systems in unintentional and unexpected ways.

      What cyber warfare has introduced is essentially a global cold war type scenario. Ironically brought about by the same people seeking supremacy as they worked hard to weaken all defences just so they can have an edge.

      --
      A 'singular oddity' is an event that cannot be explained and only happens when you are alone.
    4. Re:CIA inside job by GuB-42 · · Score: 4, Informative

      GLONASS and Beidou work fine.
      I suspect that these would be the most likely targets should the CIA want to disable competing systems. Europe is supposedly an ally, Russia and China, not so much.

    5. Re:CIA inside job by enriquevagu · · Score: 5, Interesting

      I do not work for Galileo, but I know some people involved in the Project.

      Interestingly (and unfortunately) the entities most interested in a failure and subsequent delay are neither millitary enemies nor allies (for whatever "ally" means, when you consider hacking your allies). The largest interest in a project delay comes from the many corporations directly involved in its development. This project has been feeding many millios of taxpayer euros during many years to a lot of European tech companies, and the cost uprising has been actually benefitial for them because there were no substantial economic sanctions from these delays (probably an example of a wrongly managed project). Successfully completing the project would kill this guaranteed job & income. I do not claim that delays and increased costs have been intentional, but they have been definitely benefitial for them.

    6. Re:CIA inside job by budgenator · · Score: 1

      And keeping the Galileo military-grade GPS an unreliable entity will insure that GLONASS and Beidou work fine and used exclusively until the USG sends them the back-doored kill signal at the opportune time; or at least that's the way I would do it if I were a Spy Movie script-writer.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    7. Re:CIA inside job by ArmoredDragon · · Score: 1

      Or the ESA just fails at engineering. For example they've done many Mars missions and none has been successful, and for what purpose would the CIA sabotage that?

    8. Re:CIA inside job by Cochonou · · Score: 2

      ESA contracts are fixed price. Even if there are no penalties, you do not gain any benefit by being late: actually, you do lose money as you spend more for the same payout.

  3. It's aliens by Bloke+down+the+pub · · Score: 1

    It's aliens. And if they like Donald Trump I'd say that's an asset, not a liability.

    --
    It's true I tell you, feller at work's next door neighbour read it in the paper.
    1. Re:It's aliens by sheramil · · Score: 1

      has to be aliens. their fiendish plan is to destroy the world's clocks, and then invade when we all sleep in.

  4. Tin Whiskers? by TheNarrator · · Score: 1

    They are using lead solder in these things I hope? If not, they could be shorting out because of tin whiskers. NASA even has a site devoted to this : https://nepp.nasa.gov/whisker/...

    1. Re:Tin Whiskers? by K.+S.+Kyosuke · · Score: 1

      Seeing as a lot of space equipment gets built in the EU that doesn't exhibit these failures, whatever went wrong in this particular series of hardware units quite likely has nothing to do with the solder.

      --
      Ezekiel 23:20
    2. Re:Tin Whiskers? by Brett+Buck · · Score: 2

      Solder with lead is *far* less prone to tin whiskers than lead-free solder - which is mostly tin. Most aerospace applications have special dispensation from the RoHS rules concerning lead because it's a critical component in eliminating tin whiskers.

    3. Re:Tin Whiskers? by AmiMoJo · · Score: 3, Informative

      Since the Indians are not experiencing the same failure rate on identical hardware, it seems likely that it's something about the environment in the EU's satellites that is causing the problem. Maybe power supply issues, temperature control issues, or vibration issues.

      As you say, lots of things (like test equipment and of course aerospace) are exempt from RoHS so it's unlikely to be that. They will have used the most suitable materials.

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    4. Re:Tin Whiskers? by Cochonou · · Score: 1

      Pure tin solder is prohibited on most space programs.

  5. Re: Europeans are so cute! by Anonymous Coward · · Score: 3, Interesting

    Europeans are the big boys in satellites.

  6. Not alarming by hackwrench · · Score: 4, Insightful

    The satellites have backup clocks for the very reason that something like this would happen. It just wasn't expected to happen on this scale, but despite that all satellites apparently have at least one rubidium and one hydrogen clock running. They have successfully gotten one hydrogen clock that wasn't running to run again, and there is apparently no reason to expect either way that the other clocks will or will not get restarted. Somebody in earlier comments mentioned a test that was run that may have induced the failure, so it is a bad test. Add it to the things to not do in the future. But the article seemed a bit light on what the things that can be done are, or what got that one clock working.

    1. Re:Not alarming by Hognoxious · · Score: 5, Funny

      I blame systemd.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    2. Re:Not alarming by jez9999 · · Score: 1

      That's like saying "only 3 of the 4 jet engines have blown up, we're still fine"...

    3. Re: Not alarming by PPH · · Score: 1

      I can't wait to see how this thread will wind up.

      --
      Have gnu, will travel.
    4. Re:Not alarming by Hognoxious · · Score: 1

      No, it's in binary.

      --
      Confucius say, "Find worm in apple - bad. Find half a worm - worse."
    5. Re:Not alarming by Cochonou · · Score: 1

      It is totally alarming. Redundancies are not supposed to kick in so early in the program. Such a failure rate is off scale compared to the reliability targets of a space equipment.

  7. Wow by nospam007 · · Score: 2

    My local Mom and Pop newspaper told me that a couple of days ago, I guess it also has news for nerds, and faster than /.

    1. Re:Wow by phantomfive · · Score: 3, Insightful

      Sometimes it's good to get the news later to let the facts reveal themselves. Not everything needs to be tied to the '24 hour immediate news' cycle.

      --
      "First they came for the slanderers and i said nothing."
    2. Re:Wow by mrbester · · Score: 1

      It would have been posted here before, but the clock that is used to schedule it unexpectedly failed.

      --
      "Wait. Something's happening. It's opening up! My God, it's full of apricots!"
  8. Re: by Anonymous Coward · · Score: 1

    4 clocks can only tolerate 1 failure (if you don't understand this, read https://en.wikipedia.org/wiki/Byzantine_fault_tolerance#Early_solutions), so 2 failures means they don't know what time it is, which means they have failed to have accurate time keeping.

  9. Its the capacitors by perpenso · · Score: 1

    Were the capacitors manufactured about ten years ago? ;-)

    1. Re:Its the capacitors by Dr_Barnowl · · Score: 1

      You wouldn't use electrolytic capacitors in space because the low pressure would cause the electrolyte to boil away fairly rapidly. Yes, even faster than normal.

      As Mark Watney discovers in The Martian when his laptop goes "phut" when he takes it outside of the dome, consumer electronics, even units approved for usage on NASA missions, aren't designed to withstand environments outside what's normal on Earth.

    2. Re:Its the capacitors by budgenator · · Score: 1

      lots of ceramic and solid-state-tantalum capacitors, typically in redundant parallel arrays so that single-failure doesn't matter much.

      I would more reasonably expect that Capacitors in parallel were designed that way for value adjustment rather than failure tolerance; I've seen many more caps short out than open, and one short would take out the whole circuit.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    3. Re:Its the capacitors by perpenso · · Score: 1

      Were the capacitors manufactured about ten years ago? ;-)

      You wouldn't use electrolytic capacitors in space because ...

      Years ago there were some bad production runs of capacitors that found there way into various electronic devices. Blaming the capacitors for hardware failures became a bit overused, a sort of joke, at the time. :-)

    4. Re:Its the capacitors by budgenator · · Score: 1

      Even with a fuse, the total capacitance of the circuit would decrease, which would change the operational parameters.

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
  10. Re:No surprises by Anonymous Coward · · Score: 4, Insightful

    Where is your vaunted aryan science, naziboys? Hmmm?

    At NASA. They were very eager to have it.

  11. Zoze Zwiss by codeButcher · · Score: 3, Funny

    ... with the assistance of the clock (Spectratime of Switzerland) ... manufacturer

    Seems to be a big blow to the Swiss clock makers' reputation for accuracy and reliability. But rejoice. Maybe those Swiss watches will start to sell for more realistic prices...

    Disclaimer: This post is intended to produce whooshing.

    --
    Free, as in your money being freed from the confines of your account.
    1. Re:Zoze Zwiss by ctrl-alt-canc · · Score: 1

      Cuckoo clocks don't work well in the vacuum...

    2. Re:Zoze Zwiss by Applehu+Akbar · · Score: 2

      EU to the Swiss timekeeping sector: Folks, this is not what we meant by a grand complication!

    3. Re:Zoze Zwiss by codeButcher · · Score: 1

      That's why the world needs cuckoo spacesuits!!!!

      (I believe the cuckoo clock is more a Swabian (https://en.wikipedia.org/wiki/Swabia) thing. Black Forest and all you know. Even if Swabian German and Swiss German sounds very much alike to non-Swiss non-Swabians.)

      --
      Free, as in your money being freed from the confines of your account.
    4. Re:Zoze Zwiss by budgenator · · Score: 1

      Well the bird doesn't cookoo in a vacuum but being in free-fall doesn't help a weight driven clock do anything. Most people would be perfectly fine with a bird that doesn't cookoo in a cookoo clock. Most of the people would be ecstatic if their cookoo clock didn't cookoo from when the kids went to bed to until the adults got their first cup of coffee!!

      --
      Apocalypse Cancelled, Sorry, No Ticket Refunds
    5. Re:Zoze Zwiss by ChrisMaple · · Score: 1

      I can see it now. Centrifuges in space so that weight driven clocks will work.

      --
      Contribute to civilization: ari.aynrand.org/donate
  12. Re: by phantomfive · · Score: 2

    4 clocks can only tolerate 1 failure

    That's only true if they fail simultaneously. Since they know one is bad, they can eliminate it before the next one fails. Now if one more fails, they are fine (the other two are still in agreement).

    If they have a way of externally checking the correctness then they can have three fail and still be operational, because they know which one is correct.

    --
    "First they came for the slanderers and i said nothing."
  13. But is this a Byzantine failure situation? by hackwrench · · Score: 1
    https://www.bing.com/search?q=... turns up better results.

    The multiple installation enables a satellite to keep working after an initial failure. All 18 spacecraft currently in space continue to operate, but one of them is now down to just two clocks.

    Would seem to indicate that they are solving the Byzantine General problem within the "clock" itself and the other "clocks" really are just spares, as would be expected of such satellites. Do you really think it is a good idea to launch satillites that rely on timekeeping without including clocks that really are just spares?

    https://www.bing.com/search?q=... turns up no result indicating that timekeeping is a Byzantine failure problem. That is, it is possible that a clock either always returns a correct result or an easily proven wrong result like returning a value 0. It is a bad search for this topic, but is the one I started with, so anyone reading this, learn from my mistake and do not use it for researching this topic.

  14. Re: by lastman71 · · Score: 1

    Depends on the kind of the failure. If there is a subtle drift, yes, can be hard to detect the two faulty clock. But if they just stop working, it's easy.

    Still the case of two clock drifting away can be detected from ground, because you can measure from a static station on earth and find out if the signal sent from the satellite is correct or not. So I don't see problem even with two fault clock.

  15. Re: by gnasher719 · · Score: 1

    That's only true if they fail simultaneously. Since they know one is bad, they can eliminate it before the next one fails. Now if one more fails, they are fine (the other two are still in agreement).

    We don't know what happens in case of failure - does the clock give inaccurate time, or does it give no time at all. You could have a design where the clock either works or doesn't work, and you don't need to check them for disagreement at all.

    As I read it, they put four clocks on every satellite because these clocks break sometimes, and having for maximises the chances of having at least one that is working.

  16. Ouch by lkroll4565 · · Score: 1

    Hope those that depend on these satellites for navigation don't get lost. ;) :)

  17. Re: by Anonymous Coward · · Score: 2, Informative

    This is not true for frequency references, which are the function of the on-board clocks. And they are externally disciplined by the two ground stations, which have an array of Cesium and Active Hydrogen Maser clocks.

    The Galileo satellites can operate with only a single atomic clock, at reduced performance. They can operate with two clocks of different types at full performance.

  18. The REAL question. . . by Salgak1 · · Score: 1

    . . . .is what sort of shirt were the clock engineers wearing ??

  19. Re: by Anonymous Coward · · Score: 5, Interesting

    You are wrong. Please refrain from commenting on things you have no clue about.

    And don't get "electrical engineering and physics" clocks (jargon for "frequency reference") with layman and computer science clocks (devices that output a time coordinate relative to some frame of reference). Atomic clocks are *always* clocks only in the electrical engineering/physics jargon sense. They are pure frequency references *only*.

    Also, "atomic clocks" are always disciplined. Active Hydrogen Masers (in the ground) are primary references, but you always use several of them together to reduce *jitter* (it is not to "majority vote"). Anything else is disciplined by such masers long-term, and often output far more precise short term frequency references (rubidium oscillators, for example) e.g. due to better phase error measurement characteristics, etc.

    In the case of a Galileo satellite down to one clock, it will have somewhat reduced performance, and it *will* still be disciplined by the ground telemetry signal.

  20. Re:ENDED is not a verb by Gabest · · Score: 1

    Is "started" also not a verb? Also a foreigner here. Just asking to be smarter.

  21. Yes alarming by thegarbz · · Score: 1

    The earliest satellite in the constellation was launched only 5 years ago and is less than half way through it's minimum design life.

    Yes failures have so far not knocked any satellites off line but only because of redundancy. There is no information as to if these failures are random or systematic. In either case the redundancy comes into play to prevent well controlled random errors most likely to occur as satellites approach end of life.

    A 12.5% failure rate in the core components of a system that has been in place for less than half its design life is alarming by any metric. All the rubidium failures have occurred in satellites that are less than a quarter of the way into their design life.

    On the upside, given the go live dates under EU law the rubidium clocks are covered under manufacturer warranty. But the shipping costs are going to be a bastard!

  22. Re:ENDED is not a verb by rl117 · · Score: 2

    It's the past tense, it's perfectly valid and normal English. "I ended; I am ending; I will end".

  23. So simple by Virtucon · · Score: 1

    They should have used unobtanium clocks, FTFY.

    --
    Harrison's Postulate - "For every action there is an equal and opposite criticism"
  24. Re:ENDED is not a verb by Anonymous Coward · · Score: 1

    Began - Ended

    Moroff

  25. Do you happen to know about smartphones in vacuum? by raymorris · · Score: 1

    Your post brings to mind a question I was pondering the other day, which may have useful implications.

    Other than the battery, would low pressure damage the components typically used in smartphones? I'm thinking around 2 Kpa or so.

  26. Reverse the polarity of the neutron flow by jfdavis668 · · Score: 1

    Reversing the polarity of the neutron flow always seems to fix the TARDIS, should work here, too.

  27. Proof it's NOT a CIA inside job by cmseagle · · Score: 3, Insightful

    The proof that these clock failures aren't the result of some CIA meddling is that they haven't had any actual impact. A foreign government would want to take enough of these clocks offline to make the constellation fail, and they would want to make this happen at a time of their choosing. Ho do a few sporadic clock failures, some of which have already been fixed, benefit anyone?

    1. Re:Proof it's NOT a CIA inside job by Anonymous Coward · · Score: 1

      If you want to be covert, you don't take a system like this down all at once. You nibble it away a little bit at a time so it looks like a cascade of unconnected failures over time. Maybe the first 'fix' was to install an even more useful bug.

      Or maybe the attempt to meddle simply failed because the satellites are more robust than thought.

  28. Time will tell. by UnixUnix · · Score: 2

    Years ago working on a NASA spacecraft a hitch developed and we couldn't read the system clock. We informed the mission manager. "-- Great, I'll just go tell the Director we can't do this mission because we don't know what time it is". Never lose your sense of humor.

  29. According to someone who builds vacuum electronics by raymorris · · Score: 4, Insightful

    I was just reading some posts from a guy whose job is building electronics which operate in a vacuum. As in, that's what he does all day. His first #1 tip for building electronics to be used in a vacuum is ...

    1) Don't use lead-free solder. Vacuum promotes the growth of whiskers, so lead-free solder always ended up with whiskers for us.

    I'll take it from the person who does this for a living.

  30. Re: Europeans are so cute! by TWX · · Score: 3, Funny

    Just not the British though.

    Why don't the British build satellites?

    Because they haven't figured out how to make them leak oil yet!

    --
    Do not look into laser with remaining eye.
  31. Re: by budgenator · · Score: 1

    Third Option, 2 clocks disagree, both are discarded because you can't be sure which is correct.

    --
    Apocalypse Cancelled, Sorry, No Ticket Refunds
  32. Re:like by hackwrench · · Score: 1

    Only in the same sense that both are like saying most people eat three meals a day but I only eat one.

  33. Re:alarming by any metric by hackwrench · · Score: 1

    PC's and consumer electronics must be especially alarming then. But turning them off and back on again usually fixes any failure.

  34. Re:ENDED is not a verb by LynnwoodRooster · · Score: 1
    --
    Browsing at +1 - no ACs, I ignore their posts. So refreshing!
  35. Re:alarming by any metric by thegarbz · · Score: 1

    PC's and consumer electronics must be especially alarming then.

    You have no idea how much so. Some of us work in situations where PCs need incredibly high uptime. Not only is it alarming having to go through a powercycle, it can also be incredibly dangerous. This is why we do things like add ECC memory for that 1 in a million random case, multiple PSUs, RAID, etc. Then when we actually have an issue we go through a complete root cause of failure analysis with vendor engineers.

    You see when things start costing money, failures (even ones that don't knock the device offline) ARE alarming.

  36. Re:alarming by any metric by hackwrench · · Score: 1

    Then by that standard the world should be in a perpetual state of alarm.

  37. Re:According to someone who builds vacuum electron by Hognoxious · · Score: 1

    I was just reading some posts from a guy whose job is building electronics

    Yeah? Well I was in the SAS before I was an astronaut, and he's talking crap.

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."
  38. Re: Europeans are so cute! by Bob+the+Super+Hamste · · Score: 1

    So you are implying that the Prince of Darkness Lucas Electric was involved in making these satellites?

    --
    Time to offend someone
  39. Re:ENDED is not a verb by vux984 · · Score: 1

    Both are verbs.

    start
    Ladies and gentlemen, start your engines!
    I often start my car before I put on my seat belt.
    I had trouble starting my car this morning, but it started fine yesterday. It usually starts fine.

    end:
    The exam ends at 3.
    Classes ended early today.
    I bet she will end the relationship after the trip.
    We must end the war on drugs.

  40. Re:alarming by any metric by thegarbz · · Score: 1

    Then by that standard the world should be in a perpetual state of alarm.

    Not at all. Could your computer kill someone if it needed to be reboot? I know my desktop will be just fine, and there's nothing alarming about it. Hell I can put a bullet through my drive right now and I'll be fine. If on the other hand I have 2 drives fail at once in my NAS I would be very alarmed.

    You seem to think that being alarmed has to do with the event. It doesn't. It has to do with the risk which is function of the event and the consequence.
    I have had a rubidium clock fail here at home. Nothing to be alarmed about, but then I fixed the problem for $125. Try doing that with a satellite.

  41. Re:Standards by hackwrench · · Score: 1

    That's not the standard you set in your first message. "A 12.5% failure rate in the core components of a system that has been in place for less than half its design life is alarming by any metric." I believe it was.

  42. Re: ENDED is not a verb by Streetlight · · Score: 1

    Just what I was going to say. Trump will take care of the problem. Wait for the Tweet.

    --
    In a time of universal deceit, telling the truth is a revolutionary act. George Orwell
  43. Re:ENDED is not a verb by Dread_ed · · Score: 1

    Freeze - Froze - Frozen

    Squeeze - Squoze - Squozen

    --
    When the only tool you have is a claw hammer every problem starts to look like the back of someone's skull.
  44. Re:Standards by thegarbz · · Score: 1

    Sure it is. I specified an exact number of failures in an article about a satellite, I specified the design life of a satellite. The consequence is naturally the damage of a satellite.

    Taking a statement out of its context is either showing incredibly poor reading comprehension or that you're bored and just feel like talking shit on the internet. I don't think you're stupid, but I do think you have nothing better to do than waste people's time.

    Anyway I've made my point. If you want to continue believing that the failure rate they experienced is not alarming then maybe you should call them up and tell them they are wasting all their money on the investigations and analysis. I'm sure they'll take your comments on board.

  45. Alarming vs problematic by hackwrench · · Score: 1

    There's a big difference between something that is alarming vs. probematic. Just because it isn't alarming doesn't mean they are wasting their time figuring out how to get things back in working order. In fact, one of the reasons it is not alarming is because the can do something to get things back in working order. You're like that guy who said Ray Kurzweil was afraid of death because he is interested in finding ways to avoid the unpleasant thing. I don't like lots of things; doesn't mean I am afraid of them. I do things to fix situations; doesn't mean they are alarming. Or are you alarmed every time something fails and you have to turn it off and back on again?

    "Alarm" equates to panic. It is not a waste of time to attempt to set people straight when they use a term inappropriately. I may disagree with C. S. Lewis that the proper term for a waterfall is sublime as opposed to beautiful, but he does have a point. https://www.bing.com/search?q=...

    1. Re:Alarming vs problematic by thegarbz · · Score: 1

      Or are you alarmed every time something fails

      No I'm alarmed when a failure rate is high. Such as when a several several hundred million dollar satellites suffer component failures only a few short years into their mission.

      "Alarm" equates to panic.

      No. Alarm equates to response. Just like I get an alarm when a sector relocation count on my HDD increases, or the alarm I have set right now to tell me that I need to go check the oven to see if dinner is ready.

      It is not a waste of time to attempt to set people straight when they use a term inappropriately.

      You're welcome.

  46. Re:Alarming vs get an alarm by hackwrench · · Score: 1

    Whether or not alarming equates exactly to the emotion of panic or some slightly different emotion, the term alarming as used in the article is usually used to refer to emotion. When it refers to an actual alarm, it is get an alarm as you yourself have indicated.

  47. Re:According to someone who builds vacuum electron by Anonymous Coward · · Score: 1

    So you were infiltrating and destroying IRA cells while building electrical circuits on the side and preparing to get blasted off into space?

  48. Re:According to someone who builds vacuum electron by Hognoxious · · Score: 1

    It was difficult to fit it in alongside being a forex trader and a best-selling novelist, but when you're a twenty-something you don't need much sleep.

    --
    Confucius say, "Find worm in apple - bad. Find half a worm - worse."