Slashdot Mirror

← Back to Stories (view on slashdot.org)

NTP Glitch Reverts Clocks Back To 2000

Posted by Soulskill on from the it's-about-that-time-again dept.

An anonymous reader writes "It seems a glitch of some sort wreaked havoc on some NTP servers yesterday, causing many machines to revert to the year 2000. It seems the Y2K bug that never happened is finally catching up with us in 2012."

44 of 179 comments (clear)

  1. 2012: the end of the world! by Anonymous Coward · · Score: 5, Funny

    Oh sorry. My clock's off.

  2. Not an NTP glitch by Anonymous Coward · · Score: 5, Informative

    It was a problem with the USNO servers (I.e. tick.usno.navy.mil, tock.usno.navy.mil etc.) being rebooted and starting to hand out the wrong time. Very few downstream startum 2 NTP servers should have accepted such a large skew, although they may have lost accuracy.

    Amusingly I happen to work with an ex. USNO NTP admin, so I'll be sure to take the piss for the rest of the week.

    1. Re:Not an NTP glitch by Guspaz · · Score: 3, Interesting

      And end-user systems certainly don't accept that large a skew. ntpd on end-user systems would just have been unable to sync their time while the servers were affected.

    2. Re:Not an NTP glitch by Anonymous Coward · · Score: 2, Informative

      It seems that VMware ESXi servers grabbed the configuration with little issue.

    3. Re:Not an NTP glitch by lobiusmoop · · Score: 2

      end-user UNIX systems at least. TFA talks about Windows AD servers though, maybe they aren't doing the basic sanity checks on the upstream NTP data coming in?

      --
      "I bless every day that I continue to live, for every day is pure profit."
    4. Re:Not an NTP glitch by jamesh · · Score: 2

      And end-user systems certainly don't accept that large a skew. ntpd on end-user systems would just have been unable to sync their time while the servers were affected.

      Had your home router sync'd with such a server and been rebooted, it would have booted with the wrong time, and then wouldn't have accepted the corrected time when the error was fixed...

    5. Re:Not an NTP glitch by Anonymous Coward · · Score: 4, Informative

      Yes and no.
      This article is interesting: http://support.microsoft.com/kb/884776
      Summary: Windows can do it, but before Server 2008 it defaulted to not doing any sanity checks.
      Since 2008 it still is quite generous, allowing 48 hour jumps.
      If you don't like it you have to adjust the value in the registry.
      I guess it still shows that the Internet was an afterthought for Microsoft...

    6. Re:Not an NTP glitch by egamma · · Score: 2

      Most windows systems sync to time.microsoft.com.

      --
      Battlemaster--Game with friends in medival realms
    7. Re:Not an NTP glitch by Anonymous Coward · · Score: 2, Interesting

      Well... My "out of the box" (ie. no special tweaking) Linux on my laptop did accept it...
      I had to remove tick & tock.usno.navy.mil from my NTP config.

    8. Re:Not an NTP glitch by tlhIngan · · Score: 2

      Windows implements basically Daytime using NTP. What this means is instead of NTPd trying to adjust the clock to be in sync in small increments, Windows does what you do with Daytime instead - query the Daytime server, and set the clock. Except that since very few servers provide Daytime, Microsoft uses NTP to fetch the time and date.

      Even the latest Windows still does it - though if it drifts too far, it stops syncing. Very annoying as it doesn't stay synced and quietly fails.

    9. Re:Not an NTP glitch by jamesh · · Score: 2

      Had your home router sync'd with such a server and been rebooted, it would have booted with the wrong time, and then wouldn't have accepted the corrected time when the error was fixed...

      Why would it have accepted the big skew the first time, but not the second?

      A home router would typically not have a battery backed clock, and so on boot would simply accept whatever time it was given by the ntp server. Once it's up and running though it shouldn't accept a 12 year skew. My home router (openwrt) boots to something like 1/1/1970 I think.

    10. Re:Not an NTP glitch by AmiMoJo · · Score: 2

      As it happens I recently implemented the RTC in an embedded product and used a simple trick to reject obviously wrong times. I compare the date with the firmware build date and reject it if it is earlier. In the case of a router it could easily try other NTP servers in that case.

      That simple test rejects 99% of incorrect dates where something has been reset and started from zero again (which in this case would seem to be 01/01/2000).

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
  3. The Y2K bug was REAL by Anonymous Coward · · Score: 5, Insightful

    Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard. Celebrate success on occasion, sheesh.

    1. Re:The Y2K bug was REAL by asdf7890 · · Score: 5, Insightful

      Unfortunately most of the general public think that because nothing really went wrong there was not a problem in the first place, and that it was all hyped up by the media. Some of this is the simple truth that it was over-hyped by the media who over-hype everything so people are growing desensitised, some of it is people not bothering to research their opinions or properly engage their critical thinking abilities.

    2. Re:The Y2K bug was REAL by mellon · · Score: 5, Insightful

      We have a built-in bias against successful disaster planning: since the planning was successful, the disaster didn't happen, and hence to the average observer, it looks like there wouldn't have been a disaster. The reasoning is flawed, of course, but apparently very hard to resist. This is why governments are only ever harmful—if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort. This cognitive flaw is why people who do diving catches are respected, and people who plan for the future and avoid problems are ignored, and why blithering idiots keep getting control of the reins and breaking things.

    3. Re:The Y2K bug was REAL by ShanghaiBill · · Score: 4, Interesting

      Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard.

      Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four. But those of us that actually programmed in the days when 4K was a LOT of RAM, know that we never used two ASCII chars to store a year. We used a single byte to store the offset from 1900 in binary. So there will be no overflow until 2156.

    4. Re:The Y2K bug was REAL by Cinder6 · · Score: 2

      Not sure why you expect a bank, of all things, would get basic things right...

      --
      If you can't convince them, convict them.
    5. Re:The Y2K bug was REAL by Anonymous Coward · · Score: 2, Interesting

      Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four.

      While I'm sure plenty of organizations did nothing and were fine, I know for a fact that storing years as two bytes was not a myth. At my organization I fixed such a problem in critical systems. We would have had people collecting GPS data in the field with no way to verify it's quality if I hadn't. Not doom for our organization, but not a "myth" as you call it.

    6. Re:The Y2K bug was REAL by operagost · · Score: 2

      This is why governments are only ever harmfulâ€"if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort

      That's funny, because I always see it in reverse with the US government: if we hadn't had the stimulus, unemployment would have been much worse, or if we didn't have federal loans, no one could afford college, or if we hadn't passed the Patriot Act, we would have had lots of terrorist attacks...

      Ignoring, of course, that we have unemployment higher than they predicted anyway, and we have young people racking up debt they can never pay off (and can't be discharged) to gets jobs that don't exist, and we've had terrorist attacks on our military bases, recruiting centers, and embassies.

      --

      Gamingmuseum.com: Give your 3D accelerator a rest.
    7. Re:The Y2K bug was REAL by BobNET · · Score: 3, Interesting

      We used a single byte to store the offset from 1900 in binary.

      Most TRS-80 operating systems figured 3 bits was enough to store an offset from 1980, so we've already lived through the Y1.988k bug.

    8. Re:The Y2K bug was REAL by Cartotype · · Score: 2

      I know of at least one organization which had a significant Y2K problem, even after making preparations.

      Sadly, the preparations were "Hire someone to take the fall when the shit hits the fan so we can continue with business as usual. Er... hire someone to ensure Y2K preparedness."

      The fatal glitch in the plan was that the person who got hired made friends with an exec in the parent company before the ball dropped. So, when things went south the hire got a silver parachute while the rest of the company folded.

      Quite a mess. Should certainly count as a "significant problem".

    9. Re:The Y2K bug was REAL by brentrad · · Score: 3, Informative

      I beg to differ about not experiencing significant problems on 1/1/2000. We had significant issues that caused all our approximately 2000 store servers to repeatedly shut down until we unloaded the offending software.

      I was working for Hollywood Video in the Tech Support department (support for the rental stores and all their computer equipment) leading up to and after Y2K, in Wilsonville, Oregon at the corporate office. (Of course this was before their two bankruptcies.) The software development department performed extensive (and probably expensive) testing on every facet of our current in-store software and hardware setup (custom COBOL software running on DOS 5.0 on NetWare 3.1 if you can believe it.) They were even going to scrap NetWare in favor of a brand spanking new Windows NT Remote Desktop-type setup, but we were highly disappointed when NetWare came up with a patch for NetWare 3.X series to make it Y2K compatible, so they scrapped the NT plans. But I digress...

      Came in to work on 1/1/2000 a couple hours after midnight (yep they pretty much forced us to come in, and for very little extra pay - I may have been a bit drunk still.) Everything was already chaos: Almost every single store's NetWare server shut itself down at midnight, thinking there was a power outage. And since our stores' computers ran as dumb network-booted terminals to the main server, that means all the computers were down and rentals couldn't be performed except by writing the rentals on paper.

      Problem was, in the test lab someone had commented out the UPS backup auto-shutdown software line in the servers' autoexec (or its NetWare equivalent, might have been autoexec.ncf or something.) And yes, I do know who that someone was (wasn't me.) :) So I guess no one thought to test that particular software. So all the servers would boot up, immediately think there was a power outage, and immediately shut themselves off. We did have a manager's station computer in each store that had its own hard drive and could be used in emergencies, and had pcAnywhere and a modem, so we manually dialed into each of our approximately 2000 stores (at 14.4 kbps.) Then we walked a bunch of clueless managers and minimum wage kids through taking the new autoexec we had copied to a floppy on their manager's station (and a bunch of the stores had to run out and buy a box of floppies on New Year's Day) and booting up their servers using the floppy.

      I think we got the last few stores up and working by 2 or 3 pm Pacific. And before you say "who rents movies on New Year's Day?" - EVERYONE did. New Year's Day and Christmas Day were two of our biggest movie rental days of the year. People are home with their families, the festivities are over, everyone wants something to do and streaming from the internet didn't really exist yet. What did everyone do? Rent a video or go to a theater. I'm not sure how many tens of thousands of dollars in rentals we lost that day, but I'm sure it was significant.

      TL;DR: Just because you didn't hear about any significant losses due to Y2K bugs, doesn't mean they didn't happen. It's not like businesses were eager to admit they screwed up and forgot to test something.

    10. Re:The Y2K bug was REAL by girlinatrainingbra · · Score: 3, Interesting

      Actually, most TRS-80 operating systems didn't even keep TRACK of time. It's the Disk Operating System (LDOS) that you're talking about. The TRS-80 didn't even have a built in clock / timer board / dedicated bits or bytes for knowing the time. And Level I and Level II Basic on it don't ask about the time or save it on the cassette tape system. And neither did the Apple ][+. You had to buy add-in clock boards to keep track of the time.
      Even the IBM PC did not have a real-time clock until the IBM-AT version in 1984. [Thank you wikipedia for prior info]. If you boot up an old IBM machine, one of the first things that the system asks after boot-up is to enter the time and date. [actual info from turning the old computers on in the garage].

    11. Re:The Y2K bug was REAL by Darinbob · · Score: 2

      However time_t, whether 32-bit or 64-bit, was intended for file dates. It was not intended for general all purpose date handling. And yet people do this. They use the signed 32-bit time_t for storing birthdays for example. Or the 64-bit time is used without accounting for historical calendar changes.

      And MANY embedded systems out there use a signed 32-bit Unix time (some even with unsigned 32-bit, which is perfectly logical for file creation times). There are even Unixes still running that don't do 64 bit time_t. There are DOS based systems out there still.

    12. Re:The Y2K bug was REAL by rk · · Score: 2

      Just because you're not punching kids out THIS VERY MINUTE doesn't mean your cost/benefit to the gene pool drops to near zero. Humans in modern societies take 2 decades or more to become self-sustaining, to say nothing of the aid (financial, emotional, or educational) provided by grandparents and then the very act of going to work in the morning, living somewhere and paying property taxes for schools (ideally) educating kids still adds benefit, regardless of whether you have kids of your own or not. And funny thing: People have their own metrics of cost/benefit in any case and in my experience it's not often it's the gene pool at large they're considering when they evaluate that metric.

      But you are right: you cannot be prepared for everything. That's in part why we have insurance: We turn an unpredictable but potentially unbearable cost into a predictable but certain affordable cost, turning over that risk management to people who (we hope!) are better at it than we are. Presumably this is why my flood insurance was a lot more expensive and covered less when I lived in what was basically a swamp 10 miles from the hurricane prone coast than the house I have now in high ground in the desert. :-)

  4. 2000 ZERO ZERO by Jeremiah+Cornelius · · Score: 5, Funny

    Party's over,
    Whoops! Out of time!

    --
    "Flyin' in just a sweet place,
    Never been known to fail..."
  5. Re:Maybe they could improve the algorithm? by Anonymous Coward · · Score: 3, Interesting

    NTP -- at least, the formal Unix client -- already has checks in place to reject huge clock skews like this.

    The failure, of course, was dipshit Windows clients talking directly to low-strata servers. And then blindly accepting such a massive skew. Good jorb there, Microsoft, as always.

  6. RTFA by Anonymous Coward · · Score: 2, Funny

    For those of you who didn't read the article; US Naval Observatory rebooted one of their NTP servers (tock.usno.navy.mil I believe, though not sure), and the server's time reverted to 2000.. This time change got pushed out, and thus people freaked out claiming that NTP was broken.

    A lesson can be learned from this: change your BIOS battery when it's dead (alternatively, never reboot your servers).

    1. Re:RTFA by marcosdumay · · Score: 2

      alternatively, never reboot your servers

      Or, better. Reboot your servers one at a time, a couple of times a year. That way, when a problem happens (may it be the RTC battery failing, bad init scripts or watever else) you'll catch it in a much more safe way.

      --
      Rethinking email
  7. Re:Maybe they could improve the algorithm? by DamonHD · · Score: 3, Interesting

    Generally xntpd and its ilk will not step the time more than a small amount, but will rather give up and quit instead. One machine such as (say) the RPi with no RTC that use ntpdate to *set* rather than *adjust* the clock, this is harder to avoid, but servers should not be automatically running ntpdate when configured properly.

    So there may be *two* bugs here to get the problem: one in an NTP implementation and one in use of ntpdate for anything other than manual updates on servers. But I haven't read TFA yet.

    (On the little bit of work I put into the ARCRON driver I inserted extra code to look out for year rolls, etc, on top of whatever xntpd would do. In part because ARCRON only sends 2-year dates IIRC.)

    Rgds

    Damon

    --
    http://m.earth.org.uk/
  8. The Y2K bug by geekoid · · Score: 5, Insightful

    did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

    --
    The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
    1. Re:The Y2K bug by PPH · · Score: 5, Funny

      Trust the computer industry to shorten the term "Year 2000" to Y2K.

      It was this kind of thinking that got them in trouble in the first place.

      --
      Have gnu, will travel.
    2. Re:The Y2K bug by MobyDisk · · Score: 2

      Wait, isn't Y2K going to happen in 2048?

  9. Properly configured hosts not impacted by jaredmauch · · Score: 5, Informative

    If you saw this problem, your NTP time sources were not properly configured and diverse.

    Consider using the NTP pool and not relying on so few sources to properly sync your time. Read 5.3.3 and 5.3.4 from http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers for help to correct your NTP setup.

  10. RIP Charles M. Schulz by Ol+Biscuitbarrel · · Score: 2

    Poor Elián González! At least I was able to get ILOVEYOU off of my Gateway. Have you checked out Dora the Explorer?

  11. But did it skew the pool ? by SimplexBang · · Score: 2

    pool.ntp.org is our one and only external resource that we need .

    --
    Avoid your fears , or wonder at the past
  12. Re:Maybe they could improve the algorithm? by Bigby · · Score: 2

    Or the computer just traveled near the speed of light.

  13. My work was effected by bufke · · Score: 2

    Last night AD went to year 2000. Linux clients joined to AD freaked out right away with scary security warnings. A reboot fixed them right away after AD time was set right. Or setting time manually.

    Windows had authentication trouble so Shares, Printers, etc stop working for clients. For some reason I had to rebooting them many times fixed them, no idea why one wasn't enough. Had trouble reported all today even though it was only off for 2 hours last night.

    Hard to believe Windows doesn't do sanity checking on this stuff.

  14. No Warning?! by organgtool · · Score: 2

    Why didn't you warn us about this time-based bug, John Titor?!

  15. This happened to me in 1999 by MichaelSmith · · Score: 2

    Our NTP sync came from a remote site and they were doing Y2K testing. Clocks jumped forward to March 2000 and screwed up the timestamps on our binaries. In the end we had to touch backwards to get our build system working again.

    --
    http://michaelsmith.id.au
  16. and the 2038 bug as well as the Year 2042 bug by Joe_Dragon · · Score: 2

    and the 2038 bug as well as the Year 2042 bug are still to come.

    Also maybe a Day 32,768 and 65,536 bug as well.

  17. The Worst Part about the Y2K bug by Crypto+Gnome · · Score: 2

    Was not the bug. Nor was it The Fix(ing/es/etc).

    It was THE MONEY.

    A large part of "LALALALA I CAN'T HEAR YOU" came from arguments which essentially boil down to "but if that's actually true, then fixing it would cost LOTS OF MONEY".

    Whoever said HISTORY NEVER REPEATS is a complete moron.

    These days s/Y2K bug/Global Warming/ and the situation is EXACTLY THE SAME.

    Lots of industries pushing for FLAT OUT DENIAL mainly because of attitudes which amount to not much more than "but if that's actually true, then fixing it would cost LOTS OF MONEY".

    Insert here well documented evidence that THE TOBACCO INDUSTRY knew, many many may ears ago that smoking tobacco was (a) bad for you and (b) addictive.

    Big Business KNOWS perfectly well that we're pretty much up Fitz Creek without a paddle (or even a canoe) but THEY DON'T CARE as long as they make lots of money TODAY.

    Show me one CEO where their bonuses/etc are indexed against ANYTHING more than the next 1-5 years of profit.

    An environmental (and therefore economic) CATASTROPHE that will occur no sooner than 10 years from now is akin to the heat-death-of-the-universe, we ALL know it's coming - but NOBODY CARES.

    --
    Visit CryptoGnome in his home.
  18. Those affected were mostly following instructions by jthill · · Score: 2

    A large part of NTP's sophistication lies in its ability to identify falsetickers/truechimers among multiple candidate sources. NTP is built to survive failures like this, but Microsoft apparently doesn't bother telling its Active Directory customers -- it actually instructs them to single-source their clocks.

    --
    As always, all IMO. Insert "I think" everywhere grammatically possible.
  19. Re:Maybe they could improve the algorithm? by Pieroxy · · Score: 2

    Yes, this is all Microsoft's fault because with UNIX apps, you're expected to deal with this sort of bullshit.

    You SHOULD expect this kind of bullshit. Because in real life, shit happens. Servers do reboot. Servers do see their clocks reset. Yes, even time servers.

    In general, YOU are responsible for your own system. You may query other servers to check their time, but in the end, it's your program that sets your computer's time, so it's its own responsibility to do the right thing.

    A check against something obviously wrong is just common sense.

    --
    Write boring code, not shiny code!