NTP Glitch Reverts Clocks Back To 2000

← Back to Stories (view on slashdot.org)

NTP Glitch Reverts Clocks Back To 2000

Posted by Soulskill on Tuesday November 20, 2012 @09:07AM from the it's-about-that-time-again dept.

An anonymous reader writes "It seems a glitch of some sort wreaked havoc on some NTP servers yesterday, causing many machines to revert to the year 2000. It seems the Y2K bug that never happened is finally catching up with us in 2012."

17 of 179 comments (clear)

Min score:

Reason:

Sort:

2012: the end of the world! by Anonymous Coward · 2012-11-20 09:09 · Score: 5, Funny

Oh sorry. My clock's off.
Not an NTP glitch by Anonymous Coward · 2012-11-20 09:13 · Score: 5, Informative

It was a problem with the USNO servers (I.e. tick.usno.navy.mil, tock.usno.navy.mil etc.) being rebooted and starting to hand out the wrong time. Very few downstream startum 2 NTP servers should have accepted such a large skew, although they may have lost accuracy.

Amusingly I happen to work with an ex. USNO NTP admin, so I'll be sure to take the piss for the rest of the week.
1. Re:Not an NTP glitch by Guspaz · 2012-11-20 09:18 · Score: 3, Interesting
  
  And end-user systems certainly don't accept that large a skew. ntpd on end-user systems would just have been unable to sync their time while the servers were affected.
2. Re:Not an NTP glitch by Anonymous Coward · 2012-11-20 10:02 · Score: 4, Informative
  
  Yes and no.
  This article is interesting: http://support.microsoft.com/kb/884776
  Summary: Windows can do it, but before Server 2008 it defaulted to not doing any sanity checks.
  Since 2008 it still is quite generous, allowing 48 hour jumps.
  If you don't like it you have to adjust the value in the registry.
  I guess it still shows that the Internet was an afterthought for Microsoft...
The Y2K bug was REAL by Anonymous Coward · 2012-11-20 09:14 · Score: 5, Insightful

Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard. Celebrate success on occasion, sheesh.
1. Re:The Y2K bug was REAL by asdf7890 · 2012-11-20 09:22 · Score: 5, Insightful
  
  Unfortunately most of the general public think that because nothing really went wrong there was not a problem in the first place, and that it was all hyped up by the media. Some of this is the simple truth that it was over-hyped by the media who over-hype everything so people are growing desensitised, some of it is people not bothering to research their opinions or properly engage their critical thinking abilities.
2. Re:The Y2K bug was REAL by mellon · 2012-11-20 09:27 · Score: 5, Insightful
  
  We have a built-in bias against successful disaster planning: since the planning was successful, the disaster didn't happen, and hence to the average observer, it looks like there wouldn't have been a disaster. The reasoning is flawed, of course, but apparently very hard to resist. This is why governments are only ever harmful—if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort. This cognitive flaw is why people who do diving catches are respected, and people who plan for the future and avoid problems are ignored, and why blithering idiots keep getting control of the reins and breaking things.
3. Re:The Y2K bug was REAL by ShanghaiBill · 2012-11-20 09:34 · Score: 4, Interesting
  
  Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard.
  Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four. But those of us that actually programmed in the days when 4K was a LOT of RAM, know that we never used two ASCII chars to store a year. We used a single byte to store the offset from 1900 in binary. So there will be no overflow until 2156.
4. Re:The Y2K bug was REAL by BobNET · 2012-11-20 09:57 · Score: 3, Interesting
  
  We used a single byte to store the offset from 1900 in binary.
  Most TRS-80 operating systems figured 3 bits was enough to store an offset from 1980, so we've already lived through the Y1.988k bug.
5. Re:The Y2K bug was REAL by brentrad · 2012-11-20 11:17 · Score: 3, Informative
  
  I beg to differ about not experiencing significant problems on 1/1/2000. We had significant issues that caused all our approximately 2000 store servers to repeatedly shut down until we unloaded the offending software.
  
  I was working for Hollywood Video in the Tech Support department (support for the rental stores and all their computer equipment) leading up to and after Y2K, in Wilsonville, Oregon at the corporate office. (Of course this was before their two bankruptcies.) The software development department performed extensive (and probably expensive) testing on every facet of our current in-store software and hardware setup (custom COBOL software running on DOS 5.0 on NetWare 3.1 if you can believe it.) They were even going to scrap NetWare in favor of a brand spanking new Windows NT Remote Desktop-type setup, but we were highly disappointed when NetWare came up with a patch for NetWare 3.X series to make it Y2K compatible, so they scrapped the NT plans. But I digress...
  
  Came in to work on 1/1/2000 a couple hours after midnight (yep they pretty much forced us to come in, and for very little extra pay - I may have been a bit drunk still.) Everything was already chaos: Almost every single store's NetWare server shut itself down at midnight, thinking there was a power outage. And since our stores' computers ran as dumb network-booted terminals to the main server, that means all the computers were down and rentals couldn't be performed except by writing the rentals on paper.
  
  Problem was, in the test lab someone had commented out the UPS backup auto-shutdown software line in the servers' autoexec (or its NetWare equivalent, might have been autoexec.ncf or something.) And yes, I do know who that someone was (wasn't me.) :) So I guess no one thought to test that particular software. So all the servers would boot up, immediately think there was a power outage, and immediately shut themselves off. We did have a manager's station computer in each store that had its own hard drive and could be used in emergencies, and had pcAnywhere and a modem, so we manually dialed into each of our approximately 2000 stores (at 14.4 kbps.) Then we walked a bunch of clueless managers and minimum wage kids through taking the new autoexec we had copied to a floppy on their manager's station (and a bunch of the stores had to run out and buy a box of floppies on New Year's Day) and booting up their servers using the floppy.
  
  I think we got the last few stores up and working by 2 or 3 pm Pacific. And before you say "who rents movies on New Year's Day?" - EVERYONE did. New Year's Day and Christmas Day were two of our biggest movie rental days of the year. People are home with their families, the festivities are over, everyone wants something to do and streaming from the internet didn't really exist yet. What did everyone do? Rent a video or go to a theater. I'm not sure how many tens of thousands of dollars in rentals we lost that day, but I'm sure it was significant.
  
  TL;DR: Just because you didn't hear about any significant losses due to Y2K bugs, doesn't mean they didn't happen. It's not like businesses were eager to admit they screwed up and forgot to test something.
6. Re:The Y2K bug was REAL by girlinatrainingbra · 2012-11-20 11:20 · Score: 3, Interesting
  
  Actually, most TRS-80 operating systems didn't even keep TRACK of time. It's the Disk Operating System (LDOS) that you're talking about. The TRS-80 didn't even have a built in clock / timer board / dedicated bits or bytes for knowing the time. And Level I and Level II Basic on it don't ask about the time or save it on the cassette tape system. And neither did the Apple ][+. You had to buy add-in clock boards to keep track of the time.
  Even the IBM PC did not have a real-time clock until the IBM-AT version in 1984. [Thank you wikipedia for prior info]. If you boot up an old IBM machine, one of the first things that the system asks after boot-up is to enter the time and date. [actual info from turning the old computers on in the garage].
2000 ZERO ZERO by Jeremiah+Cornelius · 2012-11-20 09:16 · Score: 5, Funny

Party's over,
Whoops! Out of time!

--
"Flyin' in just a sweet place,
Never been known to fail..."
Re:Maybe they could improve the algorithm? by Anonymous Coward · 2012-11-20 09:19 · Score: 3, Interesting

NTP -- at least, the formal Unix client -- already has checks in place to reject huge clock skews like this.
The failure, of course, was dipshit Windows clients talking directly to low-strata servers. And then blindly accepting such a massive skew. Good jorb there, Microsoft, as always.
Re:Maybe they could improve the algorithm? by DamonHD · 2012-11-20 09:22 · Score: 3, Interesting

Generally xntpd and its ilk will not step the time more than a small amount, but will rather give up and quit instead. One machine such as (say) the RPi with no RTC that use ntpdate to *set* rather than *adjust* the clock, this is harder to avoid, but servers should not be automatically running ntpdate when configured properly.
So there may be *two* bugs here to get the problem: one in an NTP implementation and one in use of ntpdate for anything other than manual updates on servers. But I haven't read TFA yet.
(On the little bit of work I put into the ARCRON driver I inserted extra code to look out for year rolls, etc, on top of whatever xntpd would do. In part because ARCRON only sends 2-year dates IIRC.)
Rgds
Damon

--
http://m.earth.org.uk/
The Y2K bug by geekoid · 2012-11-20 09:23 · Score: 5, Insightful

did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

--
The Kruger Dunning explains most post on /. http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect
1. Re:The Y2K bug by PPH · 2012-11-20 09:27 · Score: 5, Funny
  
  Trust the computer industry to shorten the term "Year 2000" to Y2K.
  It was this kind of thinking that got them in trouble in the first place.
  
  --
  Have gnu, will travel.
Properly configured hosts not impacted by jaredmauch · 2012-11-20 09:28 · Score: 5, Informative

If you saw this problem, your NTP time sources were not properly configured and diverse.
Consider using the NTP pool and not relying on so few sources to properly sync your time. Read 5.3.3 and 5.3.4 from http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers for help to correct your NTP setup.