Slashdot Mirror
Anyone Besides Zune Owners With New Year's Crashes?
aputerguy writes "My Fedora 8 Linux server crashed sometime between 18:59:40 EST (GMT -5:00) and 19:00:00 EST (GMT -5:00) on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same hardware non-stop for more than six years and other than the occasional reboot for kernel (or distro) upgrades, it has not crashed more than 1 or 2 times in 2237 days of cumulative uptime. Nothing other than background processes were running at the time of the crash. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues with their computers or electronic devices?"
Well, you know what they say, this wouldn't have happened with Red Hat.
Here in the UK, our skytv settop box crashed (lost all tv channels but not the menus precisley at 00:00 1/1/2009 needed a cold boot to get the channels back.
I let a bottle fall and it broke. Does it count?
debian etch, RHEL, centos, all 300 odd servers stayed up. so did irix and solaris boxen from ancient times of the roman empire..
No.
You are alone. Very, very alone.
My Microsoft Windows desktop crashed sometime between 18:59:40 EST (GMT -5:00) and 19:00:00 EST (GMT -5:00) on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same hardware non-stop for more than 5 hours and other than the occasional BSOD and Windows updates, it has not crashed more than 1 or 2 times in 174,237 seconds of cumulative uptime. Nothing other than spyware, malware, and System Idle Process were running at the time of the crash.
Sigs are for Terrorists.
Time mathematics are well known and extensively published. How come Microsoft cannot get it right? Every time something out of the usual happens (leapyears, daylight saving time changes, etc), Microsoft products crap out.
What tards.
Nope. Everything's fine here in New Ampst
--- What parts of "shall make no law", "shall not be infringed", and "shall not be violated" don't you understand?
Not really enough data to tell yet, but my Fedora 8 system had no issues at all. I'm also in GMT-5. Dual Xeon 32 bit system. E7520 chipset (Intel).
What sort of crash, just froze? Kernel OOPS, spontaneous reset?
Why don't you actually boot it, or failing that, take the hard drive out, perhaps look at some logs and actually find out rather than aligning it with a certain set of mystical circumstances?
Fedora 8 is rather old at this point, and there have been a number of important timezone data updates since then. It's not inconceivable that something went wrong.
My phone froze right after midnight and i had to remove the battery to make it work again.
It's a SE w810i.
> Could this be a coincidence
Yes. People are wired to see causality everywhere, even where there is none. Had your server crashed a week ago you wouldn't think anything of it (maybe 5% of all servers mysteriously crashed exactly one week ago, but because it was an 'ordinary' day nobody noticed). Anyway, since you noticed your server crashed at new year and reported it on /., and with 6 billion people on this planet we will soon hear stories about other computers that mysteriously crashed around midnight. Not because there has to be anything special, but because computers are crashing all the time and new year (and your post) made it appear special.
I doubt it has anything to do with leap seconds, if your computer ran for 6 years it survived the leap second of 2005.
As of this morning, Google thinks I have a spyware problem and every time I've tried to do a search I get the sorry page with a captcha to complete the search.
It sure scared me - how does a Linux box and a Macbook get infected while they're both asleep? I've done all the checking I can and I'm pretty sure my network is clean.
We had our proximity card access system completely shit the bed on the 31st. Don't know if it was a leap year issue or if it was just coincidence, but it caused widespread outages and was a major PITA.
How many servers in total are watched over by people posting on Slashdot? I suspect that the answer is high enough that it would be amazing if at least one of them didn't crash within 20 seconds of the New Year.
Virtually serving coffee
My parents are using a MythTV box on Fedora 8 (Athlon XP1700+) and it also froze up last night at the same time (right in the middle of a recording :-( ). That was my first thought, too, because that would have been midnight UTC. However, after restarting it today, is has frozen again.
I can't see anything in the logs, but the recording ended at 19:59 AST. It should have kept going for another hour.
I have a second MythTV/Fedora 8 box (P3, 1GHz) that I use and never had any trouble with it last night.
Fedora 8 EOL is 7th January, you should think about upgrading it to something newer...
Fedora 8 EOL Announcement
And FWIW, I've got a couple of hundred machines running varieties of Linux (but no Fedora 8) and I haven't seen any silly reboots. I think you might be making mountains out of a coincidental mole hill.
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
Sigs are too short to say anything truly profound so read the above post instead.
I haven't had a single Linux, BSD or *nix server crash like that and all Fedora 8 systems survived last night out of a pool of a several hundred servers ( all Linux, BSD and *nix ).
It sounds like unstable hardware.
Semper Fi
The Zune crash was due to a specific hardware driver. Perhaps you also have an unusual hardware driver on your setup that was affected?
I was watching the new years London celebrations on my Ubuntu 8.10 MythTV box. With 10 seconds to go to midnight, it crashed. Missed the start of the fireworks.
I think it may have happened around midnight before, so not necessarily an New Year problem.
Bunches of Linux servers (200+) running CentoOS, RHEL, CentOS, Gentoo, and a few others, all with recent versions of NTP, and all said something like:
Dec 31 18:59:59 aurora Clock: inserting leap second 23:59:60 UTC
when GMT rolled over.
On 12/30/08, I submitted a request with my pharmacy to refill a prescription to pick up on 12/31/08, and received the following email, verbatim:
Your Rite Aid prescription confirmation
Greetings from the riteaidonlinestore.com pharmacy,
Thank you for choosing to refill your Rite Aid prescription(s) online at the riteaidonlinestore.com pharmacy.
The following refills have been sent to the Rite Aid store that you selected, along with your preferred pick-up date and time:
Patient Name: ********
Rx ******** ********
Rx ******** ********
Rite Aid Store Location:
********
********, ********
********
********
Pick-up Date and Time:
Thursday December 31, 2009 at 3:00 pm
If you have any questions regarding your prescription, please contact your local Rite Aid directly at ********. Please note that you will need to pay for this prescription when you pick it up. If you have selected to self-pay for this medication, you will pay Rite Aid's price.
Thank you for visiting the riteaidonlinestore.com pharmacy. We invite you to visit us for your other prescription needs and great deals on nonprescription items. We look forward to assisting you!
Some things to note: I've got to wait until next christmas to pick up my drugs, and they were so concerned about patient privacy, they obscured all my contact information, prescription numbers and the pharmacy's phone numbers with asterisks. (I didn't do that myself!)
So, I wonder if their log files are full of java.lang.Exception logs today...
--ob
my cat hid under the bed at almost 25 seconds into the New Year. Right after he heard the first of the fireworks. However he did restart normally about 22 minutes later after a soothing saucer of milk. I wonder if ...
Posts, MyBio or Sig, may contain satire, sarcasm, bolded nouns be sardonic or even witty & be Church of SD
...to at least tell us why it crashed. Hard drive failure? Your fan gave out? Some sort of kernel panic or MCE? Was there a black cat in the server room? Maybe there's more Zune in it than you think.
--I'm so big, my sig has its own sig.
-- See?
No PC crash at midnight, but I found missing over 480'000 from my bank account, all what's left is barely 16'000. Previously I have there almost 500'000. I dunno where is the problem. By the way I'm from Slovakia.
Actually my server crashed between 00:59:52 and 01:03:53 CET so that is also very close to midnight GMT.
Like aputerguy it has been running continously for several years now, although unlike him the server was pretty busy running backups at the time of the crash.
It is running Debian testing and at that time Debian kernel 2.6.26-11.
I doubt this was the issue but this year there was a leap second added? something like that. Exactly 1 second before midnight they added a second... I think that's what I heard/read.. might have even been on slashdot.
A billing program I maintain failed to run today, I won't figure out why until tomorrow when I go back to work. Its no big deal but probably not a coinky-dink that it failed on new years day.
anyone know if it was Z2K-like?
I had purchased a Sandisk Sansa e260 4GB Media Player for my father-in-law for x-mas and it worked on thru the 30th, but it wouldn't turn-on on the 31st all day. We finally managed to turn it on today. Interesting...
Apparently, you have pre-existing stability problems with this box. The fact that it crashed yet again yesterday should come as no great surprise.
I always mod up spelling trolls.
You didn't specify your kernel version, but if it was 2.6.21, you may have hit this:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux2.6.gita=commitdiffh=746976a301ac9c9aa10d7d42454f8d6cdad8ff2b
Thankfully this was a short-lived bug which only affected 2.6.21.
I switched from Windows 95 to RedHat 6.2 many years ago, and except for reboots to upgrade the hardware (started with 200Mhz Pentium I w/ 384M and now have Dual Core w/ 2G) or OS (now on CentOS 5.2), it has crashed only twice - due to a defective USB2.0 card which I replaced.
We run LTSP so that the single server runs the entire family, using old '90s hardware for thin clients. We simply could not afford to run Windows (or Mac).
I just closed the screen before 11, and opened it again after midnight. Not saying its related to the time though.
In 2038, the UNIX time (the one most 32-bit operating systems use) will roll over back to 0. We must all get 64-bit at least before then.
If your computer was programmed to calculate time to something... .or something like that, for 30 years.... chances are it will crash while calculating it. But why, in Y2k9, might it do it?
See http://en.wikipedia.org/wiki/Y2K38 for a full explanation.
Cheers
Dan
PS. Hey it could be leap seconds, if you're synced to an NTP server, it might get confused when it's "not 2009 yet".
"Has anyone (other than Zune 30GB owners) noticed similar year-end issues with their computers or electronic devices?"
Of course they have. I didn't, but by following basic statistics (and without assuming a relation with the new year), you should expect that some of the readers here did.
My Fedora 8 system locked up after the leap second update was logged at 00:00 UT. I was my DHCP server, so the network went down.
My toast got burnt sometime between 9:59:40 EST (Eastern Standard Time in New South Wales, GMT +10:00) and 10:00:00 EST (GMT +10:00) on Jan 1, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been making toast with this same wetware non-stop for more than twenty six years and other than the occasional lapse in concentration while speaking on the phone, I have not burnt toast more than 1 or 2 times in 2237 days of cumulative toasting. Nothing other than background processes were running through my mind at the time of the burning. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues while operating toasters or electronic devices?
My Fedora 8 and Fedora 10 machines did not experience any problems. Maybe you had a power glitch, if there's nothing in the logs.
Can You Say Linux? I Knew That You Could.
I am running Redhat 7.2 on one of my servers w/o a problem. On one of my desktops, I am running IBM OS/2 (Ecs 2.0RC5) and there was no problem overnight.
Just because something is old does not mean it won't work. I have old pencils that have not crashed. My PC Lint 7.0 t-shirt from 1992 still works without a crash.
Fight Spammers!
Maybe I've missed something, but pointing out that some Linux-based servers crashed at midnight on New Years hardly seems like an advertisement to run Linux to me.
How are sites slashdotted when nobody reads TFAs?
On Debian, RHEL, Centos & Boxen! On Irix, Solaris, Ibex & Vixen!
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
The only special thing about 2008 I know of is that it is a leap year. That was the cause of the Zune problem. Could be the cause of other problems also.
Speaking of coincidences, many people said of the Z2K9 problem that it was a merely a coincidence that it happened on New Years and on a leap year--but they were wrong. One of an item (Zune, server, etc.) crashing at New Years is probably coincidence. Several probably is systemic.
Madplayer hicked three times at about 0100 CET. I thought it might have been my RAID system I had just repaired. (There was a bad sas/sata controller.) This happened over about 20 seconds. I only use Unix/Unix-like systems and to the best of my knowledge there are no embedded MS devices in this house.
Unix/Linux, etc. handles things like this well. All time sync services like NTP, DCF-77, MSF, WWVB, GPS and the rest give fair warning. I personally are in favour of ditching 'leap seconds'. Time corrections would best be made day to day, the length of today being based on yesterday. That's better, but surely someone can think up the real solution?
BillSF
PS: Frequent updates to Java caused by US daylight saving time are pathetic.
Definitely saw some funky behaviour last night...wonderful way to start the new year... Both of our UK-based DNS servers crashed about 2-3 seconds prior to midnight GMT time. Both of these systems are running Linux kernel 2.6.21 on older Dell 1850's. Our other DNS servers (US and Australia) all run 2.6.21 on newer Dell equipment (1950/2950), but none of them crashed. Both UK systems had to be power cycled, but afterwards they came up OK.
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
Remember to recreate the leap second that occurred just before midnight UTC (2008-12-31 18:59:60 EST).
Apparently my post is being censored.... lets try this again with a different subject line...
I had purchased a Sandisk Sansa e260 4GB Media Player for my father-in-law for x-mas and it worked on thru the 30th, but it wouldn't turn-on on the 31st all day. We finally managed to turn it on today. Interesting...
Take out the hard drive, blow on it and then reinsert it.
It didn't crash at the rollover time, GMT or PST, but sometime between 3am and 11am new year's day.
When I turned on the monitor this morning, a black screen with cursor flashing at top left position.
First time I've ever seen this crash, a B[lack]ScreenOfDeath on Linux.
You're certainly not entirely alone. I expect more reports to appear.
why is this front page?
Didn't yet get a call from a panic stricken rep wondering why they can't access their webapp. So I'm assuming that either the voip system is also dead or everything's going grea [...Connection Reset by Peer]
My Sony-Ericsson W910i crashed at midnight.
The X window system on my Debian box crashed at 23:59:59 UTC --- when I returned from watching the fireworks on TV, the clock was stuck at this time. As usual it wouldn't kill, so I have just finished repairing my Firefox profile due to the ensuing hard reset.
Were was Zune in 2000, when we really needed them?
I had one RHEL 4 server (out of a half dozen identically configured systems) crash at exactly the moment the leap second should have been inserted. The logs run up to 18:59:59 CST (UTC-06:00) and the system froze, when all my other Linux systems logged they were inserting a leap second. I have read reports of some Debian systems having a similar problem. The leap second code is probably one of the least tested areas of the Linux kernel (there have only been 8 leap seconds since Linux was started); there is probably some race condition related to stepping the clock that only some systems hit.
I have also read reports of problems with Oracle RAC (not stand-alone Oracle) crashing at the leap second.
Yes,
The latest Linux kernel 2.6.28 has a bug with regards to laptop screen brightness its sort of inverted.
This did not happen with 2.6.27.10
1:30AM up 495 days
no comments
My box crashed also. I've never had a kernel crash before. I asked at http://lists.debian.org/debian-user/2009/01/msg00006.html . One other person reported a similar problem. Someone mentioned it is a known problem with pre 2.6.21.6 kernels though I was running a more recent version.
Two Win2k boxes on my SOHO network came up with network access down/disabled. I was afraid my router had gone TU but this here Ubuntu box got online without problems. Reboots worked to get the Win2k machines online again.
FWIW
I bought this house and you know I'm boss
Ain't no h'aint gonna run me off
My Ubuntu Intrepid box (x86_64) went nuts last night while I was playing Warzone 2100. The timing was not especially close to the new year (maybe an hour?) but I hadn't seen it do this before, and I do play the game quite a lot. The game basically crashed and left me in a low-res xfce4 session.
Well right after new years my cellphone wouldnt let me text every time i tried it said ACK timeout. oddly enough it started working about an hour later
Apparently many Solaris systems restarted. People at NANOG are reporting this. A few banks' systems were rebooted as well: TD, Scotia, American Savings Bank, US Bank, and many more...
I saw many operating systems rebooting, even though this did not happen the last time in 2005.
Good thing I use ZFS on FreeBSD, and after I changed the loader.conf, I have a system that has stayed up for more than 2 months now, including last night.
My Roku rebooted on its own while I was watching a movie (Primer) @ ~1:30AM PST 1/1/09...
We had a report that one of our WinCE-based prototypes crashed and would not restart Wed (Dec 30).
Initially, we thought that it may have been due to an impatient user removing power during a normal shutdown, and may have occurred during a write to the flash memory drive. We can't tell exactly what happened from here (U.S.) as the prototype is in Europe.
And then again, it may just be co-incidental....
Dropped a Guru Meditation #00000025.65045338 while I was playing Giana Sisters... Is that a sign of the Apocalypse too?
Interestingly enough I saw exactly the same. Two of our production servers running fedora 8 crashed exactly at 01:01:02 GMT+1. I am beginning to suspect that this must be a Fedora 8/NTP-related bug...
My car battery died this morning. Who'd have thought that thing had a clock in it?!
My internet connection died exactly one minute before new years and stayed that way for 8 hours.....
Later, while diagnosing the situation, I discovered it was probably my big feet kicking the switch in the closet with the party favors when I went to get the toy horns, the uplink port to the firewall was partially pulled out.
At midnight GMT (well, UTC actually), we added a leap second. This seems to correspond with the time you're talking about. The most shocking thing is that this was not the first response to this story.
Could this be a coincidence or was there some 2008/2009 rollover issue going on here?
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
It might be difficult to reproduce if the system clock is NTP synched. Especially considering we had a leap second this year, where the clocks rolled from 23:59:59 to 23:59:60 to 00:00:00 GMT.
As someone else said: check your logs + try to reproduce.
--
Happy new Year
yea, my F8 box went down, but not right at midnight GMT /var/log/messages
here's the last line from
Dec 31 17:44:31 tomcat ntpd[2380]: kernel time sync status change 0011
I'm EST so it was 22:44:31 GMT
I guess it was talking to one of Redhat's ntpd servers
It's been rock solid in the past
My friends ATi x800 graphics card burst into flames after I overclocked it while he was away. He doesn't know that I've changed it for a GeForce FX hair dryer XD MWAAHAHAHAHAHA!
My slackware 12.0 box spiked the CPU at midnight GMT also... perhaps it is not the distrobution but a commonly run application which is causing the crash..
I'm only running an ircd and ejabberd on this server outside of the normal slack 12.0 service though
The answer to all your problems
It was the leap second
uptime 15:45:56 up 9 days uname -a Linux 2.6.21-7.fc7xen
since it has been running for six years (in italics!), then it has already gone through one leap year and nothing happened. sure someone might have slipped in a bus through an update, but it's unlikely.
My Fedora 9 server crashed around the same times yesterday . I found the keyboard lights flashing. I booted it at 19:51 EST. Last postfix message was at 17:59:57 Dec 31 17:59:57 bbs postfix/smtpd[1689]: messages : Dec 31 17:59:02 bbs snmpd[1871]: Received SNMP packet(s) from UDP: Dec 31 19:51:29 bbs kernel: imklog 3.20.2, log source = /proc/kmsg started.
Not 18:59, But kinda of seems odd..
I think it was the alcohol, but I'm not sure. I recovered some 8 hours later. why is the sun on the west and not the east?
Sounds like X crashed and you fell into vt7... where X normally is.
All you needed to do was press Ctrl+Alt+F1 (to switch to vt1), log in as root, and restart X.
Not sure how to do it on Ubuntu but on Gentoo it's "/etc/init.d/xdm restart".
Can a dummy NTP server be set up to reproduce this?
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
My Ubuntu 8.10 system froze around midnight GMT. There's nothing in the logs to indicate any problem, except that the last log entry is at Dec 31 15:55:02 (that's 5 minutes before midnight) for a cron job that runs every 5 minutes. I wasn't at home at the time, so I didn't see what happened, except that my ssh connection died.
When I got home a few hours later, the system was still powered on (the fan was running), but the screen was blank. I was able to reboot it with no problem.
It's not the same problem as on the Zune, since that hit 24 hours earlier. It *might* have something to do with the leap second, or maybe it's a problem that occurs at the end of the 366th day of the year.
I suppose I could set the clock back and run it through midnight again. I'm not sure I'll bother, but if somebody else wants to try it it would be interesting to see the results. Or maybe I'll try that with a live CD.
I'm a sysadmin for a small data centre.
This morning, at about 7am UTC 10 machines running Fedora Core 7 all locked up. They were on a private network inaccessible to the general internet. The lockups were all complete: not even Caps Lock on a connected keyboard worked (a good sign of kernel death). Rebooting and checking the logs gave no information: the last syslog entry was sometime after midnight and was normal. Once restarted (with a hard reset) they all seem to be functioning normally. I don't know yet what caused the lockup.
It's as if a million Fedora boxes cried out in terror and were suddenly silenced...
My wifes cell phone (Motorola) was locked up solid this morning. I pulled the battery and it did a serious reset (took about 5 minutes to start up) but it seems fine now.
Indeed, it's the known bug in 2.6.21 as mentioned on the debian lists: http://lists.debian.org/debian-user/2009/01/msg00006.html
So now I know, thanks slashdot!
My front door lock froze at midnight CET. Couldn't get back in after letting off fireworks. Fortunately there's a back door, but if that hadn't worked we could have tried the windows :-)
Every bloody emperor has his hand up history's skirt [Peter Hammill/VdGG]
One of your repliers has userid 1338699. There will be higher userids, but let's suppose that is the highest one. And let's suppose a ./ user experiences one computer crash per year. That's one crash every 23.5 seconds on average. There will be crashes near special points in time.
I run Quicken Deluxe 2002 under Crossover Office. Yesterday, under no circumstance could I get it to accept the fact that the date was December 31, 2008. Instead, it insisted the current date was January 1, 2009. I even tried simulating a Windows reboot, with no change. But this morning, all was well and Quicken once again was in sync with the system time. This is the first time in over 4 years of using Quicken 2002 under a Linux environment that it has given me any kind of problem. At least it corrected itself.
redoscar3
Happy New Year! :-)
O'WONDERWe're working on it.
My Canon 590IS camera lost it's date and time and required me to re-enter it on Jan 1, 2009.
The date and time were fine on Dec 31, but it looks like the camera didn't like the year change.
I believe all the G1's (google phone) show all messages received between Dec 29 and Dec 31 as being from the year 2009.
I personally are in favour of ditching 'leap seconds'.
How many of you are there?
Trying for some flip geek cool (if such a thing could ever actually exist) merely shows ignorance
Rudolph the Red-Nosed Reindeer says hi. You might not be able to hear him, though I don't know whether to blame the fact that the joke went miles over your head, or that you're wearing your asscheeks as earmuffs.
Raytheon ASD (Y2K project in 98 & 99) did find failure modes past 2012 due to software and firmwares. Near as I can remember some were in 2008. Industrial and commercial equipment is often be used > 30 years but little was tested for date related failures until 1997. Call it random failure unless you complete post mortem exam!
This is probably why there will never be a year of the linux desktop.
Nonsense, 2009 is the year of the linux desktop!
Flash player running on my 64bit Firefox crashed.
Onda Technology Institute
So if we don't see a bunch of "Mine crashed too!" comments we should get worried?
Some kind of new bug in the kernel? ;-/
Apart from my pacemaker, no, I didn't have any crashes.
This whole thing is weird - I've got firmware out in the wild that wouldn't encounter a bug like this. Dates for leap years are counted as the actual date, with math doing the calculations for leap years up until the end of 2100. (I figured that was enough time and if not, "I'll be dead anyway. See you in hell", as I put in the comments.)
As for space, it's on a 16LF88 with 7k of Flash. That chip was running at 32kHz most of the time, with occasional ramps up to 4MHz. The batteries were good for years at a time.
I'd know. The clients would call. I don't even work there anymore, and the clients would call.
What good is it to track the number of days in the year so far, and what barrier is being passed here? 365 isn't 2^anything, so you shouldn't have a rollover.
I've got a feeling that there's more to this than we're being told so far.
I'm just piggybacking on the troll so my post will get read.
---
ECHELON is a government program to find words like bomb, jihad, plutonium, assassinate, and anarchy.
I'm quite sure "boxen" is a germanism - and in that case it would be the correct plural form.
Want to hear the voice of GOD? cat
This is the same type of human response which says, "We've observed slightly higher levels of CO2 lately, we've observed slightly higher temperatures lately, humans burn things which create CO2 ... Humans must be destroying the planet!!"
The failure of schools to teach critical thinking is what will destroy the planet.
Our local cell tower went wonky in the early evening, around when Europe would be crossing over into 2009. Since companies like Ericsson and NSN make the vast majority of basestation equipment, I wonder if their internal time base is set to either GMT, or GMT+1 or GMT+2 to match their home country. If so, it could have been a New Year's glitch.
What do I mean by "went wonky?" Around 5:00 Central and for awhile after that, we couldn't send or receive phone calls regardless of having full-strength reception. If a call connected, it acted as if it had no reception or next to no reception. Calls dropped with a "connection error."
If you're curious, this was on AT&T, and was not a 3G connection.
Program Intellivision!
Perhaps it is the leap second that is coursing problems for computers using NTP and other time servers
null
http://isc.sans.org/diary.html?storyid=5602
"A leap second will be added to the clock at 12/31/2008 23:59:59 UTC tonight.
Hopefully most IT folks will be otherwise occupied at that time and not focusing on their system clocks."
Subprime Technologies
My old (and mostly unused) Windows XP machine, for which I bought a copy of XP a few years ago, today suddenly required re-activation. The key that came with the XP copy doesn't work anymore. But probably it has nothing to do with a year-2009-like bug.
Is your cat running windows?
Nein?
My IRC client (XChat) crashed at 00:00 GMT
Im sure they fixed this in an earlier version :|
The E segment of the last digit of the year failed at exactly midnight !!
Specifically I had one Debian machine crash, and 30 Debian machines, 5 Redhat machines, and various Linux workstations not crash. The machine that crashed has a lousy motherboard, and I see that Linux attempts to reset the CMOS clock following a leap second, so I wonder whether that was what took it out.
Perhaps this has somthing to do with it: http://www.itwire.com/content/view/22484/1066/
My old Sony-Ericsson T630 has been rebooting itself a couple of times a day over the last months (apparently known as the white screen of death). When I came home after celebrating the new year it was crashed in a rather spectacular way, though: The lights inside the keys were on, the screen was black and it responded to nothing at all. I had to remove the battery to get it going again. Since then it has only had its regular random reboots.
Against the grain
Some EVE killboards gave strange results starting on the 28th, the beginning of the 53rd weekly period in 2008, and began displaying results from the beginning of 2008. Apparently the coding of the killboards assumes years to have 52 weeks and when the 53rd week rolls around it goes all wonky.
How many of me? Just me, I'm the real BillSF..... If you mean how many are against the 'leap second' a quick search turned up this general article: http://www.newscientist.com/article/mg20026875.400-calls-to-scrap-the-leap-second-grow.html and there is quite a bit more. In fact the first comment to the New Scientist article almost sounds like me. (Its a coincidence, I'd not seen it before.)
I have a Strong 6120 satellite receiver (for french TNT SAT). I got today a "pb2" display on the LCD, never got that before, the clock was messed up. Note that this device is pure c**p as it fails to keep its settings between two reboots.
If your post is for real and not a joke, you may spefic a few things, but just some comments to point in the right direction in general:
1) Holidays are Script-Kiddie time /var/ which runs full may have the chance to do you server harm (you did not specify if i panicked or just was very unresponsive)
2) Script kiddies have the potential to fill your logs
3) A
4) You may have a cron job gone wild
5) Certificates ran out -> VPNs etc?
6) you tried to change sth like logrotate and screwed (this could have a 2 year delay) heavily
7) Something else failed (Power?)
8) Something esoteric happened (e.g. do you have a watchdog which has a bad driver?)
If you are using an standard out-of-the-box linux with out-of-the-box software, i would think you are alone.
I have a first-generation iPod Nano. Yesterday (the 31st) after I had downloaded an audiobook to it, it crashed totally. Was around 1500 my time, 2000 GMT. iTunes restore wouldn't fix it until I had reformatted the drive. Never happened before.
Probably just coincidence--but kind of spooky. FWIW I have time/date set to display in the menu bar, which is not the default.
...a reboot or shutdown from a relative who got fed up with you spending all your time online in order to haul you in for the newyear dinner? ;-)
1 system running ArchLinux and 3 machines running Ubuntu 7.04 survived the leap second :)
Oday ouyay antway otay ayplay away amegay?
I spilled champagne on my macbook because I was drinking Festive Filled Mugs in World of Warcraft and my character swayed back and forth, causing me to compensate in real life thereby knocking over the glass. Now it is bricked. Life sucks.
Vty,
Spanker @ Dragonblight Server, U.S.
P.S. Beautiful fireworks, though.
P.P.S. Blizzard, feel free to contact me about replacing the equipment. I will love you long time.
actually, I was just making a dumb pedantic joke about the fact that you used a singular subject with the plural form of "to be."
I administrate about 20 linux servers (debian and RHEL) and nothing happened. All are running fine. It's just you.
it might have been a joke this time, but how many times is it used on this site as the plural of box? I mean come on, its a valid complaint.
Could a crash be related to the power system? There could be fluctuations caused by spikes in usage, or perhaps suppliers switching sources as contracts end on the calendar year etc..
On the other hand, none of my servers suffered any ill effects... But my macbook pro was acting very strange this morning, programs were spuriously crashing (both safari and firefox, randomly crashing while browsing different sites including slashdot), turned it off for a few minutes, booted it back and everything is back to normal.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
My F8 server also "hung" some time after 18:58:59 as that was the last entry in /var/log/messages. I didn't notice it and get around to rebooting it until 19:51. (Yes, EST, that was just before 0:00 UT)
My F9 laptop continued with no apparent illness (other than taking a long time to resolve DNS names as the server was down).
My F10 test system logged the leap second. No other system in my network did.
Just more fodder for the argument.
We have about 20 servers running debian with 2.6.26-1-amd64. Four of the 20 locked up with utterly no warning around midnight GMT (we are on CDT, -6). When I got to the data center I saw another Sys Admin was working on one of his linux servers that had locked up at the same time.
All rebooted without incident. There were no log entries that gave a clue about what happened.
could it have been something to do with the leap second that was added in at 2008-12-31 23:59:60
I don't think web servers have progressed to possessing the kind of fuzzy logic where they can go 'OMG it's almost the end of a leapyear! I better freak out'
Seeing relations like that is more geared to humans, who sometimes teach computers to look for specific examples of such coincidence, but generally I believe they couldn't come up with it themselves, at least not on the webserver level.
Fedora 8, all up-to-date, uptime of 9 days. No show.
My g1 thinks that all SMS messages sent yesterday (12/32/08) were sent on 12-31-2009
bash-3.2$ cat /etc/redhat-release
Fedora release 8 (Werewolf)
bash-3.2$ date
Thu Jan 1 22:32:01 EST 2009
bash-3.2$ uptime
22:32:03 up 27 days, 8:33, 6 users, load average: 2.01, 1.94, 1.87
I do a "sudo yum update" approximately weekly.
This one surprised me. The entire ehow earnings system for its writers crashed and they still haven't fixed it. Apparently it wasn't rollover friendly. I don't remember but they've only started the system about 16-10 months ago so it's possible it has never seen a year rollover yet. Still, they're like the 39th most visited website in the US believe it or not so it's kinda odd that their entire writer's earnings system would fail.
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
A surprising number of NTP servers didn't add the leap second correctly. On the mailing list for pool.ntp.org contributors, it was reported that at just after midnight UTC that about 158 servers in the pool (about about 2000) were reporting times that were around 1000ms off. A few hours later it was only 13 that were doing that.
My own (stratum 3) NTP server got confused and declared that it couldn't determine the correct time. Some of its sources were 1000ms off from others. Given enough time, NTP will sort itself out, but I intervened manually by ditching the upstream servers that hadn't gotten it right.
If enough NTP servers were temporarily in the state that mine was in (was so unsure of itself that it wouldn't serve time to clients) then I could imagine some process that tries to sync the time and fails because ntpdate doesn't return anything useful.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
About as valid as complaining about knock knock jokes because there is really no one at the door.
He's probably a gas at parties.
My Ubuntu 8.10 box hung last night. I had a friend check IM logs... He's on Atlantic (GMT-400) time. Weird. (08:05:33 PM) qwell: that was random (08:05:41 PM) qwell: box just...froze (08:05:47 PM) qwell: wasn't even doing anything O.o
As of 10/06/03, I hate COBOL developers.
Yes, mine crashed too; last syslog entry is 5 mins before midnight GMT; next syslog (cron job) should have been a few seconds after midnight, but is not there.
I thought the crash was very unsual; this box never crashes for me; and I run heavy, CPU/RAM-intensive jobs on it (and am used to running it out of RAM). Notably: it was not even pingable --"typical" linux crashes leave the box at least pingable, so this caught my attention. (A pingable box might be swap-thrashing if it ran out of RAM)
-- AMD64 dual-core, in an ASUS M2N-E mobo
-- Ubuntu 8.04 (upgraded from older ubunutu/debians)
Another box did not crash:
-- Intel core2 duo Dell
-- Ubuntu 8.04
Both boxes were running the cpu/ram intense job. Both are used for random websurfing.
For more info, contact me at linas vepstas at gmail, (remove the space between my first and last name, and of course its at gmail.com)
My fedora 8 box has been running for 17 days (it had been running for a couple of months before we lost power for a week). So it isn't Fedora 8 per se. FWIW, my Fedora 10 box has been running for the same time (I need to think about upgrading the Fedora 8 box sometime).
I thought there was no way that Y2K9 would affect me, then the girlfriend asked me to check on a flight for her--and I found that United Airlines' website returns 2008 flight data if you search for flight information for Jan. 1 or Jan. 2 2009! How amateur is that?
~Ben
My server froze last night. It isn't the most reliable machine, maybe one issue every 2 months or so. I didn't even consider it an end-of-year thing until you posted this. I was on kernel 2.6.26.3-29.fc9. The last log entry I had before the system froze hard was at 18:55:57 EST, which is almost midnight GMT.
I supervise three fedora machines, all on the 2.6.26.3-29 kernel, and only one of them froze.
3 Ubuntu 8.10 Virtual Box guess just stopped almost like sending them a "VBoxManage controlvm xxx poweroff" command around midnight. Ubuntu 8.10 host was OK and the VMs started fine manually the next day when I got home to find mail server non existent. I have had some stability issues with 2 of the machines related to USB on the VIA chipset, but in those cases the logs document the event and 3 VMs have never bombed so completely at once.
Strange timing, my wife suggested Y2K 9 years late.
Yeah right, only that 'Box' is not a German word. Try 'Kisten' instead (plural of 'Kiste'=box). This doesn't mean that Box/Boxen is not used in colloquial German in the exact way the parent post talks about, because it is.
'Boxen' on the other hand is German for what you would call 'boxing' (the sports).
A World in a Grain of Sand / Heaven in a Wild Flower,
Infinity in the Palm of your Hand / And Eternity in an Hour.
My Fedora 8 machine (kernel: 2.6.26.6-49.fc8) crashed around midnight UTC as well. Last syslog message was at 23:40:07 UTC so it may have not happened at exactly midnight; it would be unusual not to have something logged for 20 minutes. When I got to the machine, it was completely unresponsive; couldn't get it to do anything but reboot. The hardware has been very reliable and it's on a UPS.
I have seen a thread on linux.debian.user about this happening on Debian.
Before someone points it out, yes, I know that support for Fedora 8 goes away in a week or so.
"My [thing] crashed sometime between [t] and [t+1] on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same [thing] non-stop for more than [long_time] and other than the occasional [maintenance_operation], it has not crashed more than 1 or 2 times in [toDays(long_time)] of cumulative uptime. Nothing other than [default_operations] were running at the time of the crash. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues with their [things]?"
At 12 midnite GMT all our Oracle RAC servers in both Dev and Production rebooted. I don't have to to tell you how horriffic this was. Here is What Oracle had to say.
There is an old Bug 5015469 - OPROCD REBOOTS NODE WHEN TIME IS SET BACK BY XNTPD
OPROCD initiated reboots. This differnce is big enough for the NTP daemon to set immediatly instead of drifting and when this
occurs the node is rebooted by OPROCD. Recently, we found that some of the OS platform still have this issue.
Servers' time synchronization through NTPD that cause OPROCD to reboot the node.
PROPOSED SOLUTION(S)
======================
1. Run the xntpd daemon with "-x" option to prevent server's time from moving backwards in large amounts which can cause
node eviction.
2. Apply CRS Bundle# 3, Patch 7117233, that fix the bug 5137401 - oprocd logfile is cleared after a reboot
This CRS Bundle#3 patch should be applied to CRS, ASM/RDBMS homes. It contains the fixes for Oracle Clusterware,
nothing change with database data dictionary.
3. Set the CSS DIAGWAIT parameter as per Metalink Note 559365.1. This will avoid false reboots due to OPROCD deamon.
Why bother saying, "Besides Zune owners?"
How many of those could there be?
I mean, willing to publicly admit it?
No shit? Box is not German? Next thing you'll be telling me that blinkenlights is not a German word either!
It may just be a coincidence, but my Gmail account became "disabled" on January 1. When I tried to log in the evening of the 1st, it came back with the message "Sorry, your account has been disabled. [?]". The question mark was a link to instructions on how to re-enable it, but those instructions are apparently out of date. They say that if your account is disabled, you can enter your username and password, and you'll be presented with a CAPTCHA; if you enter that, you'll be allowed to log in. However, I was never presented with a CAPTCHA, so I can't complete the steps.
Has this Gmail outage affected anybody else? Is this a repeat of the December 6th outage, or is it just me?
davidh
I had a total of 5 Linux machines running here at home and 2 of them locked hard sometime near midnight UTC.
One is my main server running Debian Etch and had been up for several months. It had not crashed once since replacing the hardware over a year ago. We were watching a recording on MythTV when it locked-up. I walked over to my workstation, running Ubuntu Intrepid, to restart the backend and it was also locked-up hard.
Very bizarre until I looked at the logs and did the math. Both logs end just before midnight UTC.
Russ W. Knize
I'm not sure of the time, but it locked up sometime during the night. After a measly 6 months of up time.
Oddly, my ISP seems to have experienced some kind of problem starting just after midnight. I monitor my aDSL modem/router activity every 10 minutes, and discovered that between 00:00:02 (the last entry in the log) and 00:10:01 the router reports zero network traffic, and continued like that until some time late yesterday afternoon. I did the usual diagnostics tests to ensure I wasn't to blame, but all seems well on my end. Without me taking any corrective action the problem disappeared, so I'm pretty confident it was in fact my ISP.
What's odd is that I was able to gain aDSL sync, and establish a PPP session (which involves logging in), but there seemed to be no IP routing going on at all. Pings would simply vanish.
It's times like this that I wish I had a more sophisticated modem/router so that I could capture the 'out' side of the device and see what's going on.
My 2nd gen 2 gig iPod nano froze yesterday. Came back after a soft boot to discover the battery was almost completely dead; was a little less that 50% when I last put it too sleep.
Over two thousand days of consecutive uptime?
Sounds like someone doesn't patch their kernel. Not a good idea, unless the machine's sole purpose was to set an uptime record.
Don't thank God, thank a doctor!
Yeah my car crashed due to the extra leap second. It was on the way home from a particularly boisterous newy years party. Damn microsoft!
I have a couple of Ware Brothers (Warebrothers.com) IP sprinkler controllers that I use, they all died on dec 31 around midnight.
What did the log files say ..
davecb5620@gmail.com
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
I suspect changing the clock will prove nothing. It is likely that a computer in isolation will happily count onwards, leap-second or no. I would have said all standalone software these days ought to be written well enough to take care of this, but there was the Zune...
The problem probably comes when you have several computers connected together, and only some of them are putting in the leap second. Initially the computers may check their clocks, as small errors are likely, but thereafter they may be synchronized. Then, all of a sudden, some of them are giving out time stamps that seem to be a whole second ahead of the others. It might be quite reasonable to reject packets of data with impossible timestamps, as their data might be corrupt, or it may be some hacker injecting fake packets, and they haven't got the clocks quite right. So you would have to reset the clocks on lots of computers - perhaps all the computers on the internet - to run a proper test, and even then there might be a freak combination of circumstances that stops it happening a second time.
Really, the only sensible thing to do is to do what the original poster did - write to Slashdot and find out whether there were other cases. He seems to have got a lot of replies saying "well duh! there are millions of computers so one is bound to crash close to the New Year, don't'cha understand probability?" which is rather unfair as you won't know you are the only one unless you ask. He hasn't got the fifty or so replies saying they saw the same thing that might have meant a systematic problem, so it was probably random.
Instead of inserting a leap second every 1.6 years, insert a leap minute every 100 years. That way the next time it happens will be 2108 and since none of us will be alive it will be someone else's problem.
naturally such a senior member of the administration is only ever served special cat milk which is very low in lactose.
Posts, MyBio or Sig, may contain satire, sarcasm, bolded nouns be sardonic or even witty & be Church of SD
There are *lots* of people read /. and those people are *likely* to have lots of machines around which all have a possibility of crashing at some stage ?
So, at *any* random time, posting this story on the front page is likely to correspond to something falling over nearby for more than a few people ?
Man, you remember 2000, when a load of shit ( didn't really very ) broke, wouldn't it be cool if that had happened again . . .
But past that nothing.
Strange enough, the hardware (pavement) turned my firmware into software, and it took a long time to recover.
My office latop was left on by me(it has been on since 6 months) as I never carried it home. Today morning when I came to office the screen saver was frozen. I run xfce and xscreensaver, screen saver freeze is common on my fedora 9. So I tried ssh and ping from another machine but none worked. It was very hot too,(i never found it hot in the past even after a return from 4 days of vacation). I had to hard reboot. From the cron log messages it seems to have died around 7 pm on Dec. 31st. /etc/cron.hourly) /etc/cron.hourly) /etc/cron.hourly)
Dec 31 16:01:01 linlap CROND[14254]: (root) CMD (run-parts
Dec 31 17:01:01 linlap CROND[16688]: (root) CMD (run-parts
Dec 31 18:01:01 linlap CROND[19115]: (root) CMD (run-parts
Jan 2 08:43:39 linlap crond[2461]: (CRON) STARTUP (1.0)
Jan 2 08:43:41 linlap anacron[2518]: Anacron 2.3 started on 2009-01-02
Jan 2 08:43:41 linlap anacron[2518]: Will run job `cron.daily' in 65 min.
Both my desktop and my coworker's desktop had to be rebooted this morning. Both run ubuntu 8.10 and both look like they crashed around midnight GMT
My server running Fedora Core 5 had no issues. Perhaps I should wait a wee bit longer to upgrade. ;-)
One of my Fedora 8 boxen runs a homebrewed perl script for monitoring and recording various network related information.
The current script runs a loop which makes it sleep for 10 seconds, and on waking checks if 60 seconds passed since the previous interrogation started in which case it will start interrogating the network again after which it continues the 10 second sleep loop.
The last entry in the log file occurred at 23:59:54 GMT, so I think it was doing a 'sleep 10' at midnight GMT (1 am local).
Nothing in system log or any files in /var/log close to that time and nothing between that time and the moment I re-booted.
I blame my crappy perl code and if it happens again on the next leap second I will certainly submit myself a bug report.
(Yes I should have used MRTG but it seemed like overkill when I started this quick and dirty - but now rather large and complex - utility)
My Fedora 8 notebook worked perfectly running lots of other programs at midnight GMT.
After I dropped my Ipod in a vat of acid, kicked it, let my brother in law borrow it, washed it down with an ultrasonic cleaning dip it seemed to work fine, but then I plugged it into a Windows Vista laptop.... :( I had to reset the ipod after that.... Still Re-Imaging the vista box though. only 1500 security patches left to go...
Whatever leads to a server crash is not a random event. There is a finite, albeit large, number of bugs in the code that can lead to a crash. In fact, a lot of things that we consider random aren't - such as dice tossing - there are just too many variables for we to track, so we just think of them as random.
But a server crash in any platform, is a different beast. THERE is a definite reason for the crash. One bug among those millions of lines of code caused the crash. It wasn't anything random, not even pseudo-random as a dice toss.
And notice that his machine was rock stable. And he wasn't running anything extraordinary. There is a definite chance it has something to do with the leap second. You're analising it as if server crashes simply occured out of thin air.
Where is that guy who'd die defending what I had to say when I need him?
I didn't see any problems on any of the company/outbound servers I'm working with out here in Europe.
Still, assumption is the f*ckup of mother nature; never just assume or your systems could be down a week later.
Whenever I hear a disk chirping I'm yanking the alarm bell immediately for further attention; same for fans failing or anything else. ... it would cost me my neck.
I don't just assume it's going to be allright just because
I've used to have a nasty bug with my linux systems rebooting every random x day(s) about a year and a half ago; it annoyed me for a week till I downgraded the kernel and gone the problem was. It was a conflict between the new kernel, existing modules and bad libraries .. By fixing the kernel I found out the other problems as well. It's running perfect since then... I just don't assume it's going to be allright while the behavior is abnormal ..
Moral for me - always be cautious when things happen differently than otherwise because that might be the start of (many) other trouble as I had with 2 systems before...
--- I am known for the ones who want to find me on the net. Is that a privacy risk or a privilege? One might wonder..
Hadn't thought about it, but my TiVo was locked up the evening of the 31st, after several years of working just fine. Though I hadn't used the TV for a couple of days so don't know when it happened.
Status/Summary of slashdot leap-second crash on new years 2008-2009
So far, 31 users reported 53 hard crashes at/near midnight, new years.
Symptoms are:
-- hard hang (systems not pingable)
-- irq's not serviced (if disk was active at time of crash,
the disk activity light stays lit)
-- cold reboot (poweroff) required
-- systems work normally after reboot
-- no messages in syslog, no kernel oops, core file crash dumps
-- not reproducible (simply setting the clock back is not enough
to reproduce; guessing that a simulation of stratum 0 ntp server
is needed to force the leap-second.)
-- The affected machines seem to be running either 2.6.21, 2.6.26 or 2.6.27
Suspect its an kernel race condition triggered by ntp bumping the second.
-- its the leap second, since this doesn't happen other years,
-- its a race condition, since some identically configured machines
didn't go down, while others did.
-- its a race condition, since majority of systems were not affected.
-- its a race condition, since affected systems seem to have been
mostly non-idle servers, or some non-idle desktops/ tv set-tops.
-- ntpd is the only service that monkeys with time adjustments.
There is a "well-known" deadlock in 2.6.21 kernels that caused this:
http://www.mail-archive.com/git-commits-head@vger.kernel.org/msg15039.html
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=746976a301ac9c9aa10d7d42454f8d6cdad8ff2b;hp=872aad45d6174570dd2e1defc3efee50f2cfcc72
aputerguy
Fedora 8 Linux server 2.6.26.6-49.fc8 Intel p4 2.8GHz.
ASUS P4PE 2 1-TB Seagate SATA hardrives, 1 200MB PATA drive. 2GB DDR. 1
pchdtv5500 card, 1 winfast 2000XP tv card, 1 nVidia 6200 graphics card.
MentalMooMan (785571) /var/log/messages at boot-time:
mythtv box (running mythbuntu) used to be something like 7.10 upgraded to 8.10
2.6.27-9.19-generic ntpd version 4.2.4p4. The CPU is an AMD Athlon XP
1700+ or 1800+. The motherboard is an EPOX EP-8KTA3Pro. message in
"warning: `ntpd' uses 32-bit capabilities (legacy support in use)"
Anonymous Coward MythTV box on Fedora 8 (Athlon XP1700+)
athakur999 (44340) Mythbuntu-based HTPC
AZPolarBear (661815) Fedora 8 system
Anonymous Coward 5 of about 70 of our production servers
Anonymous Coward I did have two 2.6.21 servers crash last night
Anonymous Coward Ubuntu 8.10 MythTV box.
SanjuroE (131728) Debian testing and at that time Debian kernel
2.6.26-11.
lukas84 (912874) internal testing machine that's still on 2.6.21
Anonymous Coward Debian testing Kernel 2.6.26
zerosumgame (1429741) kernel 2.6.21 on older Dell 1850's
Wibla (677432) Both my fileservers running debian etch installed from
custom install media (pre-etchnhalf) running 2.6.21-2 and 2.6.21-6
crashed,
Maow (620678) Ubuntu 8.04 on AMD64
Burdell (228580) RHEL 4 server
RHEL 4 update 6
kernel-smp-2.6.9-67.0.7.EL.x86_64
ntp-4.2.0.a.20040617-6.el4.x86_64
Penguin Computing Altus 2600
dual dual-core Opteron 2212 HE
4G RAM
nVidia MCP55 chipset
I have 9 servers (mostly
different hardware, but one the same as above), all running the exact
same kernel and package set. Only one crashed; the others logged the
leap second and went on fine.
Pretzalzz (577309) Travis Crump
http://lists.debian.org/debian-user/2009/01/msg00006.html
debian lenny ntp=1:4.2.4p4+dfsg-7;
Linux version 2.6.26-1-amd64 (Debian 2.6.26-4[since updated])
Processor : 2x AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
kst (168867) Ubuntu 8.10
arodland (127775) Debian 2.6.21
Goodgerster (904325)
Debian
Morgor (542294) Fedora 8 Two of our production servers running fedora 8
Lightjumper (532700) Fedora 9 server
blit (90883) 10 machines running Fedora Core 7
Qwell (684661) Ubuntu
Dear slashdot forum,
I'm a sysadmin for a small data centre. I never thought I something like this would happen to me, but this morning, at about 7am UTC 10 machines running Fedora Core 7 all locked up. They were on a private network inaccessible to the general internet. The lockups were all complete: not even Caps Lock on a connected keyboard worked (a good sign of kernel death). Rebooting and checking the logs gave no information: the last syslog entry was sometime after midnight and was normal. Once restarted (with a hard reset) they all seem to be functioning normally. I don't know yet what caused the lockup. I hope to meet again next "NYE" with these 10 Fedora boxes and have another adventure worthy of appearing on the slashdot forum pages.
When was the last time you updated your packages, subby?
I've been running my computer since the beginning of the week, and it never crashed.
yum update time From they way it sounds, yum check-update
The Rapture is NOT an exit strategy.
Identical thing happened to three production servers, two rhel5.2 xen, one rhel4.3 shared, same time. All machines required hard reboot before they were responsive again, could ping them with about a 0.1% success rate. Both rhel5.2 machines were using the latest redhat xen kernels.
Machines were in separate datacenters and operated by different companies. Spent most of that day trying to find any signs of abuse or failure but machines were fine, except for the crash.
That would explain why at 16:30 my Dell running Fedora 7 would not respond to keyboard or network input but the power light was lit. The most recent long entry was at Dec 31 15:59:39 and was from iptables logging NTP traffic. The old Red Hat 9 system was still playing music, unfazed.
Saw some CRS1's complaining they lost NTP sync at -6GMT (6pmCT). This was right after I read about the zunes, which made me quite concerned for what what about to transpire.
Haven't heard anything since. Glad this /. went up.
The crash has been reproduced, the fix has been found: See the LKML
http://lkml.org/lkml/2009/1/2/389
I'm nearly certain the grandparent meant "germanism" in the sense of "from a Germanic language", as opposed to "from the modern German language".
Nonetheless, that is false — a quick etymology lookup reveals that it came into Old English from Late Latin buxis, itself from Greek pyxis from pyxos.
"Ox" [pl. "oxen"], on the other hand, comes from Old English supposedly through Proto-Germanic; similar forms are found throughout Germanic languages, c.f. Old Norse oxi, Old Frisian oxa, Middle Dutch osse, German Ochse, Gothic auhsa.
PS: Frequent updates to Java caused by US daylight saving time are pathetic.
My first reaction was that many of these updates occurring are from other countries. Then I realized that these other countries might just be lagging in implementing their legislative changes to keep in step with specific US time zones.
Can anyone verify if most of these more recent java time updates are due to other countries aligning themselves with the US, or if they're just various countries doing their own thing?
Points of Interest
Java keeps it's time database aligned with the Olson time database.
However, if these updates remain frequent (as they are likely to do according to wikipedia) it sounds like Java needs a better mechanism for keeping time. Why does the VM keep time separately from the underlying OS anyways?
It sounds like a java connector to NTP is necessary so that running VMs can have their time databases updated on the fly (instead of having to bring down the VM and update).
two machines, running 2.6.26.2-server-1mnb and 2.6.27-desktop-0.rc8.2mnb, also locked solid here on 2009-01-01 00:00 UTC :(
I had one Gentoo Server with a 2.6.24 kernel I can't explain why it went network dead, after it was rebooted I could check the logs and see the machine had continued to work, it just didn't talk with anyone.
Vague inquiry posted on slashdot the 01 at 03:01PM -+?
open thread in lklm: 2 Jan 2009 13:25:38 +0000
http://lkml.org/lkml/2009/1/2/276
first patch: 3 Jan 2009 02:23:58
http://lkml.org/lkml/2009/1/2/389
report of fixed at 2 Jan 2009 21:49:53
http://lkml.org/lkml/2009/1/2/403
Oh, my god the debug guy start to travel time for you ...
Soon bugs will be patched before reported :)
This could make a good exemple for the quality of open source support ...
A small "lazy" post on /. and in two days 9 guy search, find a bug and have a patch.
It's not really a lazy post because the good info was there:
- "it has not crashed more than 1 or 2 times in 2237 days of cumulative uptime"
- "which remarkably corresponds to within at most 20 seconds of the New Year in GMT"