Slashdot Mirror
Anyone Besides Zune Owners With New Year's Crashes?
aputerguy writes "My Fedora 8 Linux server crashed sometime between 18:59:40 EST (GMT -5:00) and 19:00:00 EST (GMT -5:00) on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same hardware non-stop for more than six years and other than the occasional reboot for kernel (or distro) upgrades, it has not crashed more than 1 or 2 times in 2237 days of cumulative uptime. Nothing other than background processes were running at the time of the crash. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues with their computers or electronic devices?"
Well, you know what they say, this wouldn't have happened with Red Hat.
Here in the UK, our skytv settop box crashed (lost all tv channels but not the menus precisley at 00:00 1/1/2009 needed a cold boot to get the channels back.
I let a bottle fall and it broke. Does it count?
debian etch, RHEL, centos, all 300 odd servers stayed up. so did irix and solaris boxen from ancient times of the roman empire..
No.
You are alone. Very, very alone.
My Microsoft Windows desktop crashed sometime between 18:59:40 EST (GMT -5:00) and 19:00:00 EST (GMT -5:00) on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same hardware non-stop for more than 5 hours and other than the occasional BSOD and Windows updates, it has not crashed more than 1 or 2 times in 174,237 seconds of cumulative uptime. Nothing other than spyware, malware, and System Idle Process were running at the time of the crash.
Sigs are for Terrorists.
Nope. Everything's fine here in New Ampst
--- What parts of "shall make no law", "shall not be infringed", and "shall not be violated" don't you understand?
Not really enough data to tell yet, but my Fedora 8 system had no issues at all. I'm also in GMT-5. Dual Xeon 32 bit system. E7520 chipset (Intel).
What sort of crash, just froze? Kernel OOPS, spontaneous reset?
Why don't you actually boot it, or failing that, take the hard drive out, perhaps look at some logs and actually find out rather than aligning it with a certain set of mystical circumstances?
My phone froze right after midnight and i had to remove the battery to make it work again.
It's a SE w810i.
> Could this be a coincidence
Yes. People are wired to see causality everywhere, even where there is none. Had your server crashed a week ago you wouldn't think anything of it (maybe 5% of all servers mysteriously crashed exactly one week ago, but because it was an 'ordinary' day nobody noticed). Anyway, since you noticed your server crashed at new year and reported it on /., and with 6 billion people on this planet we will soon hear stories about other computers that mysteriously crashed around midnight. Not because there has to be anything special, but because computers are crashing all the time and new year (and your post) made it appear special.
I doubt it has anything to do with leap seconds, if your computer ran for 6 years it survived the leap second of 2005.
As of this morning, Google thinks I have a spyware problem and every time I've tried to do a search I get the sorry page with a captcha to complete the search.
It sure scared me - how does a Linux box and a Macbook get infected while they're both asleep? I've done all the checking I can and I'm pretty sure my network is clean.
We had our proximity card access system completely shit the bed on the 31st. Don't know if it was a leap year issue or if it was just coincidence, but it caused widespread outages and was a major PITA.
How many servers in total are watched over by people posting on Slashdot? I suspect that the answer is high enough that it would be amazing if at least one of them didn't crash within 20 seconds of the New Year.
Virtually serving coffee
My parents are using a MythTV box on Fedora 8 (Athlon XP1700+) and it also froze up last night at the same time (right in the middle of a recording :-( ). That was my first thought, too, because that would have been midnight UTC. However, after restarting it today, is has frozen again.
I can't see anything in the logs, but the recording ended at 19:59 AST. It should have kept going for another hour.
I have a second MythTV/Fedora 8 box (P3, 1GHz) that I use and never had any trouble with it last night.
Fedora 8 EOL is 7th January, you should think about upgrading it to something newer...
Fedora 8 EOL Announcement
And FWIW, I've got a couple of hundred machines running varieties of Linux (but no Fedora 8) and I haven't seen any silly reboots. I think you might be making mountains out of a coincidental mole hill.
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
Sigs are too short to say anything truly profound so read the above post instead.
I haven't had a single Linux, BSD or *nix server crash like that and all Fedora 8 systems survived last night out of a pool of a several hundred servers ( all Linux, BSD and *nix ).
It sounds like unstable hardware.
Semper Fi
The Zune crash was due to a specific hardware driver. Perhaps you also have an unusual hardware driver on your setup that was affected?
I was watching the new years London celebrations on my Ubuntu 8.10 MythTV box. With 10 seconds to go to midnight, it crashed. Missed the start of the fireworks.
I think it may have happened around midnight before, so not necessarily an New Year problem.
Bunches of Linux servers (200+) running CentoOS, RHEL, CentOS, Gentoo, and a few others, all with recent versions of NTP, and all said something like:
Dec 31 18:59:59 aurora Clock: inserting leap second 23:59:60 UTC
when GMT rolled over.
On 12/30/08, I submitted a request with my pharmacy to refill a prescription to pick up on 12/31/08, and received the following email, verbatim:
Your Rite Aid prescription confirmation
Greetings from the riteaidonlinestore.com pharmacy,
Thank you for choosing to refill your Rite Aid prescription(s) online at the riteaidonlinestore.com pharmacy.
The following refills have been sent to the Rite Aid store that you selected, along with your preferred pick-up date and time:
Patient Name: ********
Rx ******** ********
Rx ******** ********
Rite Aid Store Location:
********
********, ********
********
********
Pick-up Date and Time:
Thursday December 31, 2009 at 3:00 pm
If you have any questions regarding your prescription, please contact your local Rite Aid directly at ********. Please note that you will need to pay for this prescription when you pick it up. If you have selected to self-pay for this medication, you will pay Rite Aid's price.
Thank you for visiting the riteaidonlinestore.com pharmacy. We invite you to visit us for your other prescription needs and great deals on nonprescription items. We look forward to assisting you!
Some things to note: I've got to wait until next christmas to pick up my drugs, and they were so concerned about patient privacy, they obscured all my contact information, prescription numbers and the pharmacy's phone numbers with asterisks. (I didn't do that myself!)
So, I wonder if their log files are full of java.lang.Exception logs today...
--ob
my cat hid under the bed at almost 25 seconds into the New Year. Right after he heard the first of the fireworks. However he did restart normally about 22 minutes later after a soothing saucer of milk. I wonder if ...
Posts, MyBio or Sig, may contain satire, sarcasm, bolded nouns be sardonic or even witty & be Church of SD
...to at least tell us why it crashed. Hard drive failure? Your fan gave out? Some sort of kernel panic or MCE? Was there a black cat in the server room? Maybe there's more Zune in it than you think.
--I'm so big, my sig has its own sig.
-- See?
Actually my server crashed between 00:59:52 and 01:03:53 CET so that is also very close to midnight GMT.
Like aputerguy it has been running continously for several years now, although unlike him the server was pretty busy running backups at the time of the crash.
It is running Debian testing and at that time Debian kernel 2.6.26-11.
A billing program I maintain failed to run today, I won't figure out why until tomorrow when I go back to work. Its no big deal but probably not a coinky-dink that it failed on new years day.
anyone know if it was Z2K-like?
I had purchased a Sandisk Sansa e260 4GB Media Player for my father-in-law for x-mas and it worked on thru the 30th, but it wouldn't turn-on on the 31st all day. We finally managed to turn it on today. Interesting...
Apparently, you have pre-existing stability problems with this box. The fact that it crashed yet again yesterday should come as no great surprise.
I always mod up spelling trolls.
You didn't specify your kernel version, but if it was 2.6.21, you may have hit this:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux2.6.gita=commitdiffh=746976a301ac9c9aa10d7d42454f8d6cdad8ff2b
Thankfully this was a short-lived bug which only affected 2.6.21.
I switched from Windows 95 to RedHat 6.2 many years ago, and except for reboots to upgrade the hardware (started with 200Mhz Pentium I w/ 384M and now have Dual Core w/ 2G) or OS (now on CentOS 5.2), it has crashed only twice - due to a defective USB2.0 card which I replaced.
We run LTSP so that the single server runs the entire family, using old '90s hardware for thin clients. We simply could not afford to run Windows (or Mac).
Fedora 8's end of life doesn't occur until January 7th, so it would still get timezone updates.
This is here so you don't ignore the last two lines of my posts.
My Fedora 8 system locked up after the leap second update was logged at 00:00 UT. I was my DHCP server, so the network went down.
My toast got burnt sometime between 9:59:40 EST (Eastern Standard Time in New South Wales, GMT +10:00) and 10:00:00 EST (GMT +10:00) on Jan 1, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been making toast with this same wetware non-stop for more than twenty six years and other than the occasional lapse in concentration while speaking on the phone, I have not burnt toast more than 1 or 2 times in 2237 days of cumulative toasting. Nothing other than background processes were running through my mind at the time of the burning. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues while operating toasters or electronic devices?
My Fedora 8 and Fedora 10 machines did not experience any problems. Maybe you had a power glitch, if there's nothing in the logs.
Can You Say Linux? I Knew That You Could.
I am running Redhat 7.2 on one of my servers w/o a problem. On one of my desktops, I am running IBM OS/2 (Ecs 2.0RC5) and there was no problem overnight.
Just because something is old does not mean it won't work. I have old pencils that have not crashed. My PC Lint 7.0 t-shirt from 1992 still works without a crash.
Fight Spammers!
Maybe I've missed something, but pointing out that some Linux-based servers crashed at midnight on New Years hardly seems like an advertisement to run Linux to me.
How are sites slashdotted when nobody reads TFAs?
On Debian, RHEL, Centos & Boxen! On Irix, Solaris, Ibex & Vixen!
My first Journal Entry ever, in 8 years! http://slashdot.org/journal/365947/aphelion-scifi-fantasy-horror-poetry-webzine
The only special thing about 2008 I know of is that it is a leap year. That was the cause of the Zune problem. Could be the cause of other problems also.
Speaking of coincidences, many people said of the Z2K9 problem that it was a merely a coincidence that it happened on New Years and on a leap year--but they were wrong. One of an item (Zune, server, etc.) crashing at New Years is probably coincidence. Several probably is systemic.
Madplayer hicked three times at about 0100 CET. I thought it might have been my RAID system I had just repaired. (There was a bad sas/sata controller.) This happened over about 20 seconds. I only use Unix/Unix-like systems and to the best of my knowledge there are no embedded MS devices in this house.
Unix/Linux, etc. handles things like this well. All time sync services like NTP, DCF-77, MSF, WWVB, GPS and the rest give fair warning. I personally are in favour of ditching 'leap seconds'. Time corrections would best be made day to day, the length of today being based on yesterday. That's better, but surely someone can think up the real solution?
BillSF
PS: Frequent updates to Java caused by US daylight saving time are pathetic.
Definitely saw some funky behaviour last night...wonderful way to start the new year... Both of our UK-based DNS servers crashed about 2-3 seconds prior to midnight GMT time. Both of these systems are running Linux kernel 2.6.21 on older Dell 1850's. Our other DNS servers (US and Australia) all run 2.6.21 on newer Dell equipment (1950/2950), but none of them crashed. Both UK systems had to be power cycled, but afterwards they came up OK.
Take out the hard drive, blow on it and then reinsert it.
It didn't crash at the rollover time, GMT or PST, but sometime between 3am and 11am new year's day.
When I turned on the monitor this morning, a black screen with cursor flashing at top left position.
First time I've ever seen this crash, a B[lack]ScreenOfDeath on Linux.
You're certainly not entirely alone. I expect more reports to appear.
why is this front page?
Didn't yet get a call from a panic stricken rep wondering why they can't access their webapp. So I'm assuming that either the voip system is also dead or everything's going grea [...Connection Reset by Peer]
The X window system on my Debian box crashed at 23:59:59 UTC --- when I returned from watching the fireworks on TV, the clock was stuck at this time. As usual it wouldn't kill, so I have just finished repairing my Firefox profile due to the ensuing hard reset.
I had one RHEL 4 server (out of a half dozen identically configured systems) crash at exactly the moment the leap second should have been inserted. The logs run up to 18:59:59 CST (UTC-06:00) and the system froze, when all my other Linux systems logged they were inserting a leap second. I have read reports of some Debian systems having a similar problem. The leap second code is probably one of the least tested areas of the Linux kernel (there have only been 8 leap seconds since Linux was started); there is probably some race condition related to stepping the clock that only some systems hit.
I have also read reports of problems with Oracle RAC (not stand-alone Oracle) crashing at the leap second.
Yes,
The latest Linux kernel 2.6.28 has a bug with regards to laptop screen brightness its sort of inverted.
This did not happen with 2.6.27.10
1:30AM up 495 days
no comments
My box crashed also. I've never had a kernel crash before. I asked at http://lists.debian.org/debian-user/2009/01/msg00006.html . One other person reported a similar problem. Someone mentioned it is a known problem with pre 2.6.21.6 kernels though I was running a more recent version.
Two Win2k boxes on my SOHO network came up with network access down/disabled. I was afraid my router had gone TU but this here Ubuntu box got online without problems. Reboots worked to get the Win2k machines online again.
FWIW
I bought this house and you know I'm boss
Ain't no h'aint gonna run me off
Apparently many Solaris systems restarted. People at NANOG are reporting this. A few banks' systems were rebooted as well: TD, Scotia, American Savings Bank, US Bank, and many more...
I saw many operating systems rebooting, even though this did not happen the last time in 2005.
Good thing I use ZFS on FreeBSD, and after I changed the loader.conf, I have a system that has stayed up for more than 2 months now, including last night.
My Roku rebooted on its own while I was watching a movie (Primer) @ ~1:30AM PST 1/1/09...
We had a report that one of our WinCE-based prototypes crashed and would not restart Wed (Dec 30).
Initially, we thought that it may have been due to an impatient user removing power during a normal shutdown, and may have occurred during a write to the flash memory drive. We can't tell exactly what happened from here (U.S.) as the prototype is in Europe.
And then again, it may just be co-incidental....
Dropped a Guru Meditation #00000025.65045338 while I was playing Giana Sisters... Is that a sign of the Apocalypse too?
Interestingly enough I saw exactly the same. Two of our production servers running fedora 8 crashed exactly at 01:01:02 GMT+1. I am beginning to suspect that this must be a Fedora 8/NTP-related bug...
My car battery died this morning. Who'd have thought that thing had a clock in it?!
My internet connection died exactly one minute before new years and stayed that way for 8 hours.....
Later, while diagnosing the situation, I discovered it was probably my big feet kicking the switch in the closet with the party favors when I went to get the toy horns, the uplink port to the firewall was partially pulled out.
At midnight GMT (well, UTC actually), we added a leap second. This seems to correspond with the time you're talking about. The most shocking thing is that this was not the first response to this story.
Could this be a coincidence or was there some 2008/2009 rollover issue going on here?
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
It might be difficult to reproduce if the system clock is NTP synched. Especially considering we had a leap second this year, where the clocks rolled from 23:59:59 to 23:59:60 to 00:00:00 GMT.
As someone else said: check your logs + try to reproduce.
--
Happy new Year
It was the leap second
uptime 15:45:56 up 9 days uname -a Linux 2.6.21-7.fc7xen
I just assumed everyone knew it was the leap second causing a problem when the author said "some 2008/2009 rollover issue going on here?"
My Fedora 9 server crashed around the same times yesterday . I found the keyboard lights flashing. I booted it at 19:51 EST. Last postfix message was at 17:59:57 Dec 31 17:59:57 bbs postfix/smtpd[1689]: messages : Dec 31 17:59:02 bbs snmpd[1871]: Received SNMP packet(s) from UDP: Dec 31 19:51:29 bbs kernel: imklog 3.20.2, log source = /proc/kmsg started.
Not 18:59, But kinda of seems odd..
Try once yourself to code conversion from "seconds since 1/1/1970 00:00:00" to any other user digestible presentation.
It's not as easy as it might seem.
All hope abandon ye who enter here.
I think it was the alcohol, but I'm not sure. I recovered some 8 hours later. why is the sun on the west and not the east?
Sounds like X crashed and you fell into vt7... where X normally is.
All you needed to do was press Ctrl+Alt+F1 (to switch to vt1), log in as root, and restart X.
Not sure how to do it on Ubuntu but on Gentoo it's "/etc/init.d/xdm restart".
Can a dummy NTP server be set up to reproduce this?
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
My Ubuntu 8.10 system froze around midnight GMT. There's nothing in the logs to indicate any problem, except that the last log entry is at Dec 31 15:55:02 (that's 5 minutes before midnight) for a cron job that runs every 5 minutes. I wasn't at home at the time, so I didn't see what happened, except that my ssh connection died.
When I got home a few hours later, the system was still powered on (the fan was running), but the screen was blank. I was able to reboot it with no problem.
It's not the same problem as on the Zune, since that hit 24 hours earlier. It *might* have something to do with the leap second, or maybe it's a problem that occurs at the end of the 366th day of the year.
I suppose I could set the clock back and run it through midnight again. I'm not sure I'll bother, but if somebody else wants to try it it would be interesting to see the results. Or maybe I'll try that with a live CD.
Unless things have changed, Im pretty certain Microsoft doesnt use posix or "unix" time...
Windows NT time is specified as the number of 100 nanosecond intervals since January 1, 1601. UNIX time is specified as the number of seconds since January 1, 1970. There are 134,774 days (or 11,644,473,600 seconds) between these dates.
Maybe things have changed since NT... but Id have my doubts.
Not that that changes your point. It doesnt. Just sayin.
I was simply giving an example. Nothing more. NT time vs. Unix time makes not much difference.
Time conversions e.g. hardware clock to OS time (where Zune30 borked) are by no mean trivial functions anymore as they were in past: DST, leap years, non-leap years and now leap seconds. Probability of coding error are pretty high and even testing not always helps since people also tends to make mistakes on date calculations.
All hope abandon ye who enter here.
I'm a sysadmin for a small data centre.
This morning, at about 7am UTC 10 machines running Fedora Core 7 all locked up. They were on a private network inaccessible to the general internet. The lockups were all complete: not even Caps Lock on a connected keyboard worked (a good sign of kernel death). Rebooting and checking the logs gave no information: the last syslog entry was sometime after midnight and was normal. Once restarted (with a hard reset) they all seem to be functioning normally. I don't know yet what caused the lockup.
It's as if a million Fedora boxes cried out in terror and were suddenly silenced...
Indeed, it's the known bug in 2.6.21 as mentioned on the debian lists: http://lists.debian.org/debian-user/2009/01/msg00006.html
So now I know, thanks slashdot!
My front door lock froze at midnight CET. Couldn't get back in after letting off fireworks. Fortunately there's a back door, but if that hadn't worked we could have tried the windows :-)
Every bloody emperor has his hand up history's skirt [Peter Hammill/VdGG]
Happy New Year! :-)
O'WONDERWe're working on it.
Converting a *nix timestamp to a date is the kind of thing that should be part of a high-school level programming class.
The code may be a little longer than people expect, but it is easy.
The only difficulty at all is added by programmers that want to optimize it to be shorter and 'cooler' (i.e. incomprehensible to anyone maintaining the code because you found a way to make it shorter by having it call itself recursively).
These are reusable functions, and are simple enough that you can check them by hand for test dates thousands of years in the future. There's absolutely no excuse to be using new and buggy implementations in every product.
I believe all the G1's (google phone) show all messages received between Dec 29 and Dec 31 as being from the year 2009.
I personally are in favour of ditching 'leap seconds'.
How many of you are there?
Trying for some flip geek cool (if such a thing could ever actually exist) merely shows ignorance
Rudolph the Red-Nosed Reindeer says hi. You might not be able to hear him, though I don't know whether to blame the fact that the joke went miles over your head, or that you're wearing your asscheeks as earmuffs.
Raytheon ASD (Y2K project in 98 & 99) did find failure modes past 2012 due to software and firmwares. Near as I can remember some were in 2008. Industrial and commercial equipment is often be used > 30 years but little was tested for date related failures until 1997. Call it random failure unless you complete post mortem exam!
This is probably why there will never be a year of the linux desktop.
Nonsense, 2009 is the year of the linux desktop!
Flash player running on my 64bit Firefox crashed.
Onda Technology Institute
So if we don't see a bunch of "Mine crashed too!" comments we should get worried?
Some kind of new bug in the kernel? ;-/
I'm quite sure "boxen" is a germanism - and in that case it would be the correct plural form.
Want to hear the voice of GOD? cat
This is the same type of human response which says, "We've observed slightly higher levels of CO2 lately, we've observed slightly higher temperatures lately, humans burn things which create CO2 ... Humans must be destroying the planet!!"
The failure of schools to teach critical thinking is what will destroy the planet.
Our local cell tower went wonky in the early evening, around when Europe would be crossing over into 2009. Since companies like Ericsson and NSN make the vast majority of basestation equipment, I wonder if their internal time base is set to either GMT, or GMT+1 or GMT+2 to match their home country. If so, it could have been a New Year's glitch.
What do I mean by "went wonky?" Around 5:00 Central and for awhile after that, we couldn't send or receive phone calls regardless of having full-strength reception. If a call connected, it acted as if it had no reception or next to no reception. Calls dropped with a "connection error."
If you're curious, this was on AT&T, and was not a 3G connection.
Program Intellivision!
The UNIX time rollover happens on 2038-01-19. So, I don't see how it would have anything to do with a problem that occurred at the end of a year.
Perhaps it is the leap second that is coursing problems for computers using NTP and other time servers
null
http://isc.sans.org/diary.html?storyid=5602
"A leap second will be added to the clock at 12/31/2008 23:59:59 UTC tonight.
Hopefully most IT folks will be otherwise occupied at that time and not focusing on their system clocks."
1601??? WTF???? Does Microsoft just love large unnecessary numbers? Or is this so people can wind back the clock in their accounting software to backdate an invoice that needed to be created 400 years ago?
OMG!
You moved your mouse. Please restart Windows for changes to take effect.
Subprime Technologies
My old (and mostly unused) Windows XP machine, for which I bought a copy of XP a few years ago, today suddenly required re-activation. The key that came with the XP copy doesn't work anymore. But probably it has nothing to do with a year-2009-like bug.
Is your cat running windows?
Wikipedia
+0 Meh
The E segment of the last digit of the year failed at exactly midnight !!
Specifically I had one Debian machine crash, and 30 Debian machines, 5 Redhat machines, and various Linux workstations not crash. The machine that crashed has a lousy motherboard, and I see that Linux attempts to reset the CMOS clock following a leap second, so I wonder whether that was what took it out.
It's related to an ANSI standard, according to MS documentation.
According to the font of dubious knowledge, 1601 is (in the Gregorian calendar) a common year starting on a Monday. It's also the most recent first year of a century prior to 2001 that meets that criteria.
I suspect someone picked that date for that reason, and/or because it was back far enough in time to allow the date/time of most most historical data and business records to be represented.
"...always new atoms but always doing the same dance, remembering what the dance was yesterday." -Richard Feynman
My old Sony-Ericsson T630 has been rebooting itself a couple of times a day over the last months (apparently known as the white screen of death). When I came home after celebrating the new year it was crashed in a rather spectacular way, though: The lights inside the keys were on, the screen was black and it responded to nothing at all. I had to remove the battery to get it going again. Since then it has only had its regular random reboots.
Against the grain
I believe it is actually 0.1 microsecond intervals.
But in all seriousness, I'm really annoyed with the ambiguity of "since January 1, 1601" as that would mean that Jan 2, 1601 00:01 and Jan 1, 1601 00:01 could be interpreted as the same time.
Just one of those little things that's going to cause another NASA disaster... /rant
Modding me -1 troll doesn't make me wrong.
How many of me? Just me, I'm the real BillSF..... If you mean how many are against the 'leap second' a quick search turned up this general article: http://www.newscientist.com/article/mg20026875.400-calls-to-scrap-the-leap-second-grow.html and there is quite a bit more. In fact the first comment to the New Scientist article almost sounds like me. (Its a coincidence, I'd not seen it before.)
If your post is for real and not a joke, you may spefic a few things, but just some comments to point in the right direction in general:
1) Holidays are Script-Kiddie time /var/ which runs full may have the chance to do you server harm (you did not specify if i panicked or just was very unresponsive)
2) Script kiddies have the potential to fill your logs
3) A
4) You may have a cron job gone wild
5) Certificates ran out -> VPNs etc?
6) you tried to change sth like logrotate and screwed (this could have a 2 year delay) heavily
7) Something else failed (Power?)
8) Something esoteric happened (e.g. do you have a watchdog which has a bad driver?)
If you are using an standard out-of-the-box linux with out-of-the-box software, i would think you are alone.
I have a first-generation iPod Nano. Yesterday (the 31st) after I had downloaded an audiobook to it, it crashed totally. Was around 1500 my time, 2000 GMT. iTunes restore wouldn't fix it until I had reformatted the drive. Never happened before.
Probably just coincidence--but kind of spooky. FWIW I have time/date set to display in the menu bar, which is not the default.
1 system running ArchLinux and 3 machines running Ubuntu 7.04 survived the leap second :)
Oday ouyay antway otay ayplay away amegay?
actually, I was just making a dumb pedantic joke about the fact that you used a singular subject with the plural form of "to be."
I administrate about 20 linux servers (debian and RHEL) and nothing happened. All are running fine. It's just you.
Could a crash be related to the power system? There could be fluctuations caused by spikes in usage, or perhaps suppliers switching sources as contracts end on the calendar year etc..
On the other hand, none of my servers suffered any ill effects... But my macbook pro was acting very strange this morning, programs were spuriously crashing (both safari and firefox, randomly crashing while browsing different sites including slashdot), turned it off for a few minutes, booted it back and everything is back to normal.
http://spamdecoy.net - free throwaway anonymous email - avoid spam!
I find it very amusing that Microsoft follow the ANSI standard, yet they cannot handle daylight savings time in any useful fashion. Unlike Unix, which has tzdata to note the different changes in DST between regions, Microsoft only stores the start of DST and the end of DST for the current year. So when governments change DST for one year they need to issue a patch to change DST, but then they need to issue a new patch after the DST ends to switch it back to normal.
This, of course, plays havoc for Exchange calendaring and any software that relies on accurate times to schedule in meetings, etc. So nice going Microsoft, you STILL can't get it right even though you have brought out Windows Server 2008 and you are soon to release Windows 7. Just how hard could it be to FIX this?
XML is like violence. If it doesn't solve the problem, use more.
could it have been something to do with the leap second that was added in at 2008-12-31 23:59:60
I don't think web servers have progressed to possessing the kind of fuzzy logic where they can go 'OMG it's almost the end of a leapyear! I better freak out'
Seeing relations like that is more geared to humans, who sometimes teach computers to look for specific examples of such coincidence, but generally I believe they couldn't come up with it themselves, at least not on the webserver level.
My g1 thinks that all SMS messages sent yesterday (12/32/08) were sent on 12-31-2009
Try once yourself to code conversion from "seconds since 1/1/1970 00:00:00" to any other user digestible presentation.
It's not as easy as it might seem.
Done:
$ perl -MHTTP::Date -e 'print time2str(1230796800)';
Thu, 01 Jan 2009 08:00:00 GMT
thegodmovie.com - watch it
bash-3.2$ cat /etc/redhat-release
Fedora release 8 (Werewolf)
bash-3.2$ date
Thu Jan 1 22:32:01 EST 2009
bash-3.2$ uptime
22:32:03 up 27 days, 8:33, 6 users, load average: 2.01, 1.94, 1.87
I do a "sudo yum update" approximately weekly.
This one surprised me. The entire ehow earnings system for its writers crashed and they still haven't fixed it. Apparently it wasn't rollover friendly. I don't remember but they've only started the system about 16-10 months ago so it's possible it has never seen a year rollover yet. Still, they're like the 39th most visited website in the US believe it or not so it's kinda odd that their entire writer's earnings system would fail.
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
A surprising number of NTP servers didn't add the leap second correctly. On the mailing list for pool.ntp.org contributors, it was reported that at just after midnight UTC that about 158 servers in the pool (about about 2000) were reporting times that were around 1000ms off. A few hours later it was only 13 that were doing that.
My own (stratum 3) NTP server got confused and declared that it couldn't determine the correct time. Some of its sources were 1000ms off from others. Given enough time, NTP will sort itself out, but I intervened manually by ditching the upstream servers that hadn't gotten it right.
If enough NTP servers were temporarily in the state that mine was in (was so unsure of itself that it wouldn't serve time to clients) then I could imagine some process that tries to sync the time and fails because ntpdate doesn't return anything useful.
Prime numbers are exactly what Alan Greenspan says they are -S. Minsky
Except that based on the responders that there were hundreds of such crashes at exactly that time (GMT) -- including people who had 10 or more servers crash at exactly that moment.
My Ubuntu 8.10 box hung last night. I had a friend check IM logs... He's on Atlantic (GMT-400) time. Weird. (08:05:33 PM) qwell: that was random (08:05:41 PM) qwell: box just...froze (08:05:47 PM) qwell: wasn't even doing anything O.o
As of 10/06/03, I hate COBOL developers.
My fedora 8 box has been running for 17 days (it had been running for a couple of months before we lost power for a week). So it isn't Fedora 8 per se. FWIW, my Fedora 10 box has been running for the same time (I need to think about upgrading the Fedora 8 box sometime).
I thought there was no way that Y2K9 would affect me, then the girlfriend asked me to check on a flight for her--and I found that United Airlines' website returns 2008 flight data if you search for flight information for Jan. 1 or Jan. 2 2009! How amateur is that?
~Ben
On the 30th... the news anchors at WBZ-TV Boston joked about getting an extra second of sleep from the leap second. Actually, since Midnight GMT is at the start of their broadcast, they got an extra second on-the-air.
Nyet!
sudo eat my shorts
Yeah right, only that 'Box' is not a German word. Try 'Kisten' instead (plural of 'Kiste'=box). This doesn't mean that Box/Boxen is not used in colloquial German in the exact way the parent post talks about, because it is.
'Boxen' on the other hand is German for what you would call 'boxing' (the sports).
A World in a Grain of Sand / Heaven in a Wild Flower,
Infinity in the Palm of your Hand / And Eternity in an Hour.
My Fedora 8 machine (kernel: 2.6.26.6-49.fc8) crashed around midnight UTC as well. Last syslog message was at 23:40:07 UTC so it may have not happened at exactly midnight; it would be unusual not to have something logged for 20 minutes. When I got to the machine, it was completely unresponsive; couldn't get it to do anything but reboot. The hardware has been very reliable and it's on a UPS.
I have seen a thread on linux.debian.user about this happening on Debian.
Before someone points it out, yes, I know that support for Fedora 8 goes away in a week or so.
"My [thing] crashed sometime between [t] and [t+1] on Dec 31, 2008 which remarkably corresponds to within at most 20 seconds of the New Year in GMT. I have been running this same [thing] non-stop for more than [long_time] and other than the occasional [maintenance_operation], it has not crashed more than 1 or 2 times in [toDays(long_time)] of cumulative uptime. Nothing other than [default_operations] were running at the time of the crash. Could this be a coincidence or was there some 2008/2009 rollover issue going on here? Has anyone (other than Zune 30GB owners) noticed similar year-end issues with their [things]?"
Why bother saying, "Besides Zune owners?"
How many of those could there be?
I mean, willing to publicly admit it?
It may just be a coincidence, but my Gmail account became "disabled" on January 1. When I tried to log in the evening of the 1st, it came back with the message "Sorry, your account has been disabled. [?]". The question mark was a link to instructions on how to re-enable it, but those instructions are apparently out of date. They say that if your account is disabled, you can enter your username and password, and you'll be presented with a CAPTCHA; if you enter that, you'll be allowed to log in. However, I was never presented with a CAPTCHA, so I can't complete the steps.
Has this Gmail outage affected anybody else? Is this a repeat of the December 6th outage, or is it just me?
davidh
I had a total of 5 Linux machines running here at home and 2 of them locked hard sometime near midnight UTC.
One is my main server running Debian Etch and had been up for several months. It had not crashed once since replacing the hardware over a year ago. We were watching a recording on MythTV when it locked-up. I walked over to my workstation, running Ubuntu Intrepid, to restart the backend and it was also locked-up hard.
Very bizarre until I looked at the logs and did the math. Both logs end just before midnight UTC.
Russ W. Knize
I'm not sure of the time, but it locked up sometime during the night. After a measly 6 months of up time.
Oddly, my ISP seems to have experienced some kind of problem starting just after midnight. I monitor my aDSL modem/router activity every 10 minutes, and discovered that between 00:00:02 (the last entry in the log) and 00:10:01 the router reports zero network traffic, and continued like that until some time late yesterday afternoon. I did the usual diagnostics tests to ensure I wasn't to blame, but all seems well on my end. Without me taking any corrective action the problem disappeared, so I'm pretty confident it was in fact my ISP.
What's odd is that I was able to gain aDSL sync, and establish a PPP session (which involves logging in), but there seemed to be no IP routing going on at all. Pings would simply vanish.
It's times like this that I wish I had a more sophisticated modem/router so that I could capture the 'out' side of the device and see what's going on.
My 2nd gen 2 gig iPod nano froze yesterday. Came back after a soft boot to discover the battery was almost completely dead; was a little less that 50% when I last put it too sleep.
Over two thousand days of consecutive uptime?
Sounds like someone doesn't patch their kernel. Not a good idea, unless the machine's sole purpose was to set an uptime record.
Don't thank God, thank a doctor!
What did the log files say ..
davecb5620@gmail.com
set the system time back a few mins before the crash occured and see if your server crashes again... otherwise it's idle speculation
I suspect changing the clock will prove nothing. It is likely that a computer in isolation will happily count onwards, leap-second or no. I would have said all standalone software these days ought to be written well enough to take care of this, but there was the Zune...
The problem probably comes when you have several computers connected together, and only some of them are putting in the leap second. Initially the computers may check their clocks, as small errors are likely, but thereafter they may be synchronized. Then, all of a sudden, some of them are giving out time stamps that seem to be a whole second ahead of the others. It might be quite reasonable to reject packets of data with impossible timestamps, as their data might be corrupt, or it may be some hacker injecting fake packets, and they haven't got the clocks quite right. So you would have to reset the clocks on lots of computers - perhaps all the computers on the internet - to run a proper test, and even then there might be a freak combination of circumstances that stops it happening a second time.
Really, the only sensible thing to do is to do what the original poster did - write to Slashdot and find out whether there were other cases. He seems to have got a lot of replies saying "well duh! there are millions of computers so one is bound to crash close to the New Year, don't'cha understand probability?" which is rather unfair as you won't know you are the only one unless you ask. He hasn't got the fifty or so replies saying they saw the same thing that might have meant a systematic problem, so it was probably random.
Instead of inserting a leap second every 1.6 years, insert a leap minute every 100 years. That way the next time it happens will be 2108 and since none of us will be alive it will be someone else's problem.
naturally such a senior member of the administration is only ever served special cat milk which is very low in lactose.
Posts, MyBio or Sig, may contain satire, sarcasm, bolded nouns be sardonic or even witty & be Church of SD
There are *lots* of people read /. and those people are *likely* to have lots of machines around which all have a possibility of crashing at some stage ?
So, at *any* random time, posting this story on the front page is likely to correspond to something falling over nearby for more than a few people ?
Man, you remember 2000, when a load of shit ( didn't really very ) broke, wouldn't it be cool if that had happened again . . .
But past that nothing.
Strange enough, the hardware (pavement) turned my firmware into software, and it took a long time to recover.
My office latop was left on by me(it has been on since 6 months) as I never carried it home. Today morning when I came to office the screen saver was frozen. I run xfce and xscreensaver, screen saver freeze is common on my fedora 9. So I tried ssh and ping from another machine but none worked. It was very hot too,(i never found it hot in the past even after a return from 4 days of vacation). I had to hard reboot. From the cron log messages it seems to have died around 7 pm on Dec. 31st. /etc/cron.hourly) /etc/cron.hourly) /etc/cron.hourly)
Dec 31 16:01:01 linlap CROND[14254]: (root) CMD (run-parts
Dec 31 17:01:01 linlap CROND[16688]: (root) CMD (run-parts
Dec 31 18:01:01 linlap CROND[19115]: (root) CMD (run-parts
Jan 2 08:43:39 linlap crond[2461]: (CRON) STARTUP (1.0)
Jan 2 08:43:41 linlap anacron[2518]: Anacron 2.3 started on 2009-01-02
Jan 2 08:43:41 linlap anacron[2518]: Will run job `cron.daily' in 65 min.
Both my desktop and my coworker's desktop had to be rebooted this morning. Both run ubuntu 8.10 and both look like they crashed around midnight GMT
My server running Fedora Core 5 had no issues. Perhaps I should wait a wee bit longer to upgrade. ;-)
After I dropped my Ipod in a vat of acid, kicked it, let my brother in law borrow it, washed it down with an ultrasonic cleaning dip it seemed to work fine, but then I plugged it into a Windows Vista laptop.... :( I had to reset the ipod after that.... Still Re-Imaging the vista box though. only 1500 security patches left to go...
Whatever leads to a server crash is not a random event. There is a finite, albeit large, number of bugs in the code that can lead to a crash. In fact, a lot of things that we consider random aren't - such as dice tossing - there are just too many variables for we to track, so we just think of them as random.
But a server crash in any platform, is a different beast. THERE is a definite reason for the crash. One bug among those millions of lines of code caused the crash. It wasn't anything random, not even pseudo-random as a dice toss.
And notice that his machine was rock stable. And he wasn't running anything extraordinary. There is a definite chance it has something to do with the leap second. You're analising it as if server crashes simply occured out of thin air.
Where is that guy who'd die defending what I had to say when I need him?
I didn't see any problems on any of the company/outbound servers I'm working with out here in Europe.
Still, assumption is the f*ckup of mother nature; never just assume or your systems could be down a week later.
Whenever I hear a disk chirping I'm yanking the alarm bell immediately for further attention; same for fans failing or anything else. ... it would cost me my neck.
I don't just assume it's going to be allright just because
I've used to have a nasty bug with my linux systems rebooting every random x day(s) about a year and a half ago; it annoyed me for a week till I downgraded the kernel and gone the problem was. It was a conflict between the new kernel, existing modules and bad libraries .. By fixing the kernel I found out the other problems as well. It's running perfect since then... I just don't assume it's going to be allright while the behavior is abnormal ..
Moral for me - always be cautious when things happen differently than otherwise because that might be the start of (many) other trouble as I had with 2 systems before...
--- I am known for the ones who want to find me on the net. Is that a privacy risk or a privilege? One might wonder..
Hadn't thought about it, but my TiVo was locked up the evening of the 31st, after several years of working just fine. Though I hadn't used the TV for a couple of days so don't know when it happened.
That's one crash every 23.5 seconds on average.
By your reasoning, you would expect 1-2 /. users to experience a random crash within 20 seconds of midnight (midnight +/- 20 seconds). Did you actually read any of the replies [from numerous other people who experienced similar odd crashes] before you posted this?
Alexander Peter Kristopeit bought his basement from his mommy for one dollar.
When was the last time you updated your packages, subby?
I've been running my computer since the beginning of the week, and it never crashed.
yum update time From they way it sounds, yum check-update
The Rapture is NOT an exit strategy.
why the fuck doesn't Microsoft Exchange use that? /had to pay $4000 for a time zone table in 2007...
Perhaps you've answered your own question?
Alexander Peter Kristopeit bought his basement from his mommy for one dollar.
That would explain why at 16:30 my Dell running Fedora 7 would not respond to keyboard or network input but the power light was lit. The most recent long entry was at Dec 31 15:59:39 and was from iptables logging NTP traffic. The old Red Hat 9 system was still playing music, unfazed.
Our Fedora Core 7 server ran seemlessly throughout the new years time change.
Meh. My karma is excellent, therefore I post with a score of 2. What can I say? You post as an AC, you post with a score of 0.
Interesting though. So they finally have sorted it out in Vista and Windows Server 2008. How many years has this taken now? And what about deployments of Windows Server 2003 and Windows XP? Is there a hotfix planned for these operating systems?
XML is like violence. If it doesn't solve the problem, use more.
In what way did he answer his own question?
XML is like violence. If it doesn't solve the problem, use more.
the part where he mentioned that, by not using that function, they got him to pay another $4k to make it work?
Alexander Peter Kristopeit bought his basement from his mommy for one dollar.
Where did he say that he paid Microsoft?
XML is like violence. If it doesn't solve the problem, use more.
An appropriately suspicious person would believe that Microsoft got their cut, naturally. :p
Alexander Peter Kristopeit bought his basement from his mommy for one dollar.
$ perl -e 'print localtime()."\n"'
This one is shorter. (hint/trick: 'localtime' should be called in scalar() context (."\n" implies that) to out user readable date; perldoc -f localtime).
All hope abandon ye who enter here.
I'm nearly certain the grandparent meant "germanism" in the sense of "from a Germanic language", as opposed to "from the modern German language".
Nonetheless, that is false — a quick etymology lookup reveals that it came into Old English from Late Latin buxis, itself from Greek pyxis from pyxos.
"Ox" [pl. "oxen"], on the other hand, comes from Old English supposedly through Proto-Germanic; similar forms are found throughout Germanic languages, c.f. Old Norse oxi, Old Frisian oxa, Middle Dutch osse, German Ochse, Gothic auhsa.
Try once yourself to code conversion from "seconds since 1/1/1970 00:00:00" to any other user digestible presentation.
It's not as easy as it might seem.
Done:
$ perl -MHTTP::Date -e 'print time2str(1230796800)';
Thu, 01 Jan 2009 08:00:00 GMT
I see your perl and raise you one coreutils:
> date -d @1230796800
Thu Jan 1 03:00:00 EST 2009
I had one Gentoo Server with a 2.6.24 kernel I can't explain why it went network dead, after it was rebooted I could check the logs and see the machine had continued to work, it just didn't talk with anyone.