Slashdot Mirror


'Daylight Savings Bugs' Loom

An anonymous reader writes "ZDNet has front page coverage of the looming daylight savings changeover, and the bugs that may crop up this year. With the extension of daylight savings time by four weeks, some engineers and programmers are warning that unprepared companies will experience serious problems in March. While companies like Microsoft have already patched their software, Gartner is warning that bugs in the travel and banking sectors could have unforeseen consequences in the coming months. ' In addition, trading applications might execute purchases and sales at the wrong time, and cell phone-billing software could charge peak rates at off-peak hours. On top of that, the effect is expected to be felt around the world: Canada and Bermuda are conforming to the U.S.-mandated change, and time zone shifts have happened in other locales as well.'" Is this just more Y2K doomsaying, or do you think there's a serious problem here?

41 of 403 comments (clear)

  1. Things you should know. by suso · · Score: 5, Informative

    http://www.bloomingtonlinux.org/wiki/DST_Time_Chan ge_Issues

    A year ago, after most of Indiana went through its first timezone change in 40+ years, we found out that it presented a few problems in Linux, I tried to post a story to Slashdot about it to warn other people in the US that they would be dealing with this problem later when the rest of the US changes to the new DST. I tried several times to post it and they were all rejected.

    Basically, you need to make sure that if you change your timezone data on your system that you restart everything, otherwise when the time does change, some programs continue to use the old timezone data and are an hour off.

    1. Re:Things you should know. by Anonymous Coward · · Score: 5, Funny

      A year ago, after most of Indiana went through its first timezone change in 40+ years, we found out that it presented a few problems in Linux, I tried to post a story to Slashdot about it to warn other people in the US that they would be dealing with this problem later when the rest of the US changes to the new DST. I tried several times to post it and they were all rejected.
      Mod parent down, informative.
    2. Re:Things you should know. by jcgam69 · · Score: 3, Informative

      We're already having serious problems with this change. Patched windows workstations show different appointment times than unpatched workstations. We're planning to roll out the windows patch, http://support.microsoft.com/kb/928388, to all computers ASAP.

    3. Re:Things you should know. by Ubergrendle · · Score: 4, Interesting

      Our datacentre has ~ 500 Solaris / HP-UX / AIX boxen, and ~ 1000 Windows servers.

      15 minute change window to apply patch, another 15 minutes to reboot successfully and come back online. Multiply 30 min x 1500 = 45,000 minutes, or 750 hours. But we only have one weekly change window, Sunday mornings from 2-6am. Assuming finite number of staff, contingency (there's always going to be some problems), etc... we started last September. We might just make the deadline.

      So yes, I think its a bit of a problem. There's also the unspoken assumption that people learned their lessons during Y2k and have sufficient date handling logic to address changes to DST...nothing hard coded in the underlying applications.

      --
      John Maynard Keynes: "When the facts change, I change my mind. What do you do?"
    4. Re:Things you should know. by shawn(at)fsu · · Score: 4, Insightful

      Multiply 30 min x 1500 = 45,000 minutes, or 750 hours
      Yeah that sounds about right if you do each machine one at a time. I should hope a datacentre is a little more efficient than that.

      --
      500 dollar reward for tip(s) leading to the arrest of the person(s) who stole my sig.
    5. Re:Things you should know. by vadim_t · · Score: 3, Insightful

      Why do your programs use the local timezone, anyway?

      Programs should handle and store dates in UTC, and convert to the local timezone only for display.

    6. Re:Things you should know. by Nimey · · Score: 4, Funny

      That's a damned lie and demeaning to Windows to single it out like that.

      MS-DOS is the same way. Apple ProDOS too.

      --
      Hail Eris, full of mischief...

      E pluribus sanguinem
    7. Re:Things you should know. by Marillion · · Score: 3, Interesting

      Most programs that use the standard C library do use UTC and just don't realise it. The most important thing to realise about daylight savings time is that Time isn't changing. The sun still rise the after DST as before DST (astronomical adjustments due to Earth's heliocentric orbit not withstanding). But, how we read the clock is changing.

      I heard one company just say, "we're going to just change the clock on the computer." This makes me cringe.

      All file time timestamps on all versions of UNIX and versions of Windows derived from NT store times as Julian seconds since 1970-01-01 00:00:00 GMT. Changing the system clock means that the times will be stored wrong even though they display properly in the local timezone.

      Other places to watch out for are applications that manage their own timezone data. Java is a prime example. Major database vendors would be another.

      --
      This is a boring sig
    8. Re:Things you should know. by pe1chl · · Score: 3, Interesting

      When you are so careful about file timestamps, you should know that Windows does not translate them correctly anyway.
      Windows has no knowledge about timezone history. It translates the UTC time to local time using its current time-offset, which depends on the current DST status only.
      So, when you now look at a file timestamp (in the GUI) that you created last summer, you will find that its time is one hour off.

      Even when Microsoft would finally fix this (they consider it a feature rather than a bug), they would probably not fix the historic aspect.
      I.e. now that the beginning of DST shifts one month, and you would look at a file created last year in that one month window, it would probably still be off.

      Timezone handling in Windows just sucks. It does not have to be that bad, it works fine in Linux (including historic changes). Microsoft just has decided to make a bad implementation and then never fix it, in the name of backward compatability.

    9. Re:Things you should know. by danielblair · · Score: 3, Informative

      Amen!

      Seagate and Peregrine/HP have some really nice, robust, Data Center Management software for managing 10s, 100s, or even thousands of computers and the software on them... you can push, say, SP2 to all 5000 machines, and it's like a few clicks to do it.. and it takes care of everything! Then you check the error report, etc. to make sure that none of the known machines encountered an error (or were turned off at the time), and then you address them individually...

      With the # of servers that you are talking about, I wouldn't even think of doing it manually.. unless you get paid by the hour and literately have nothing else to do for the next few months (or year - as per your statement about almost being done after having started the update(s) a number of months back..) It would be well worth your investment to install something like this so that you could push a patch, or a simple update (like windows malicious software removal tool monthly update, etc.) to selected machines, or all machines, and do it from a single console..

      To each his own though..

      --
      -- Daniel R. Blair Senior Software Architect/Unix & FreeBSD Guru/DJ w: http://unixcoders.org t: @freebsd_hacker
    10. Re:Things you should know. by tushar · · Score: 4, Insightful

      Why do your programs use the local timezone, anyway? Programs should handle and store dates in UTC, and convert to the local timezone only for display.
      Saving the time in UTC does not get rid of all problems. Programs that store the data for future events will still run into issues. An event that was stored in the database before the system was patched will be off by one hour if the time falls within the dates that have the daylight savings rule changed.
    11. Re:Things you should know. by Threni · · Score: 4, Funny

      > 15 minute change window to apply patch, another 15 minutes to reboot successfully and come back
      > online. Multiply 30 min x 1500 = 45,000 minutes, or 750 hours

      Yeah, someone needs to tell my local cinema about that. They only show films for a few weeks, but with 1000 people in each showing, and the film lasting 90 minutes, that's 90,000 minutes, or nearly 9 weeks! There's going to be a lot of disappointed people...

  2. y2k = media working for once by TinBromide · · Score: 5, Insightful

    The reason y2k was such a letdown was because everybody took heed of the media hype and patched their stuff. If there had been no hype before, there might have been the problems that the hype was warning about. (or not, sensationalism is sensationalism)

    Its like you're driving along and there's a huge backup for miles because of an accident in the middle of the road after a bend. Now this huge backup may have slowed you down and made you aware that there was a problem. If it was just you and the wreck, you may have plowed into it if you weren't paying attention.

    Same thing with this hype. We should tolerate the hype because the heightened coverage will get bosses talking to programmers about fixing the software, and it'll turn out to be nothing.

    --
    Is it sad that I am more likely to recognize you and your posts by your sig than your name or UID?
    1. Re:y2k = media working for once by DrXym · · Score: 4, Informative
      I think y2k was a let down because it was fairly straightforward to fix software and tell if it was going to get bitten or not. Basically anything that stored dates as two digits had to be fixed. The bigger problem will come in 2037 when lots of clunking software with no source code wraps around.

      The DST thing is pretty evil too because it's usually up to runtime stacks like Java and CRT to decide on the timezone and time. If they give you the wrong time you're screwed. For the most part you might be okay if everything resolves down to some registry entries or timezone data files but that isn't always the case. There are functions such as Microsoft's _tzset() which are HARDCODED to a particular behaviour and apps that link to the CRT or have their own DLLs will be broken unless you recompile them.

    2. Re:y2k = media working for once by h2g2bob · · Score: 3, Funny

      Mod parent up!

      Changing daylight savings time or y2k will be childsplay compared to the Year 2038 32-bit time_t overflow. That is a really big problem with no easy fix. 32-bit times/dates are in everything from VCRs and microwaves to servers and desktops. 2038 will be everything that Y2K wasn't.

      In principle, Linux and friends can fix this by redefining time_t to 64-bit - but lots of communication protocols and even file formats like tar use 32-bit dates. Admittedly we have 30 years to fix it, but we will need all 30.

  3. rates? by cascadingstylesheet · · Score: 4, Funny


    and cell phone-billing software could charge peak rates at off-peak hours


    Aiyeeee!!!!!!!!!!!!!

    1. Re:rates? by johnlcallaway · · Score: 4, Insightful

      And could charge off-peak rates for peak hours. The number of hours for peak and non-peak will remain the same, only the start times will change.

      Everyone wants a credit when they are over billed, but no offers to give money back if they are under billed.

      --
      I rarely read replies, it's my opinion and if you thought about your opinion a little more, I'm OK with that.
    2. Re:rates? by Qzukk · · Score: 3, Insightful

      Except that peak and off-peak mean something, namely volume of calls. So as a matter of fact, no, they are not "equally likely to charge you off-peak rates during peak periods" because you're more likely to be making a peak period call than an off-peak call.

      --
      If I have been able to see further than others, it is because I bought a pair of binoculars.
  4. A site that will give you USEFUL Info..... by 8127972 · · Score: 4, Informative

    ... On how to deal with this is below:

    http://www.reganfamily.ca/dst/

    This is likely more useful than the original article. It has resources for everything from Blackberries to UNIX.

    --
    This is my opinion. To make sure you don't steal it, it's covered by the DMCA.
  5. Ahem, Not Exactly by Nom+du+Keyboard · · Score: 4, Informative
    While companies like Microsoft have already patched their software,

    Ahem, not exactly. No patch for the perfectly good Exchange 5.5 server we're using with Outlook 2000. Suddenly we have to update to the latest Exchange and Outlook 2003 on every d@mn desktop. And I'm in Arizona were we don't even have daylight savings time!!!

    --
    "It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
    1. Re:Ahem, Not Exactly by FormulaTroll · · Score: 3, Informative

      Not true. For Outlook 2000, patch the underlying OS and things will be just fine. The same applies to Exchange 5.5, patch the OS and you'll be fine as far as basic Exchange services go. It's Exchange 5.5 CDO applications, like OWA, that don't have a publicly available patch. The worst case scenerio for most people is your appointments show up an hour late during the extended DST period, and Microsoft has released tools to fix the appointments themselves - and the tool works on Exchange 5.5. An upgrade is certainly not a requirement.

  6. Cool by Anon-Admin · · Score: 4, Funny

    And I thought that the year 1906 would pass with out any issues.

  7. Doomsaying? by eln · · Score: 3, Insightful

    I don't see anyone saying planes are going to fall out of the sky or anything like that. Trades could be executed at the wrong time, costing people money. Cell phones could charge peak rates at the wrong times, costing people money. These could very easily happen if these companies were not on the ball about getting this patched early. Luckily, most operating systems required a simple patch (sometimes a reboot) to fix this, and that's about it. The extensive code fixes that the Y2K bug required simply aren't necessary here.

    However, because of the perceived simplicity of the fix, there is a real possibility that some companies waited too late to address the issue and may not make it in time. This may cause minor glitches, but it's not like the nukes are going to start flying.

    As for Y2K, obviously the people who were stockpiling ammunition and moving to the mountains were nuts, but there were real problems that could have occurred that did not because of the countless hours that were put in to fix the issues. It drives me crazy that after we spent millions of dollars and countless man hours fixing buggy code for Y2K, people look back and see that nothing happened and think all that money was a waste. If all that effort had not been expended, more computer systems would have had problems, and so the money was definitely not wasted. During Y2K, there were scattered reports of various computer systems crashing. It is likely there would have been many more such reports had we not taken the steps we did to address the issue.

  8. Already has ramifications by michaelmalak · · Score: 4, Interesting

    Newer JDK's already have the new time zone rules encoded, and this can cause subtle bugs to suddenly surface. It turns out that Date.equals() does a deep object compare, including the time zone rules (not just which time zone you're in, but the rules regarding when daylight savings starts and ends). Thus if you have multiple JVMs involved, such as one on a database server and one on an application server running slightly different JVM revs (e.g. 1.4.2_08 vs. 1.4.2_11), then naive date comparisons (using equals() instead of equality on getTime()) can fail.

  9. Get rid of daylight saving altogether by Viol8 · · Score: 4, Insightful

    It serves no useful purpose any longer in what is almost a 24 hour society. What difference does it make to the vast majority of people what time the sun rises and sets anymore? All it does is add a small extra layer of confusion and complexity thats no longer necessary? If people really don't want to get up when its dark then go to work an hour later and leave an hour later. With flexitime its really not an issue anymore.

  10. Old OSes and Old JREs are the biggest concern by poopie · · Score: 4, Insightful

    Think of how many companies have old systems that just continue to run forever. Most OS vendors drop OS patch support after about 5 years.

    Okay, so all system processes should use UTC. We all know that. Users don't set their watches to UTC though.

    Want a DST patch for Solaris 8? RHAS 2.1? Windows NT? You're going to have to shuffle and maybe you'll need to update the timezone files with 'zic' yourself. Have hundreds or thousands of these machines. Sucks to be you.

    Oh, and the big killer is that Java has timezone rules embedded in it. That's right. Java VIRTUAL MACHINE. Java tracks timezones and DST changes INDEPENDENT of the OS since Java wants to be it's own OS.

    So, if your company standardized on j2ee when you moved off the legacy systems for y2k, I'll almost bet you that the OS those java apps are running on won't have DST patches from the vendor, and your apps could have multiple JVMs that contain the wrong DST rules. You'll need to fix both of those if your java apps have anything to do with timezones and if you care about the times displayed.

    I'd really like to get a list of everyone who voted for the 2005 dst timezone changes and start a movement to make them take responsibility for the huge business cost of their stupid legislation. It has to be 100X the cost of what they expected the changes to save...

  11. A round of applause for the tz guys by Whiteout · · Score: 3, Informative

    The tz database http://www.twinsun.com/tz/tz-link.htm underlies time zone handling for the GNU C Library, FreeBSD, NetBSD, OpenBSD, Mac OS X, Solaris and many more, and is kept current by a dedicated team of (mostly?) volunteers. For time nerds, the historical comments in the plain text files of the tz ftp distribution (ftp://elsie.nci.nih.gov/pub/tzdata2007b.tar.gz) are required reading.

    If you're a Firefox person, FoxClocks (see my URL above) puts nice little world clocks on your statusbar. And yes, it uses tz too. Thanks guys. Andy

  12. Worse than Y2K because of Java by wsanders · · Score: 3, Funny

    This is worse than Y2K because Java needs to be patched, and JVMs proliferate on hosts like cockroaches. Older JVMs cannot be patched.

    There are nearly 50 java instances on some of our hosts. The filthy little bastards hide everywhere.

    Fortunately the fix can be automated and is very fast to install.

    Using java's extensive built-in patch management and version management capabilities, of course.

    --
    Give a man a fish and you have fed him for today. Teach a man to fish, and he'll say "WHERE'S MY FISH, YOU IDIOT?"
  13. Re:Not such a big deal by GogglesPisano · · Score: 4, Informative

    We have literally hundreds of servers running Windows 2000, and this DST patch was a major headache. As the parent noted, Microsoft did not include Win2K in their publicly released update.

    There is a freeware utility to apply the DST patch on Win2K machines here (as a bonus, it also supports WinNT).

    Note that you may also need to update the Java JRE/JDK.

  14. No big deal if you don't update... by thewiz · · Score: 3, Insightful

    You'll catch up to the rest of us in three weeks!

    --
    If "disco" means "I learn" in Latin, does "discothèque" mean "I learn technology"?
  15. Apple by metroplex · · Score: 3, Informative

    Apple just pushed an update through Software Update that fixes potential daylight saving time problems. You can grab it here if you use Tiger, or here if you still use Panther. It also released a similar update for Java. here is the Tiger version and here is the Panther one.

    --
    "Words of wisdom: drop that zero and get with the hero" -- Vanilla Ice
  16. daylight savings time is stupid by circletimessquare · · Score: 3, Interesting

    i recognize the interest in giving people more daylight hours for harvest/ farming purposes... and how that's still nice in a service/ industrial work setting to have barbeque time after work

    but why does that mean that time has to be shifted twice every year? why not just shift time by an hour once, just one time, and be done with the nonsense forever? why is it necessary to go back to "real" time in the winter?

    heck, sometimes i think we should redefine 6 am as 3 am. then everyone wakes up and goes to work in the middle of the "night", and, after work in the summer, you have daylight until midnight!

    we're dealing with an abstract concept here. we can do whatever we want with time. we don't have to abide by some weird need to swithc back to "real" time in the winter. just shift it once in favor of farmers/ after work barbequers and be gone with it. it's just so stupid and pointless and a waste of effort to constantly shift back and forth

    --
    intellectual property law is philosophically incoherent. it is your moral duty to ignore it or sabotage it
    1. Re:daylight savings time is stupid by curunir · · Score: 3, Interesting

      FYI...This latest "fiddling" with DST is designed to figure out just how much energy DST actually saves (if any). By adjusting the date when DST goes into effect, and measuring the difference in energy consumption compared to previous years, we should get a good idea of whether DST is providing any energy savings and whether that savings is worth the trade-offs (there are quantifiable costs associated with DST...things like increased auto accident rates around the changeover dates).

      There's also some notion that in the modern world DST actually increases the country's energy consumption. DST was originally conjured up in a world where the bulk of the energy consumption came from lighting. In that world, giving people daylight at certain hours of the day reduced their need for artificial light. In our modern world, however, things aren't that simple. For one, we have lighting that uses less energy than before. But the biggest difference is that we now have air conditioners, something that uses significantly more energy than our modern lighting. In the modern world, by ensuring that there is natural light when people get home from work, we increase the likelyhood that they will need to use their air conditioners.

      So, we really have no idea whether DST actually serves its purpose anymore or if it's merely an unecessary inconvenience for any modernized country. This year's change in the time that DST goes into effect will give us a good indication of whether we can eliminate DST entirely.

      --
      "Don't blame me, I voted for Kodos!"
  17. Re:no need o worry by lucabrasi999 · · Score: 4, Funny
    daylight savings times and zones change constantly in australia and everything is fine here

    Everything is fine in Australia? Remember folks, this announcement is coming from the country that gave us The Wiggles.

  18. Verify your Linux box is correct. by OrangeTide · · Score: 3, Informative

    % zdump -v /etc/localtime| grep 2007 /etc/localtime Sun Mar 11 09:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 PST isdst=0 gmtoff=-28800 /etc/localtime Sun Mar 11 10:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 PDT isdst=1 gmtoff=-25200 /etc/localtime Sun Nov 4 08:59:59 2007 UTC = Sun Nov 4 01:59:59 2007 PDT isdst=1 gmtoff=-25200 /etc/localtime Sun Nov 4 09:00:00 2007 UTC = Sun Nov 4 01:00:00 2007 PST isdst=0 gmtoff=-28800

    notice that the isdst changes from 0 to 1 on March 11. This means I have the correct zoneinfo file in my system. /etc/localtime is a symbolic link to the default timezone for your machine. (users can run their own timezone with the TZ environment variable).

    % ls -l /etc/localtime
    lrwxrwxrwx 1 root root 30 2006-09-24 21:50 /etc/localtime -> /usr/share/zoneinfo/US/Pacific

    PS - likely the steps to check this on FreeBSD are similar. Post your experiences.

    --
    “Common sense is not so common.” — Voltaire
  19. Herding cats by Dachannien · · Score: 4, Funny

    Let's switch to the metric system while we're at it.

  20. Re:Y2K was an oddity and mis-explained by LMacG · · Score: 4, Interesting

    "In no way were dates routinely stored as two byte characters (99 being the max) when 1 byte would get you to 255 easily." Wrong. In EVERY WAY dates were routinely stored as three sets of two byte characters.

    What you are completely ignoring is that the vast majority of the code that had to be examined and patched was written in COBOL. COBOL that store dates as a string of six digits. Digits that were stored in many cases as EBCDIC characters, not hexadecimal integer values. And just to make it fun, in some cases the source code was not available.

    "[A]nyone that created a four digit date by String Concat: "19" + String(date) would " probably not have been born yet when the programs that needed to be fixed were written. It wasn't the programs that were written in the 1990's that had to be dealt with, it was the ones written in the 1960's. And if you don't believe there were any of those in use, then I suggest you have no idea what's really happening at your bank. Or in the US air traffic control system, for that matter.

    --
    Slightly disreputable, albeit gregarious
  21. Windows and CMOS clock by dokebi · · Score: 3, Insightful

    Why does Windows not run UTC on the CMOS clock? Doing so would solve all of this "The computer has changed the clock" twice a year. The clock wouldn't be changed, just synced every now and then, but the displayed time would automatically be adjusted. POSIX and MacOS does this correctly, and 99.99% of Mac users don't even realize their CMOS clock runs UTC. Changing Daylight Time would be updating a single file, even in a closed OS like windows.

    I've heard all sorts of dumb reasons against running UTC on the CMOS, like "who cares about UTC, My time is local" and "why should I keep two different times on my computer".
    But, the OS will hide the UTC from you, and besides, when was the last time you used the BIOS time as your clock?

    Forcing UTC on the CMOS clock is surprising since WindowsNT has used UTC for all their internal time tracking for some time. But they *calculate* it from local time, which changes twice a year, _even though_ Windows uses NTP time servers. Doh. It's gotta be *the* dumbest backward compatibility "feature". See here: http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html

    --
    In Soviet Russia, articles before post read *you*!
  22. How to test if your linux machine is ready by Se7enLC · · Score: 5, Informative


    > date --date="Mar 10 15:00:00 UTC 2007"
    Sat Mar 10 10:00:00 EST 2007
    > date --date="Mar 11 15:00:00 UTC 2007"
    Sun Mar 11 11:00:00 EDT 2007

    This won't set your clock or anything, it just does the timezone conversion from UTC and displays the results according to the local timezone you have selected.

    1. Re:How to test if your linux machine is ready by srvivn21 · · Score: 4, Informative


      > date --date="Mar 10 15:00:00 UTC 2007"
      Sat Mar 10 10:00:00 EST 2007
      > date --date="Mar 11 15:00:00 UTC 2007"
      Sun Mar 11 11:00:00 EDT 2007

      This won't set your clock or anything, it just does the timezone conversion from UTC and displays the results according to the local timezone you have selected. Or just run zdump -v /etc/localtime | grep 2007" and make sure it says "Mar 11" instead of "Apr 1".

      updated> zdump -v /etc/localtime | grep 2007
      /etc/localtime Sun Mar 11 10:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 AKST isdst=0 gmtoff=-32400
      /etc/localtime Sun Mar 11 11:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 AKDT isdst=1 gmtoff=-28800
      /etc/localtime Sun Nov 4 09:59:59 2007 UTC = Sun Nov 4 01:59:59 2007 AKDT isdst=1 gmtoff=-28800
      /etc/localtime Sun Nov 4 10:00:00 2007 UTC = Sun Nov 4 01:00:00 2007 AKST isdst=0 gmtoff=-32400

      needswork> zdump -v /etc/localtime | grep 2007
      /etc/localtime Sun Apr 1 10:59:59 2007 UTC = Sun Apr 1 01:59:59 2007 AKST isdst=0 gmtoff=-32400
      /etc/localtime Sun Apr 1 11:00:00 2007 UTC = Sun Apr 1 03:00:00 2007 AKDT isdst=1 gmtoff=-28800
      /etc/localtime Sun Oct 28 09:59:59 2007 UTC = Sun Oct 28 01:59:59 2007 AKDT isdst=1 gmtoff=-28800
      /etc/localtime Sun Oct 28 10:00:00 2007 UTC = Sun Oct 28 01:00:00 2007 AKST isdst=0 gmtoff=-32400
  23. Re:Mod Parent Up by Stanistani · · Score: 3, Interesting

    Where I worked, we had a lot of old but reliable systems. We tested our mainframes, minicomputers, PCs, and network systems - they failed in varying and catastrophic ways.

    We spent millions of dollars for all new systems, and thousands of hours in planning, procurement, implementation, and testing. We literally pushed all the boxes on the datacenter floor to the wall and built anew.

    It was a horrendous chore, and I didn't get to spend New Year's with my family.

    Perhaps we should have let you all freeze in the dark.