Slashdot Mirror
The Long Shadow of Y2K
Hugh Pickens writes "It seems like it was only yesterday when the entire world was abuzz about the looming catastrophe of Y2K that had us both panicked and prepared. Ten Years ago there were doomsday predictions that planes would fall from the sky and electric grids would go black, forced into obsolescence by the inability of computers to recognize the precise moment that 1999 rolled over to 2000 and for many it was a time to feel anxious about getting money out of bank accounts and fuel out of gas pumps. "Nobody really understood what impact it was going to have, when that clock rolled over and those digits went to zero. There was a lot of speculation they would reset back to 1900," says IT professional. Jake DeWoskin. The Y2K bug may have been IT's moment in the sun, but it also cast a long shadow in its wake as the years and months leading up to it were a hard slog for virtually everyone in IT, from project managers to programmers."
"'People were scared for their jobs and their reputations," says CIO Dick Hudson, Staffers feared that if they were fired for failing to remedy Y2K problems, the stigma would prevent them from ever getting a job in IT again. "Then there was the fear that someone like Computerworld would report it, and it would be on the front page," Hudson adds. Although IT executives across the globe were confident that they had the problem licked, a nagging fear followed them right up until New Year's Eve. While most people were out celebrating the turn of the century, IT executives and their staffs were either monitoring events in the office or standing by at home. Afterwards came the recriminations and backlash as an estimated $100 billion was spent nationwide for problems that turned out to be minimal. Others says the nonevent was evidence the Y2K effort was done right. "It was a no-win situation," says Paul Ingevaldson. "People said, 'You IT guys made this big deal about Y2K, and it was no big deal. You oversold this. You cried wolf.' ""
We see exactly the same reaction today about all the issues that face us (whether personal, local, national or world-wide). The considered, thoughtful and measured responses that would (given a chance) produce equitable solutions with a minimum of fuss get washed away by the ignorant but vocal commentators in the media. These people don't care about the problem, or finding a solution. All they want is the cameras pointing in their direction.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
In the couple of years leading up to Y2K, I saw my company pour millions into updating any outdated infrastructure. Since were all techies, I'm betting that we all have similar stories. All the negativity aside, is it also possible that we moved ourselves ahead with this non-existent catastrophe? I mean shoot, I know I at least got a new laptop out of the deal ;^)
"Before God we are all equally wise - and equally foolish"
Albert Einstein
http://xkcd.com/607/
"There are no facts, only interpretations." --Friedrich Nietzsche.
A great many computer systems used two digit dates, and would treat '00' as a date in the past. Changing this fundamental fact would take an awful lot of work; not changing it would mean that all these computer systems break on Jan 1st 2000.
Allot of work was done, and most all important computer systems didn't suffer from any serious problems.
What is being oversold?
I suppose there were 'cowboy' consultants exploiting the problem by offering to come in and look at your recently acquired IT infrastructure, charging huge amounts for a simple thumbs up. That doesn't undermine the severity of the problem though.
I think 38 years should be long enough for us to sort things out before Y2K.
A pizza of radius z and thickness a has a volume of pi z z a
It was real, but hyped. None of us seriously expected 747s to invert on crossing the International Date Line, as some more fevered commentators speculated, nor did we expect nuclear power stations to destabilize.
However, we knew that all our systems had to interact correctly for the business to deliver correctly. I was working as a contractor for a major airline, and we knew that lots of our most fundamental systems had been written in the 60's and 70's. They HAD to be checked, and HAD to be tested through the full extent of the workflow.
Moreover, it was always journalist bullshit that it was all going to happen at the stroke of midnight. There were plenty of opportunities for problems to occur at other times. A major food and clothing retailer started rejecting shipments of canned food in September 1999 because the dates on the cans said the Sell-By date was 100 years ago. This really happened.
And yet stuff DID happen at the stroke of midnight - and that news got suppressed because it was embarrassing, and anyway most of the incidents were minor - we had successfully fixed everything major.
"Cock Up Your Beaver" does not mean what you think. This sig is intended to clog filters and annoy do-gooders
See this:
http://it.slashdot.org/article.pl?sid=07/02/25/2038217
When F22 fighter planes have stupid bugs that cause problems on crossing the international date line, I can't really have that much confidence that planes won't be falling out of sky on 2038 ;).
Why go as far as twitter? Slashdot fell over a year or so ago because message ids were stored in a 24-bit integer, which overflowed. Who ever imagined when Slashdot was created that it would come close to 17 million posts? 2^16 probably seemed like a lot, so wasting another byte per post probably seemed enough to give some headroom. A decade later, it turned out not to be.
If you were writing software in the '70s, every byte mattered. A lot of mainframes around then used 6-bit bytes with binary coded decimals, so you'd be using 12 bits for a two digit date or 24 for a four digit one. Software was much cheaper than hardware, so saving 50% of the storage requirement and requiring the software to be rewritten in 30 years would have been a huge saving overall, especially on machines where 1KB took up an entire rack and cost thousands of dollars. And, because the software worked, people kept using it.
Architectures like IBM's OS/360 and Burroughs Large Systems have maintained backwards ABI compatibility since the '70s, so there was no reason to touch the code. The space saving went from saving them the need to spend $10K on an extra memory module to saving them a tiny fraction of a percent of the machine's total capacity, but no one cared, because the code still worked. Then 1999 rolled around and people found that their system would break next year. The old code was lost, or written by people who had long since died or retired, so in a lot of cases needed completely rewriting. Fortunately, programming languages have advanced a lot since those days (unless you had a Lisp machine or a Xerox Alto back then, in which case they've bone backwards to a painful degree) and so it didn't take much programmer time to rewrite them, although migrating the data and testing took a lot of effort.
I am TheRaven on Soylent News
People wanted to fear it.
I was at Wal-Mart getting an oil change (for the record never go there for that) in 1999 while in the waiting area a conversation was struck up between myself and another person waiting on a vehicle. It came out that I worked for an ISP and had done all kinds of other computer/networking work. The person wanted to know my thoughts on Y2K.
I answered "I think there's going to be a few hiccups and glitches. I don't think they're going to be all that big, we've done a pretty good job of preparing, and many things may fail over to a wrong date, but will continue to work anyways. All in all whatever problems come of it a majority will be fixed in the first couple of days and a few may take longer, but I don't think there will be much impact."
The person became visibly annoyed at my answer. We stopped talking very quickly after that. I had many other conversations with people along these lines, a couple of them even sited Art Bell and how his show was talking about the doom and gloom to come. I listened to Art Bell. He must have made a fortune selling crank radios, flash lights, and other survival gear in preparations for Y2K, not to mention his business model relies on crazies and they were coming out of the woodwork for this.
I was working the night shift during the roll over. I wasn't worried about our equipment failing. I went to work armed, I was worried about crazies who might decide our company was going to be the cause of the downfall of civilization.
The only thing I noticed was the IRC chat room had some sort of a reset, 90% of the people connected dropped off at midnight, that was actually the event that caused me to check the clock. Us other 10% stayed connected, I'm guessing it was one of dial up routers dropping everyone.
People were practically begging for the doom and gloom scenario. It gave me insight into the human condition, I'll say that for sure.
The preceding post was not a Slashvertisement.
The news must be slow to report on an event that didn't happen 10 years ago.
Actually I just fixed a 2010 bug, where someone did exactly as you suggested. They created a system back in 2000 or 2001 where the last digit of the year was used as a key. Someone realized there might be a problem back in early November...
Basically it was management decisions to not spend the money on the computer storage.
Sometimes really stupid ones. I know a programmer who was disciplined because management decided that, statistically, the "skip a leap year if the year is divisible by 100" correction for date change was important enough to include but not the "unless the year is also divisible by 400" rule. Therefore he was somehow "wasting storage" by removing the first correction to fix things until the year 2100, even though the program got smaller.
There were quite a few systems with BIOS/CMOS clocks, OSes, etc that were going to screw up one way or the other without being replaced or upgraded. Said screwups, with rare exceptions, might seem disasters to managers who treat any unexpected problem as one, but not by the general population; still fixing them in advance was probably cheaper than after the fact.
The Y2K problem is only one expression of the common problem of a data value occurring greater in magnitude than what that given data type can store or represent. This still can occur and presents as much of a problem for critical computer systems. I've found a bug that would have suddenly adjusted the suspension of police cruisers or other models of a vehicle very poorly if they exceeded 128.5 MPH before it ended up in a production vehicle. That did not stop me from wincing back in 1999 at radio commercials from a used car dealer trying to scare people into buying his "Y2K-verified" products, lest they perhaps be left stranded if their car suddenly died on New Years day.
I wonder what happened to those kooks who sold their homes, and bought farms or that stocked up with 2 years worth of spegheti-Os, etc.
I was an analyst for Gartner in the years leading up to Y2K. As usual, the real story is nothing like what is reported in the press.
First of all, the systems failed not because the date itself rolled over to January 1, 2000, but when systems attempted to do a calculation that spanned both centuries and thus did the math wrong. In 1970, 30-year mortgages started having glitches because they calculated into the year 00, and started calculating interest based on 99 years’ worth of time. Called, the “Time Horizon to Failure,” these types of failures increased on a log scale in the 90s as we approached 1/1/2000. Few if any systems based on microcontrollers (say, elevators) care at all about the date, much less that the year is 2 digits.
The bug was very real. There was literally billions of lines of mainframe code written in the 60s, 70s, 80s and even 90s that used two digits for dates. There was actually a 1970 bug, where some systems used only one digit for the date in the 60s. Remember we are talking 80 byte punch cards and memory that was hundreds of dollars per byte. The fixes weren’t hard but there was a LOT of code to slog through, much of which was not documented and in some cases they didn’t even have the source.
Why weren’t there more visible problems? in the early and mid 90s, all the IT departments alerted their managers to the problem, showed where in the code it needed to be fixed, and what the consequences were. But few managers acted, because nobody believed the “hype” and budgets were needed for more pressing initiatives.
Enter the Wall Street Journal, who wrote an article, I think it was in late 1996 or 1997, that said to company executives that their Errors and Omissions insurance would not cover them if their company experienced Y2K failures because the bug was widely publicized and the threat was well known. This means that the executives were personally liable (e.g. they could lose their houses) for Y2K failures that happened in their companies.
The next day, thousands of companies started Y2K projects, and fixed the issues. So, no serious bugs were reported, and those who labeled it hype had all the evidence they needed to support their theory. But it took a legal threat for managers to act.
I hate being bipolar; it's awesome!
In 99, a friend of mine was doing a live migration from a mainframe software that was too expensive to fix for Y2K. This was a critical billing system for the business so they had to keep the mainframe working until the migration to the new software was complete. The complex project was scheduled to be over on Dec 15.
What they did not expect was that the end-of-month calculation routine in the old software used a "clever" trick: add one month, remove one day...
So on Dec 1st the software went down in flames (and my friend did not get his Y2K bonus).
They called it the 12/99 bug.
lucm, indeed.
On Y2K day, the website calendar of the US Naval Observatory (our observational time keeping experts; National Bureau of Standards count them, these guys tell us when they start and stop and need readjusting) read JAN 1, 19001.
See if there's still a screen capture of that around, I know several circulated back then. Then if anyone challenges you, simply show it to them and say "We didn't oversell. We got it right. They didn't."
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
Ecstasy
The attack of 9/11/2001 took out the WTC and other buildings near ground zero. This was the heart of the financial district and the IT base of many firms.
In the hours following the attack, the offsite backup sites for many of those firms seamlessly took over. Nobody noticed that.
I firmly believe that without Y2K remediations, 911 would have been a big IT disaster too.
Agony
At the successful conclusion of Y2K remediation efforts, the upper and middle level managements treated themselves to celebrations at luxury resorts. Meanwhile, many IT grunts who put in all the extra hours got nothing more than pink slips. In most cases, the companies didn't even offer to buy them a beer as thanks for their long hours.
It was the most ungracious treatment of labor I ever witnessed. Compare it to calling Viet Nam vets baby killers.
I thought it was known as "banking".
Huh? You do understand that even today some back-end systems are run on very old machines with very old programs. The reason they don't get updated is that it costs money to update. Unlike the average /. geek, businesses don't replace their systems whenever something new appears. If it works, there has to be a reason to update. The Y2K bug was such a reason. But like other areas, even when experts warn of something doesn't mean management is listening especially if the problem isn't happening today or tomorrow.
This phenomenon is not relegated to just IT. You remember that event called Hurricane Katrina? The Army Corps of Engineers warned that the levees were not enough. Their warnings started almost 30 years ago. Every year the asked for money to address the levees; every they were promised money but not actually given any. Then the levees broke and the government leaders wanted to blame the Corps. The Corps had documentation that they warned the government well in advance.
Well, there's spam egg sausage and spam, that's not got much spam in it.
None of us seriously expected 747s to invert on crossing the International Date Line, as some more fevered commentators speculated, nor did we expect nuclear power stations to destabilize.
Software Bug Halts F-22 Flight
Posted by kdawson on Sunday February 25 2007, @06:35PM
it.slashdot.org
On Feb. 11, twelve Raptors flying from Hawaii to Japan were forced to turn back when a software glitch crashed all of the F-22s' on-board computers as they crossed the international date line. The delay in arrival in Japan was previously reported, with rumors of problems with the software. CNN television, however, this morning reported that every fighter completely lost all navigation and communications when they crossed the international date line. They reportedly had to turn around and follow their tankers by visual contact back to Hawaii...
You do understand that even today some back-end systems are run on very old machines with very old programs. The reason they don't get updated is that it costs money to update.
I worked for a Very Big Corporation back around the turn of the century. We had quite a bit of old equipment back then that was at risk for the date rollover. Without exception, one of the questions management asked about each piece of equipment was, "Can we patch it just one more time?"
Sadly, one of the systems was hosted on hardware with a h/w clock that wasn't going to make it. Not due to a two digit year, but a table of leap years stored in ROM ran out the following February, resulting in a system that wouldn't boot. he vendor had long since discontinued h/w support. Management's solution: bump the systems back 20 years and add a script to add it back in to the O/S clock after boot up. Plus some tweaks to the utilities that wrote the date to the h/w clock. It turned out that several different departments had the same hardware and each was responsible for implementing their own fix. Rather than standardizing on one date offset, each group used some multiple of 4 years (to keep leap years aligned) as a decrement. This resulted in transforming the Y2K problem into a Y2K + N problem, where N varied by department. After a while, it became evident that N was selected to move the new failure date to a point just after each manager was due to retire.
Have gnu, will travel.