Bringing E-Com Sites Down for Y2K?

dlb asks: "With Jan 1, 2000 just days away, the large wholesaler that employs me made the decision to disconnect our e-com web site from the rest of the 'Net. This was a heated debate for the past two months in the upper ranks between the paranoid and those who believe that bringing the site down manually is no different than some external entity creating the DoS for us (not to mention the loss of sales). For the other IT Professionals out there, are your companies bringing their sites offline this weekend? Why or why not?" Well, I guess if you are going to buy the hype, it's better safe than sorry, right?

Comments

*WHAT* sales?

[lorimer]

It's New Year's Day. What sales did they think they were going to generate ANYWAY? Everyone will be recovering from their hangovers, watching TV, or doing something TOTALLY unproductive.

"normal" stores close on New Year's Day and don't seem to suffer any significant impact, right? I figure it like this: if one day per year is going to actually MATTER in your finances, it is time to quit using the company AMEX for those $1000-a-night strip club outings. Sheesh.

What does it accomplish

[rde]

In my mind, there's no doubt that this is the equivalent of a DoS. My question is this: is the site being brought down for a specific reason, or is it just vague paranoia? If the latter, then the bringer-down is responsible for any lost business. 'Something bad might happen, but I'm not sure what' would be an acceptable excuse for a mall owner to lock the front gates, and it shouldn't be acceptable for an ISnon-P.

VW.com is off the air until 4am January 1, 2000.

[Speare]

http://www.vw.com/

I went to check on something there, and was faced with a 1960s style television test signal image, saying that VW.com is off the air until the night passed.

As an aside, I want to make a personal thank you to the Volkswagen Corporation... all through this year I'd been dreading the inevitable marketing hype about "The most anticipated event, the new Millennium Bug," or "the VW2K." Never saw a license plate Y2KBUG or anything. Kudos to avoiding schlock advertising!

I don't understand.

[Kyobu]

Why would you bring your site down? If the server is left on, it either crashes or it doesn't. If it doesn't, then you're fine. If it does, then you're not. If you turn the server off, however, then it's bad whether or not it's Y2K-ready. If it is, then you just DoS'ed n people, but if it isn't, then it'll explode or whatever as soon as you turn it on and it realizes it hasn't been invented yet. The problem isn't the changeover per se -- the problem is the first time it needs to know the year and it gets it wrong.

If your site is down, you need a new IS manager

[Bruce Perens]

If your site is down over New Years, think seriously about hiring a new IS manager.

There are essentially two kinds of IS managers: those with a solid computer science background, and the other kind. To the other kind, computers are magic, programmers perform an un-understandable task, and what could happen is infinite because they have no rational means of assessing risk. They cover up the fact that they don't understand the computers by using buzzwords and keeping current with all of the trade rags so that they seem to be on top of trends.

If your site can hold up on the average day, it should have no problem this weekend. There will not be a reign of terror by computer criminals (oh yes, if your IS manager calls them "hackers", that's another sign he's not a computer science pro). There will not be unforseen bugs from outside your site that damage you, and if you haven't fixed the inside bugs, well, some dates will be wrong. Big deal. Your backup tapes will not be magically erased on the very shelves where they lie.

My sites will be up tonight.

Bruce Perens

Re:If your site is down, you need a new IS manager

[jedinite]

If your site is down over New Years, think seriously about hiring a new IS manager.

Amen, Bruce.

I'm hear at work monitoring my sites (here and here to name a mere few), and I'll be here for the next 20+ hours.

Our upper management approached me with this same idea... should we pull our sites, or shut down our email, or etc, . My flat out response... NO WAY!

We're talking very important, very critical e-commerce, e-banking, and e-you_name_it sites that we've spent multiple millions on to keep running 24-7 x 365. Bulletproof sites which practically CANNOT go down due to disaster or mayhem, with state-of-the-art intrusion detection... so I'll be damned if i'm taking them offline due to the fear of a massive "CrackAttackY2k".

In fact, those sites pulling their servers offline are most likely going to lose my future business (or viewership, or whatever)... because they've definately lost my confidence. Such a big part of a website is public perception... I can't see how pulling your site offline can help that perception.

I think HNN said it best responding to the Pentagon and the Military Taking Down Their Sites

If your web site is vulnerable today it will be vulnerable tomorrow. This tells me that you are not confident enough in your own web sites ability to fend off attack but you expect the American public to remain calm during the Y2K rollover

---------
Question: How do I leverage the power of the internet?

Re:If your site is down, you need a new IS manager

[jedinite]

And here it is, the crucial data, according to one of my MANY servers:

root@www2[/opt/apache/logs]date
Fri Dec 31 23:57:56 CST 1999
root@www2[/opt/apache/logs]date
Sat Jan 1 00:02:51 CST 2000

The calendar has rolled to the new year, so far EVERYTHING is up, and no Year2000 glitches anywhere near any of my systems.

Now, re-examine BP's post. Those "suits" who took their sites down are responsible for the greatest DoS in history... and it's not from a distributed synflood or any group of elite crackers... but a group of PHB's giving in to FUD.

Happy New Year, SlashDot....

---------
Question: How do I leverage the power of the internet?

Protection against errors, not attacks, silly.

[hatless]

There are good reasons to bring an e-commerce site offline for a few hours if you haven't tested the hell out of every last bit of functionality. You don't want order tables to be corrupted with records with incorrect timestamps, you don't want a bunch of old promotional prices to get reactivated, and so forth. You don't want to be vulnerable to similar problems in external systems your site uses as data sources. And when it's a commerce site, it's not just a cosmetic risk.. it's a business risk. Extremely cautious? Sure. But it's not an irrational move.

Similarly, if your webservers are running on an OS particularly vulnerable to viruses like, say, NT with Office installed (for generating RTF documents, etc.), you may just want to sit out a few particularly high-risk hours.

Where I work, I started only a couple of months ago and haven't had a chance to centralize and lock down virus protection. So prior to both Christmas and New Year's Eve, I made sure all Windows desktop systems and our lone NT server were all powered off, and they're staying that way until January 2. And all the fileservers got a full, level-0 backup a couple of hours before.

I'm not worried about the Mac server we have or the Linux boxes.. The former doesn't have MS Office on it and its System folder isn't shared, and the Linux boxes were installed and configured by me.

I want to enjoy this weekend, not spend it wondering if I'm going to spend Monday restoring systems from tape or cleaning a corrupted database.

The opposite

[barzok]

When the company I work for listed all the systems that HAD to stay online when the clock ticks over, the public website was quite high on the list. Site goes down, people notice, customers worry about your compliance and how good your systems * software are.

Even if you have PLANNED downtime and announce it, it will shake the customers' confidence.

You're never free from risk--learn 2 deal with it!

[c o r e]

I'm a security specialist so I've dealt with this already in my company:

It is ridiculous to shut down sites as a precaution against "hacker" or virus attacks. Ask yourself this question:

When I bring the site back up, has the risk of compromise gone away?

The answer is a resounding "NO". There is always a risk of compromise. If the Internet is so dangerous that you have to occasionally disconnect from it to protect yourself, then why do you even reconnect?!?! When you reconnect, nothing has changed except the calendar. Also, how do you know that the hacking hype wasn't designed to get you to disconnect now, and then reconnect days later only to have a false sense of added security since y2k is over and get 0wn3d on the 5th?? Isn't this an unknown, unsubstantiated risk too? You'd better never reconnect then...

The idea of disconnecting due to a y2k virus trigger is equally as ridiculous. April 1 is a more common day for virus and hoax triggers. Should every company disconnect then as well? Also, out of the thousands of viruses, only a handful have been very widespread. A massive virus infestation is historically unlikely.

Disconnecting due to some unknown, unsubstantiated threat is especially ridiculous (look at Seattle shutting down the y2k party...). It's CYA for lame IS and security people, IMHO. There are always going to be unknown, unsubstantiated threats. IS and security folks' jobs are to set up defenses to protect from day to day--that will work regardless of the amount of attacks. Shutting a site down for fear of someone breaking in is a self-induced DoS. E.g. the military sites that are being shut down (see http://www.hackernews.com for yesterday and today) during y2k are still going to have the same holes they did on the 1st....

Check out more specific information on y2k virus hype, "precautionary disconnects", etc. at the following links and see what:

"Precautionary disconnect" -- a disturbing new trend

OVERBLOWN: "Y2k Viruses"

Y2K viruses: "It's Orson Wells all over again"

Fearmonger vs. skeptic: a Y2K virus conversation

The virus grinches who tried to steal Christmas

-core

Not putting down the self-educated

[Bruce Perens]

Actually, I didn't say where their computer science background came from. I happen to be self-educated in computer science, and my software has flown on the space shuttle, it's been used to make movies for Pixar, etc. I've never taken a computer course, but I read a lot of books and got a lot of hands-on experience. I majored in communication arts.

Bruce

nonsense; let's be rational about this

[peterw]

Somebody rate Bruce's post down as flame-bait. (Somehow it got the automatic Oh-My-God-It's-Bruce-Perens-Again 4 point bonus)

There are essentially two kinds of IS managers: those with a solid computer science background, and the other kind. To the other kind, computers are magic

Always good to start off with an irrational assertion.

If your site can hold up on the average day, it should have no problem this weekend.

So you know the status of my electric utility, and the capabilities of my UPS?

There will not be a reign of terror by computer criminals

which is not the only reason to go offline

oh yes, if your IS manager calls them "hackers", that's another sign he's not a computer science pro

Right. Using the wrong word is a clear indication of stupidity. And if you say "Afro-American" or "black" instead of "African-American", you're a racist. Thhhppppt!

There are lots of factors, costs, and probabilities that a rational business must take into account when deciding if they should go offline. Like factors beyond the companies' control. Like expected benefit/revenue of staying online and the cost of dealing with a worst-case scenario.

If a company expects to take in some 1 percent of an average days' sales between 11pm and 1am on New Year's (who's shopping, really?), but their systems would cost millions of dollars and three days (== something like 250 times as much revenue as they would lose in a volunatry, two-hour shutdown, plus hardware and staff costs) to restore if heavily damaged in a worse-case-scenario, then who could blame them for giving up very small profits in order to be certain they avoid very high costs?

Bruce, you're getting hyterical about the "technology" and missing the business case. You don't really think we're going to see a headline in the Wall Street Journal like "Ford overtakes General Motors in Q4 1999 due to GM Web site being offline for 120 minutes", or "Amazon underperforms; missed out on big New Year's Eve midnight sales", do you?

Get real.

-Peter

Re:nonsense; let's be rational about this

[Python]

Somebody rate Bruce's post down as flame-bait. (Somehow it got the automatic Oh-My-God-It's-Bruce-Perens-Again 4 point bonus)

Yeah sure, lets make sure no one can read it! Thats always the best way to respond to someone elses argument - deny everyone else the ability to read it! Talk about an irrational response. You should have added your response to his thread and sank with it. Its a bit hippocritical, IMHO, to rate down the original post and yet have your response rated up. Let people read the whole thing.

There are essentially two kinds of IS managers: those with a solid computer science background, and the other kind. To the other kind, computers are magic.

Always good to start off with an irrational assertion.

How is this irrational? There basically are two categories of people in this work, with regards to technology (IT managers are no different): Those that understand how the technology actually works, and those that treat any sufficiently advanced technology as magic. Its not an ad hominem, its not untrue and its not irrational. It explains alot about how people think about and treat technology. So why would this be an irrational assertion?

If your site can hold up on the average day, it should have no problem this weekend.

So you know the status of my electric utility, and the capabilities of my UPS?

How is midnight tonight any different from any other day, when the power could go out just as easily? The power goes out all the time and yet we don't see people pulling their sites down because it might happen. If you have a good disaster recovery plan (which includes things like "what do we do if the power is down for more than a few minutes", backup tapes and so on) it is true that you will survive y2k without any more disruption than you would have on any other day. If what you are asserting is that you are not prepared for what could happen any day, then you have other more important things to be concerned about and maybe you should shutdown altogether. These things can happen at any time. The power could go out for hours at a stretch (look at the ice storms in Canada for an example of that, the power was out for a week!), you could have an Earthquake that might not out your NOC for months, you might have a poor security model that makes your customers lose confidence in you and so on. These are daily risks and if you can't deal with those same risks at midnight tonight then you have bigger problems that shutting down your website for a few hours will not fix.

You talk about being rational, consider for a moment the propability that enough things will go wrong to outweigh the lost profit and the bad press for shutting down your site because you might have a failure. If your business is so unprepared for the midnight rollover, you do need to get a new MIS manager!.

oh yes, if your IS manager calls them "hackers", that's another sign he's not a computer science pro

Right. Using the wrong word is a clear indication of stupidity. And if you say "Afro-American" or "black" instead of "African-American", you're a racist. Thhhppppt!

Your straw man argument aside for the moment (and making funny noises, which is always a clear indication of a poor argument), yes calling a computer criminal a hacker is a clear indicator of cluelessness. Its like calling the internet "Netscape" or similiarly indicating you do not understand what you are talking about. If any MIS manager that works for few me starting calling crackers "hackers" I would look into getting another MIS manager (and yes, I do have MIS managers that work for me, and no none of them are that clueless).

If your company hasn't squared away your computers, programs and network or taken steps to prevent system failure (shutting it all down because you aren't prepared is basically the same thing as a failure) by now you need to fire the person(s) responsible for that. Its not like we found out about this yesterday, everyone has had years to prepare! Closing down your website for a few hours also demostrates a clear lack of understanding about the threats involved as well. Any "y2k" attacks will not just occur at midnight EST5EDT, they won't just happen for a few hours, technically they could have been happening all day (it doesn't just become midnight once on planet Earth). The bugs involved don't just manifest themselves during the rollover, the big bad y2k viruses won't just get sent at exactly midnight EST and so on. Again, I think Bruce is very correct about this: some people treat these technologies like its magic.

Regardless, this is the most ignorant thing I think I've heard of and its going to make a laughing stock out of the companies that do it. And if you can't understand a business case analysis for that, you understand the business word less than you think.
--
Python

Re:nonsense; let's be rational about this

[cburley]

Uh, guys, what most or all of you seem to be missing is that there's a whole class of (Y2K) bug that starts with miscalculating elapsed time from some kind of wall-clock times using only dynamic memories (i.e. times not saved between reboots).

Ideally, all Y2K (and other) bugs have been found and fixed, but assuming that they have is the disease of the modern computer professional -- the sort of person for whom the famous quote about programmers vs. builders vs. woodpeckers was invented.

So, you've got a choice. Leave the systems running over Y2K (my personal preference), which risks hitting that particular class of bug. The results could be reasonably catastrophic, depending on all sorts of factors (and I've certainly seen plenty of such results from simple bugs like this), but you get that extra, what, 1 hour of uptime? Or shut the system down and avoid that class of bug entirely.

Downsides, though: that sort of bug isn't necessarily local-time based -- it might be GMT-based; and there's a (my-guess-much-smaller) class of bug that prevents systems booting shortly after Y2K but doesn't affect their running through it. (I've seen non-time-related bugs like this.)

So it boils down to a simple choice. If your systems are specified to be up 7x24, leave 'em up (unless you know they'll fail over Y2K, of course, and can't do anything about it).

Otherwise, it's not a big problem for the systems to be down for an hour or two and skip a whole class of bug potentially biting.

After all, it's already been pointed out that systems go down ("DoS") due to power outages and other things not Y2K-related. Why shouldn't that lead one to the opposite conclusion for which these assertions have been intended, and accept that another hour or so downtime, especially in light of the fact that the systems will be least likely to be used at that time, isn't going to hurt anyone any more, and probably less, than any other outage?

Next point: rare activities, like doing incremental backups, since they invoke rarely-executed and rarely-seen code, are more likely to contain hidden Y2K bugs, perhaps including some not necessarily visible during certain forms of testing.

Given that, it's reasonable to do a "final Y1K" backup, right?

Now, as soon as Y2K rolls around, do another backup, then carefully verify all backups (perhaps moreso than usual).

Only problem -- what about transactions entered into the system, say by "enthusiastic" employees, between the last Y1K backup and the first Y2K backup, if that backup fails and the system gets corrupted?

Since that's more predictable (Y2K, after all) then any other random outage, it's not unreasonable to do the final Y1K backup with the system effectively shut down to further transactions.

That way, there is much less risk of lost transactions due to Y2K failures in rarely-executed code.

It simply is not stupid to shut down systems over Y2K, if that's what a reasonable analysis of the overall situation suggests. My wife's facility is doing this even as we speak (she's not the IT manager, but he works for her, and we just visited the site). Yes, I had an urge to say "that's stupid", and 20 years ago, when I was less experienced and less able to rationally assess risk, I would have.

Fortunately, I know better now.

I am concerned about how much hysteria might result from people reporting downed web sites in the early hours of Y2K, due to widespread use of the shutdown strategy.

But I'd rather people think, for a few hours, that Y2K bugs themselves shut these systems down than for actual Y2K bugs to cause real problems just because some overly macho IT managers decided to leave some non-critical systems on through Y2K.

And, really, would anyone here claiming this shutdown strategy is stupid (hi, Bruce! ;-) prefer that the world's nuclear arsenal be left on over Y2K, instead of being shut down and rebooted, on the theory that someone might want to use it? (Okay, that's a loaded question...sure wouldn't want to announce to the world that the USA's arsenal will be off-line for two hours starting at Y2K.... ;-)

Shutting systems down over Y2K. It's not what Joe Macho Hacker would do, but it's reasonably sane. And leaving it off permanently, if it's running any version of Windows, is especially sane. (I was watching my wife's organization actually shut down one of its few remaining VAX 6000 machines, permanently, while I was there tonight, by the way. It took me back a few years seeing the VMS diagnostics on the screen. Though, back when I actually worked there, their main computer was running TOPS-10....)

Y2K == CYA: Cover Your A$$

[tekan]

One possible (hidden) motive for bringing down some of these ecommerce sites is that it is the perfect cover for doing some task that would be prohibitive during normal business hours. An example would be an internal audit of a amazon.com type ebiz, or DB work on a site like eBay.com. So if something seriously goes wrong they can at least play the "Y2K" card.

It would seem that doing anything with this mindset would be, at the least, bad practice, but I know of some exec's that would stop at nothing to cut costs, and cut corners.

That's bologna

[Malor]

In the real world, where most of us live, there is a lot of Microsoft software. It has not been shown to be especially reliable, and I can't look at the code or hire someone else to look at the code for me. I have no idea what bugs lurk there -- and I don't think Microsoft does either, to be frankly honest.

Personally, I was in favor of taking our systems down overnight, simply to prevent date subtraction bugs. Someone else pointed out that this was making a change right before a major event, and that this probably wouldn't be wise -- a compelling argument, to which I acceded.

Regardless, claiming that I am somehow incompetent because I wanted to shut down systems over NY is flat stupid. Computers are not magic; they are highly predictable devices. However, the software that runs on a large fraction of them is not well understood by anyone. Trusting it unconditionally is foolish.

Consider that Microsoft was still releasing patches as of December 15.

Strikes me that you have a mighty strong opinion about how to run large networks, when it appears your expertise is not in that area. From what I can tell, you are a programmer, and a very good one. That's wonderful, but does not qualify you to make pronouncements about system administration. You probably don't deal, every day, with the stupid bugs and problems caused by unforeseen interactions in closed-source software. You live in a tightly controlled world of your own code. I don't have that luxury.

I don't presume to tell you how to do your job, and expect the same respect in return. And it strikes me that making public pronouncements on the competence of people working, every day, in an area you don't is not just arrogant, it's foolhardy.

You can trust I won't value your opinion as much in the future.

Yahoo Store Y2K problem

[Animats]

Much to my annoyance, my E-commerce site will go down and stay down tonight, because Yahoo Store refused to upgrade their Cybercash client to a Y2K compatible version. Yahoo Store wants everyone to switch to processing credit cards through First Data, and this way they have an excuse to force their merchants to switch credit card processors.

It turns out that Cybercash has been sending upgrade notices to Yahoo Store about this for months, but only in the last few weeks did Yahoo Store tell me about it. They notified me of this about two weeks ago, and First Data sales hasn't gotten back to me yet. (They have a "don't call us, we'll call you" sales policy.) So I'm offline for a few days. I can run transactions through by hand if I have to, so it's not too serious.

From The Providers Point Of View

[ajeskey]

I work in a NOC, for a fair sized east-coast-based ISP, and I'll tell you this. TURN YOU CPE BACK ON! All of you that are turning off you equipment are making my new-years-eve a living hell! I can't imagine being at a UUnet, or AT&T right now. Those poor operators have to be pulling their hair out, calling all the down customers. So please out of the kindness of you hearts go to work for 5 min before you go out tonight to get loaded, and turn your crap back on!

Re:Not so fast

[jedinite]

Not to be argumentative or anything, but I think I'll disagree with your disagreement with the original generalization (heh).

If your site is down over New Years, think seriously about hiring a new IS manager

I've got to disagree with this generalization.

I believe the key word in BP's statement was think. If your IS manager bought into the hype without a VERY valid reason (a valid reason such as the one included in your post, for example) then think about replacing him/her. Of course, I think that most MIS workers should think about replacing thier management on a daily basis, anyways ;)

---------
Question: How do I leverage the power of the internet?

Linux is fine.

[Inoshiro]

The Linux kernel has had code to correct for the BIOS jumping "back" 99 years since mid 1995. The code resets the centenial portion of the RTC to 20 if it detects the jump.
---

You mean let's be irrational and fearful

[Bruce Perens]

So you know the status of my electric utility, and the capabilities of my UPS?

That's a very fearful statement. If you've looked into the situation at all, you know that not only is your electric utility ready to meet the challenge, they have extra staff on duty tonight.

IS facilities are not in business to provide downtime. If they can't cope with the Y2K roll-over while hot, it's a sign of long-term mismanagement, because the problems should have been fixed years ago.

Again, if your site is down tonight, it's because your pants are down, buddy.

Bruce

Re:What have they go to lose by shutting down?

[gorilla]

But what have e-com sites got to lose by shutting down for New Years'?

Customers?