Slashdot Mirror
Bringing E-Com Sites Down for Y2K?
dlb asks: "With Jan 1, 2000 just days away, the large wholesaler that employs me made the decision to disconnect our e-com web site from the rest of the 'Net. This was a heated debate for the past two months in the upper ranks between the paranoid and those who believe that bringing the site down manually is no different than some external entity creating the DoS for us (not to mention the loss of sales). For the other IT Professionals out there, are your companies bringing their sites offline this weekend? Why or why not?" Well, I guess if you are going to buy the hype, it's better safe than sorry, right?
It's New Year's Day. What sales did they think they were going to generate ANYWAY? Everyone will be recovering from their hangovers, watching TV, or doing something TOTALLY unproductive.
"normal" stores close on New Year's Day and don't seem to suffer any significant impact, right? I figure it like this: if one day per year is going to actually MATTER in your finances, it is time to quit using the company AMEX for those $1000-a-night strip club outings. Sheesh.
Of course, it's an NT server and it's for a group of about 60 people who work standard 40 hour weeks on the weekdays. I'm more concerned about power fluctuations because our site pulls a lot of juice.
Now the Linux server I use for my websites and mail (not at work!) is staying up....
--
how to invest, a novice's guide
In my mind, there's no doubt that this is the equivalent of a DoS. My question is this: is the site being brought down for a specific reason, or is it just vague paranoia? If the latter, then the bringer-down is responsible for any lost business. 'Something bad might happen, but I'm not sure what' would be an acceptable excuse for a mall owner to lock the front gates, and it shouldn't be acceptable for an ISnon-P.
I went to check on something there, and was faced with a 1960s style television test signal image, saying that VW.com is off the air until the night passed.
As an aside, I want to make a personal thank you to the Volkswagen Corporation... all through this year I'd been dreading the inevitable marketing hype about "The most anticipated event, the new Millennium Bug," or "the VW2K." Never saw a license plate Y2KBUG or anything. Kudos to avoiding schlock advertising![
The company I work for never mentioned bringing our e-commerce site down. However, we (unfortunately) have it hosted at cihost, so we don't have much choice in the matter . . .
---
--
If I actually could spell I'd have spelled it right in the first place.
Where I work, the servers are shut down for two reasons:
1. This way no one has to stay there and watch them.
2. We don't have to worry about damage due to power problems.
Why would you bring your site down? If the server is left on, it either crashes or it doesn't. If it doesn't, then you're fine. If it does, then you're not. If you turn the server off, however, then it's bad whether or not it's Y2K-ready. If it is, then you just DoS'ed n people, but if it isn't, then it'll explode or whatever as soon as you turn it on and it realizes it hasn't been invented yet. The problem isn't the changeover per se -- the problem is the first time it needs to know the year and it gets it wrong.
Switch the . and the @ to email me.
A site my company hosts is going to be going down from 6pm CST, Dec. 31(00:00 GMT, Jan. 1) until Sunday, 2pm CST. The company we host this for has requested that we take it down, not due to Y2K issues, but due to hackers trying to exploit servers due to Y2K issues.
:|
We run linux(duh), with apache and postgres. I personally have no qualms about the machine staying up, and I am not afraid of the server succomming to backdoors. But I don't pay the bills.
To turn it off, all we are going to do is ifconfig down the alias for their server. The machine will still stay up, running just the same as always.
There are essentially two kinds of IS managers: those with a solid computer science background, and the other kind. To the other kind, computers are magic, programmers perform an un-understandable task, and what could happen is infinite because they have no rational means of assessing risk. They cover up the fact that they don't understand the computers by using buzzwords and keeping current with all of the trade rags so that they seem to be on top of trends.
If your site can hold up on the average day, it should have no problem this weekend. There will not be a reign of terror by computer criminals (oh yes, if your IS manager calls them "hackers", that's another sign he's not a computer science pro). There will not be unforseen bugs from outside your site that damage you, and if you haven't fixed the inside bugs, well, some dates will be wrong. Big deal. Your backup tapes will not be magically erased on the very shelves where they lie.
My sites will be up tonight.
Bruce Perens
Bruce Perens.
There are good reasons to bring an e-commerce site offline for a few hours if you haven't tested the hell out of every last bit of functionality. You don't want order tables to be corrupted with records with incorrect timestamps, you don't want a bunch of old promotional prices to get reactivated, and so forth. You don't want to be vulnerable to similar problems in external systems your site uses as data sources. And when it's a commerce site, it's not just a cosmetic risk.. it's a business risk. Extremely cautious? Sure. But it's not an irrational move.
Similarly, if your webservers are running on an OS particularly vulnerable to viruses like, say, NT with Office installed (for generating RTF documents, etc.), you may just want to sit out a few particularly high-risk hours.
Where I work, I started only a couple of months ago and haven't had a chance to centralize and lock down virus protection. So prior to both Christmas and New Year's Eve, I made sure all Windows desktop systems and our lone NT server were all powered off, and they're staying that way until January 2. And all the fileservers got a full, level-0 backup a couple of hours before.
I'm not worried about the Mac server we have or the Linux boxes.. The former doesn't have MS Office on it and its System folder isn't shared, and the Linux boxes were installed and configured by me.
I want to enjoy this weekend, not spend it wondering if I'm going to spend Monday restoring systems from tape or cleaning a corrupted database.
eBay Availability on New Year's
The eBay site will be unavailable for Y2K verification from 15:30 PST to 18:00 PST on Friday, December 31 and from 23:00 PST, Friday, December 31 to 03:00 PST, Saturday, January 1. If you try to connect to eBay during these times, you may receive a "Failed to connect" error message.
We hope you'll read Meg's Letter to the Community. Thanks for your understanding and see you in the Year 2000!
signal, noise, to me it's all the same.
Whoa... this guy's thing was posted right after the audi story...
I don't see why any company should take down their website for Y2k... If the website is going down for Y2K (unlikely) then let it die a natural death. If there are no problems then you haven't lost any business have you.
Restating the obvious since nineteen aught five.
I left everything up and running... the last thing i need is customers calling and not getting our voicemail, or other amenities, and then thinking these outages are related to the date. If sh*t happens, I wont be alone, and people will be far busier with their own problems to be harassing me.
.. as long as the game is played at Ralph Wilson Stadium, and the Bills win I will be happy....
I have taken all precautions, done tests, applied patches... blah blah blah
btw, i am pretty sure both my linux boxes (at home) aren't gonna rollover, but they are staying on.
I defy the "bug", I will make my stand here!
"there's a big difference between kneeling down, and bending over" - FZ
I think that all this Y2K paranoia IS the Y2K bug.
More often than not, remedies for Y2K were worse than the problem. Senseless date expansion in interface files caused needless work.
I hope that VW is really upgrading their site. As a VW driver, I found using their site quite unworthy of their automobiles.
As an outsourced function, my companies web site will stay up as long as our ISP doesn't have any problems hosting it. There are no date sensitive components on our site.
Of course, I think it is silly that my company grounded the fleet over midnight local time, but is in full swing at 00:00 GMT. sigh
This is a boring sig
A more likely cause of a computer shutdown tonight is probably some drunk driver ramming a nearby power pole and shutting power to the entire building (and region). I would place that as many times more likely than a Y2K glitch. Dastardly
I think there is a valid PR reason to do it. There are enough factors that are out of the control of any IT manager: the power to the building, connectivity to the net, etc. Even if you have taken every measure within your power to be sure that you are Y2K compliant, your site may disappear. And that is really bad for public relations. People get the wrong idea, and nobody fully believes that it wasn't your fault. If you voluntarily take a site down and then bring it up early on January 1st when you are sure that everything around you is okay, you look a bit overcautious. None of this means that I think that there is any reason to be worried. I don't. I expect a quiet night, and I am on call.
The net will not be what we demand, but what we make it. Build it well.
The IT building at my university is going to backup power pre-emptively. There is a small power plant on campus that will take over if the main grid goes down. As such most of WSU's site will be up. The downside is that any non-UPS'd machines will go down during the 15 seconds it takes to transfer from external to internal power.
Even if you have PLANNED downtime and announce it, it will shake the customers' confidence.
I'm a security specialist so I've dealt with this already in my company:
It is ridiculous to shut down sites as a precaution against "hacker" or virus attacks. Ask yourself this question:
When I bring the site back up, has the risk of compromise gone away?
The answer is a resounding "NO". There is always a risk of compromise. If the Internet is so dangerous that you have to occasionally disconnect from it to protect yourself, then why do you even reconnect?!?! When you reconnect, nothing has changed except the calendar. Also, how do you know that the hacking hype wasn't designed to get you to disconnect now, and then reconnect days later only to have a false sense of added security since y2k is over and get 0wn3d on the 5th?? Isn't this an unknown, unsubstantiated risk too? You'd better never reconnect then...
The idea of disconnecting due to a y2k virus trigger is equally as ridiculous. April 1 is a more common day for virus and hoax triggers. Should every company disconnect then as well? Also, out of the thousands of viruses, only a handful have been very widespread. A massive virus infestation is historically unlikely.
Disconnecting due to some unknown, unsubstantiated threat is especially ridiculous (look at Seattle shutting down the y2k party...). It's CYA for lame IS and security people, IMHO. There are always going to be unknown, unsubstantiated threats. IS and security folks' jobs are to set up defenses to protect from day to day--that will work regardless of the amount of attacks. Shutting a site down for fear of someone breaking in is a self-induced DoS. E.g. the military sites that are being shut down (see http://www.hackernews.com for yesterday and today) during y2k are still going to have the same holes they did on the 1st....
Check out more specific information on y2k virus hype, "precautionary disconnects", etc. at the following links and see what:
"Precautionary disconnect" -- a disturbing new trend
OVERBLOWN: "Y2k Viruses"
Y2K viruses: "It's Orson Wells all over again"
Fearmonger vs. skeptic: a Y2K virus conversation
The virus grinches who tried to steal Christmas
-core
My employer shut down all its websites (at least at my location) not so much to prevent mischief, but rather to rule it out should any problems arise. Like the deductive principle so-often attributed to (but never explicitly uttered by) Sherlock Holmes, once you rule out the impossible (crackers getting into a system isolated from the rest of the world -- no modems, no internet, etc), then whatever's left (Y2K, loitering malicious code, etc) must be possible.
Christopher A. Bohn
cb
Oooh! What does this button do!?
The History of Y2K Problems
1994:
VP of IT: I'd like you all to meet Jimmy, the new Intern. Jimmy is a Sophomore from State U. Don't mind his complexion - it'll clear up, he just left his job at BurgerCzar.
Jimmy, it'll be your job to maintain these old systems. Ralph, you've been here 15 years
1995:
VP of IT: Ralph, we find it much cheaper to have interns maintain our code. Sorry, 3.8% raise this year.
1995:
Programmer: Ralph, heard you quit! Good luck in the Consulting market... I'm sure you'll be doubling your income.
1999:
VP of IT: Ralph, this is your old VP Ted. These old systems we have are screwed up! And we understand that your company manages Y2K conversions. Can you help? We'll pay anything!
2000:
CEO: Good job Ted, you saved our bacon! Let's not do that again - let's think about outsourcing all our IT functions to RalphCo. They're the pros. By the way, the president's son, Jimmy, works for RalphCo.
Bruce
Bruce Perens.
There are lots of factors, costs, and probabilities that a rational business must take into account when deciding if they should go offline. Like factors beyond the companies' control. Like expected benefit/revenue of staying online and the cost of dealing with a worst-case scenario.
If a company expects to take in some 1 percent of an average days' sales between 11pm and 1am on New Year's (who's shopping, really?), but their systems would cost millions of dollars and three days (== something like 250 times as much revenue as they would lose in a volunatry, two-hour shutdown, plus hardware and staff costs) to restore if heavily damaged in a worse-case-scenario, then who could blame them for giving up very small profits in order to be certain they avoid very high costs?
Bruce, you're getting hyterical about the "technology" and missing the business case. You don't really think we're going to see a headline in the Wall Street Journal like "Ford overtakes General Motors in Q4 1999 due to GM Web site being offline for 120 minutes", or "Amazon underperforms; missed out on big New Year's Eve midnight sales", do you?
Get real.
-Peter
While I can appreciate your zeal for placing MIS into two discreet factions, it just isn't that simple.
First, you have no idea what legacy connections exist between front line servers to the Internet a.k.a. web servers. All people see when they go to many sites is just that... a web server. There is no database box or ancient mainframe wide open on the net... also, if there is integration with authentication systems there is a possbility that an internal edict affects the external perception and functionality of a "site".
So, if you want to control input for a time when people will simply NOT be around and there is risk assessment regarding the personal lives of the professionals that report to you. For many the escalation plan is a pager on a belt loop.
Basically, if you airgap a web server you have just cut down the possible attack paths by at least 50% since nobody can come around to hit the site. Or, you have complied with the team decision to take it offline to take any possible stressors off internal systems that form a basis for external funcitonality.
Third, if it isn't a mission critical site then you take it offline and recall the functions. Most good commerce sites will engineer a boolean off value for maintenance purposes. It doesn't hurt anyone... are you intent on browsing heavily while getting toasted on champaign or sparking fruit juice tonite? I have bought some guitar strings tonite and might browse around but you know it isn't critical to me. :)
If you are a business you likely pay salary individuals to ride out situations like this. Since y2k is "hype" and misplaced concerns why not give people a night off so that they don't have to worry about the lesser qualified less certified more likely to play Quake on the corporate network at the expense of the website?
Shutting things down isn't a bad thing. Uptime is cool... but if it is a site that connects to other systems that require additional MIS staffing in the event of a unforseen circumstance are you as a "manager" going to explain to everyone why they need to stay alert just in case?
If your site is down this New Years, think seriously about wanting to be at work on New Years and buy your MIS manager a beer.
I respect what you are saying about IS managers not knowing what is up... but there is more to understanding a complex system than a computer science background.... you just open a whole can of worms when you go there gf.
Most seasoned IS managers know enough NOT to do something stupid.
I just think there is more than one way of looking at things in this area. So, unless you burned in the belly of corporate MIS and was there when things really hit the fan you might want to consider alternative views.
I know I am NOT one of those so I reserve judgement since I don't know all the pieces or the politics. Computers are still run by people ya know.
My sites will be up tonite too...
http://www.mp3.com/fudge/
http://fudge.org
Imagine an internet provider with the feature that they will cause your site downtime when it hits an arbitrary transfer limit for reasons that are entirely out of your control. It's practicaly an advertisement to find another provider.
Bruce
Bruce Perens.
If your site is down over New Years, think seriously about hiring a new IS manager
I've got to disagree with this generalization.
At our company, the MIS reports to me. Back in May, he said he planned to down all but our external servers.
Is he an idiot? Should we fire him?
He had just spent a weekend having the *entire* company's systems do a Y2K rollover, and then did transactions with all critical business apps. He found many problems, mostly small. One issue was that several older systems would not roll over correctly, but, once set to a post Y2K date, they were fine.
Rather than have a hardware/firmware remediation party, he figured we could just manually set the RTCs on boot after the new year. Sounded good to me.
The reason those throttle controls exist in Apache are for very specific reasons.
If you are looking for a shared hosting environment it is that same error message which allows other websites a chance at being seen for their payment of the exact same fees as Mr. Joe Popular website.
Price it out and do the math sometime... most providers use other means such as network throttles that don't afford you the 500 transfer limit message... also... that message can be tailored to have a more meaningful message.
Apparently, you have never read about people writing robots for site indexing that DO NOT conform to RFCs meant to govern the manners of a robot.
Its a sign that you are getting what you pay for from your provider _perhaps_.
Heck, do a Altavista search and see sites like OpenGL.Org which have that word indexed in the search engine database... it happens.
http://www.mp3.com/fudge/
http://fudge.org
It would seem that doing anything with this mindset would be, at the least, bad practice, but I know of some exec's that would stop at nothing to cut costs, and cut corners.
In the real world, where most of us live, there is a lot of Microsoft software. It has not been shown to be especially reliable, and I can't look at the code or hire someone else to look at the code for me. I have no idea what bugs lurk there -- and I don't think Microsoft does either, to be frankly honest.
Personally, I was in favor of taking our systems down overnight, simply to prevent date subtraction bugs. Someone else pointed out that this was making a change right before a major event, and that this probably wouldn't be wise -- a compelling argument, to which I acceded.
Regardless, claiming that I am somehow incompetent because I wanted to shut down systems over NY is flat stupid. Computers are not magic; they are highly predictable devices. However, the software that runs on a large fraction of them is not well understood by anyone. Trusting it unconditionally is foolish.
Consider that Microsoft was still releasing patches as of December 15.
Strikes me that you have a mighty strong opinion about how to run large networks, when it appears your expertise is not in that area. From what I can tell, you are a programmer, and a very good one. That's wonderful, but does not qualify you to make pronouncements about system administration. You probably don't deal, every day, with the stupid bugs and problems caused by unforeseen interactions in closed-source software. You live in a tightly controlled world of your own code. I don't have that luxury.
I don't presume to tell you how to do your job, and expect the same respect in return. And it strikes me that making public pronouncements on the competence of people working, every day, in an area you don't is not just arrogant, it's foolhardy.
You can trust I won't value your opinion as much in the future.
It turns out that Cybercash has been sending upgrade notices to Yahoo Store about this for months, but only in the last few weeks did Yahoo Store tell me about it. They notified me of this about two weeks ago, and First Data sales hasn't gotten back to me yet. (They have a "don't call us, we'll call you" sales policy.) So I'm offline for a few days. I can run transactions through by hand if I have to, so it's not too serious.
I work in a NOC, for a fair sized east-coast-based ISP, and I'll tell you this. TURN YOU CPE BACK ON! All of you that are turning off you equipment are making my new-years-eve a living hell! I can't imagine being at a UUnet, or AT&T right now. Those poor operators have to be pulling their hair out, calling all the down customers. So please out of the kindness of you hearts go to work for 5 min before you go out tonight to get loaded, and turn your crap back on!
This struck me personally as a tad draconian, but I can't really fault management; there's no reason to keep the facility open, especially since most of us weren't going to be there anyway. :)
They do understand the repercussions: a full shutdown means insane amounts of work just for our IMS department, let alone the actual R&D labs.
I suppose this is somewhat off-topic, since I'm talking about more than just web sites, but I thought I'd mention it and find out if any other companies are doing something similar.
Any other employers doing full shutdowns?
For everyones amusement, the syslog of our web distribution machine over y2k:
:)
Dec 31 23:46:36 util -- MARK --
Jan 1 00:06:36 util -- MARK --
Needles to say we didn't take our machines down. And its all working nicely thankyou
You can't win a fight.
The Linux kernel has had code to correct for the BIOS jumping "back" 99 years since mid 1995. The code resets the centenial portion of the RTC to 20 if it detects the jump.
---
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
That's a very fearful statement. If you've looked into the situation at all, you know that not only is your electric utility ready to meet the challenge, they have extra staff on duty tonight.
IS facilities are not in business to provide downtime. If they can't cope with the Y2K roll-over while hot, it's a sign of long-term mismanagement, because the problems should have been fixed years ago.
Again, if your site is down tonight, it's because your pants are down, buddy.
Bruce
Bruce Perens.
OK, you've got a bigger problem than Y2K. Your IS manager picked the wrong software, because everybody uses it. That software is downtime prone, but your IS person can point a finger at Microsoft, say but we have to use it, everybody does, and provide excuses rather than running systems.
Believe it or not, people don't have to continue to buy unreliable software. OK, you might think I'm uncompromising, but if that's what is happening in your organization, you already had a reason to find a new IS person before Y2K came around.
Thanks
Bruce
Bruce Perens.
I meant the whole site, with hundreds of servers and somewhere over four thousand people. Luckily, I only have the one big box.
--
how to invest, a novice's guide
Customers?
Actually, my utility has said a lot about its readiness. I happened to visit Hoover Dam recently, and they made a point of showing how they could manage the system with switches and relays, and without a computer, when necessary. As things played out, we lost one transformer here when someone shot it out, putting about 6000 people in Oakland in the dark. That is the only failure known for Pacific Gas and Electric at this time.
I think there's an emotional factor in this for me, too. Pride, I guess. I wouldn't feel proud to shut down for Y2K. I left my systems going, unattended, while I went to a party. The FTP log says the server was in use continuously, across midnight, by programs performing unattended downloads of the U.S. Map database. The Zope server log says that access of my web sites kept on throughout the night. Nothing has gone wrong.
Thanks
Bruce
Bruce Perens.
You know, timezones are not the same all over the world, so while it may be midnight here in Europe, in New York it is still around 18:00 in the afternoon. Why would americans stop buying at midday 31st?
If we wanted to switch off our servers for midnight, we would have lost a whole day of sales. But we didn't, and we were right.
Who where those panic-makers? Where are they now? Let's lough!
ms
OK - one of my companies does web development in Cold Fusion. We left our sites up over NYE for the following reason:
:)
:) All they had to do was turn them off on Dec 31 and turn them back on in the new year - no problems. That's what was done and what we're doing.
:) Once we figured what our risk parameters were, we could enact a plan (run with staff on hand/don't run/run without staff/etc).
1. They're outsourced at a hosting center which has 24/7 staffing, UPS, health-checks, etc etc etc.
2. Our sites are behind a firewall.
3. We did tests of our own to simulate the roll-over.
4. Full backups of all data, etc prior to rollover.
5. We had access to tech staff if necessary to resolve issues.
6. Close monitoring of data & performance over the first couple of weeks of Jan and the leap year to ensure "sneaky" corruptions get through.
Following assessment of the risks (power issues, communications issues, [cr/h]ackers, viruses, etc) we felt that we had done what was possible and that all should be OK. If there were any major hassles, it was likely that everyone would be in the excrement so we wouldn't be alone
Now, my other company does consulting to various clients. In the Small to Medium Business area, we recommended that they apply the latest patches and check their PC's for compliance. Some had PC's that failed the "tick over" in RTC and/or BIOS but worked fine in DOS, on the leap year and when rebooting post-1999. We recommended that they not throw out those machines (keep the $$$ to pay us more consulting fees, thank you
For those that did not need their systems turned on during this time, we recommended that they shut everything off and unplug it. While the electricity companies had stated that they were ready, they had (naturally) used guarded language. As such, when we reviewed the possibilities of power issues (brown-outs, surges and/or spikes) comm's issues (modems & ISDN connections) and software issues (relying on patches and information off the net, etc), we figured it was better to just avoid the whole thing so we could all be out partying and not sitting there watching a bunch of computers tick over.
So, in the end, it was all based on risk assessment. What level of testing had been done, were the systems required over the transition, what the unknowns were and how much risk the client could afford. It was easier to turn it all off, have fun and start it all up again when we knew what we were dealing with.
Of course, if I were the MIS Manager in some company, I would have been doing reviews, tests, simulations and so on for all systems. The results of all this would have been assessed with business management (MIS does not tell business what to do, we help them make their decisions
I left my body to science, but I'm afraid they've turned it down...