How Much Does Your Work Depend on the Internet?

malord asks: "I work for a small company that has recently had problems finding a stable internet connection. It started when we moved our office in order to upgrade our connection speed. We decided to go with cable internet through Comcast, since they offered the best speed for the price and told us that it would be available before we moved. Unfortunately, Comcast did not provide any service for two months after we moved, so we piggy backed on an existing (slow and unreliable) wireless account with another company in the meantime. When Comcast finally came around, the service that was provided was far from adequate with a consistent 30% packet loss and multiple disconnects everyday, which was confirmed through Comcast's tech support. Throughout this process, we have realized that having a reliable internet connection is more important than having a phone line and almost as necessary as electricity. What would you do if your internet was suddenly like dial-up for weeks at a time? How much money would your workplace lose if it was out for an hour or an entire day?"

Comments

How timely!

[Southpaw018]

On Monday, 8/14, we were due to hook up a T1 line with our new ISP. We hadn't had any severe problems with the old one, but our contract with them was up and they seemed apathetic when looking at negotiating a new one. So, we were going to cut over the lines, run the services concurrently for 2 weeks, and then terminate the old one on 8/28. On Saturday morning, our line went down at 1:01 AM. I was in the office at 6 AM Saturday, and I was NOT HAPPY to say the least. Tech support, however, seemed happier beating off than trying to help. They told me they'd give me a call back. The line was down all weekend. Monday was an exercise in frustration; instead of taking 2 weeks to do a changeover to avoid any interruption, we did the whole damn thing at once. We were up and running, completely changed over, DNS and all, by 4 PM.

You may think: hey, that's not bad. You only lost one day - really less than a full work day. Oh, but that's where the pain comes in. I run all our services in house: Goodlink (a Blackberry-like system), Exchange 2003, DNS, everything. Plus, while the lines were down, anyone who called our office heard five rings and was then disconnected. The loss in customer service is irreparable to one major client, and three unbelievably important emails were lost forever - the kind where the intended recipients weren't really in a position to say "Hey, can you resend that for me?" We'll never know exactly how many emails were lost. In a world that works 24/7, business never stops, and an important email that comes in at 3 AM is just as critical as the important email that comes in at 9 AM sharp.

Direct answer to your question: Our T1 line is beyond essential to the daily operation of the organization. It's absolutely mission critical that we're connected at all times, without interruption or major packet loss.

Re:How timely!

[Anubis350]

hate to say it, but if it's *that* critical, you *should* have 2 concurrent lines running, from different providers, on different trunks, with your servers set to fail over to the secondary if the primary dies...

Re:How timely!

[Night Goat]

Hopefully you come away from this with some new insight on how important the service is to you. Consider redundant T1s from two different companies. And consider using backup MX records for your e-mail, so that mail is queued rather than getting lost. Also, rather than having all of your phone lines running over your T1, you definitely should have at least one POTS line in case of power outages. Some of this was your ISP's fault, and some of it rests squarely on your shoulders for being unprepared.

Re:How timely!

[mgabrys_sf]

re:"Tech support, however, seemed happier beating off than trying to help"

I think that would apply to just about everyone on the planet (and many animals from what I've seen at the zoo). Why single out tech support?

Re:How timely!

[techno-vampire]

In some areas, two lines aren't enough. I worked for an ISP with a data-center near a major fault line. They had six different OC48s going out in different directions to make sure that if the data-center survived, it would have at least one connection to the outside world. Of course, most places don't need that much reduncancy, but putting all your eggs in one basket is never a good idea.

Re:How timely!

[scdeimos]

I wasn't fishing for snide comments!

South, the only person likely to read that as snide is you.

He's completely correct - if connectivity is as critical to your business as you're trying to make out then you should have at least N+1 redundancy not just for your comms links, but for your core servers like mail and web (if you're hosting your own).

You work for a non-profit business. That doesn't mean you work for a no-money business! Make a business case to your management *now* to get a redundant link so you're not a repeat victim. Don't wait one, two or six months to do this, do it now while the pain is still fresh in their memories! You may not be planning to change providers any time soon, but do you honestly think you'll always have completely unimpeded 100% uptime?

If you fail to do anything about this then you're no better than the noob at home who thinks his RAID array is enough for backups and then complains about losing his multi-terabyte porn collection when he's defrag'd after "accidentally" deleting it.

Re:How timely!

[samkass]

If it's that critical, and you're not yourself an ISP, then you shouldn't be pretending you are. Pay some ISP to do their job and get all the critical services off-site immediately. This is one case where you get what you pay for, and if your company is on the line, it's worth more than a single T1 without redundancy. Whoever decided on that setup should probably be replaced by someone who knows how to set up a company's infrastructure.

Re:How timely!

[peted20]

We are in the similar situation having Exchange in-house behind a (quite stable) DSL line. Thankfully the DSL has been out only about 30 minutes total in our first year, but unfortunately our Exchange server can't say the same. We've gotten an amazing value using a backup mx service, which silently queues mail for us until our server returns. It works amazingly well-- once our server is back up, the queued mail comes flowing in. Its a beautiful thing.

We specifically use EasyDNS's DNS service which includes the backup MX service. We use their DNS Plus service which only costs about $40/year, and allows us to use their CLUSTER of backup MX servers (How cool is that!?)! Its also available on their DNS-only service (~$20/yr). I don't work for EasyDNS (just a happy customer). You can also get the same service from lots of other places as well.

Realistically, I think you need to use an external DNS service to do this for network outages (since other mail servers will need access to your domain's MX records to find to the backup MX servers). For us, this meant we needed to use a different DNS server inside our local network. The external dns points people to our mail server's public IP. The internal dns points to our internal ips.

Another note, we use PFSense as our firewall (great product!). Recently, I think I saw support for NAT Reflection was added (allowing internal machines to contact internal servers using a public IP address), which might negate the need for the "split" dns described above. Haven't tried that yet, though.

Re:How timely!

[TufelKinder]

Okay, this doesn't really make sense. What if your company
runs on extremely tight deadlines? Where clients don't
always double-check that their emailed order was received
but expect a job to be done/shipped/delivered the same day?

It's not exactly feasible for most companies to offsite
one or more employees just to maintain a constant internet
presence.

Web hosting isn't the only vital internet service.

Lost forever?

[XanC]

Wouldn't the sending email server have continued to retry for four days or so? And wouldn't the sender have gotten a notification that the message failed if it had?

Re:Lost forever?

[Anonymous Coward]

Not if he hosts DNS behind the T1. All SMTP servers I know of cause permanent failure when they can't look up the MX record. This is why you should always have redundant DNS. I can see letting a backup MX slide as it can be complicated to set up, but hosting all DNS behind one connection just shows you don't care.

Don't use a consumer-grade service for buisness!

[inio]

> We decided to go with cable internet

Mistake #1.

You're a business. There's no reason a business should be using anything less than SDSL. It costs more for a reason - it's reliable.

quoth http://www.speakeasy.net/business/dsl/

> Symmetrical dedicated line DSL with throughput SLAs, rigorous uptime and repair time.

That means they guarantee it'll be fast, it'll work, and if it doesn't, they'll fix it fast.

If a couple hundred per month for internet is too much for your internet-dependent business it sounds like you've got bigger issues than packet loss.

Irreversable Damage

[AjStone]

My company would almost cease to exist if the Internet went down locally. I mean, it would be the end of life as we know it. That's why we are going to invest (in the new office building) in two seperate connections to the World Wide Web, with two completely independant companies.

Using two ISP's is a relatively untapped resource today, much like mirroring hard disk drives in a RAID array was a few years ago. Today, nobody will build a server without at least one redundant drive. I believe Internet connections should be the same way. How often do businesses complain of "sorry, our network/Internet is down" and lose customers? Do a Google search on a "Dual-WAN" router and see there are a few products around. I love my HotBrick LB-2 router that I use at home. There are about half a dozen people that can easily stress a standard RoadRunner connection. Using my friend's DSL connection going to the same house, it both load-balances and has failover capability. I don't even think twice before unplugging my cable modem. Without any downtime, the router will use the DSL line to pick up the slack.

Is it affordable? Well, that's the same question people were asking about mirrored hard disk drives years ago. The question becomes, is it nessesary? I'm not willing to move into a house that doesn't have the availablility of having two ISP's.

Aj

is this the right place for this question?

[macadamia_harold]

How Much Does Your Work Depend on the Internet?

Pretty much all of it. But then, look at the crowd you're asking.

Re:is this the right place for this question?

[NoData]

Is this the right place? I was thinking is this the right decade for this question. If I was going to be a snide Slashdotter (and I am, in fact, about to be) I'd say hey, 1996 called. It wants its Ask Slashdot back.

None at all

[Terminal Saint]

How much money would my workplace lose if we didn't have connectivity for an hour or even a day? None. In fact, we don't have an internet connection or even a dedicated fax line.

Hooray for state parks!

Same here

[mir@ge]

We had the same problem with Comcast here as well. They were largely unresponsive to our requests for assistance. After suffering with it for about 3 months, I finally convinced the boss to dump the money on a replacement. I called Comcast and explained to them that their service was unsatisfactory and we would be stopping it, breaking the contract and no longer paying them anything. It was fixed within a few hours and we have not had trouble with it since. Get tough with them. I think they save all the good technicians for when the customers threaten to leave. Typical.

Re:100%

[CthulhuDreamer]

"If my Internet connection is down- I go home."

If my Internet connection goes down, then I have to go to work.

Answer to the question

[starrsoft]

What would you do if your internet was suddenly like dial-up for weeks at a time?

Commit suicide.

Redundant feeds

[mschuyler]

And THAT'S why redundant feeds from different providers is necessary for any peace of mind. By the time I left my last job I had two T-1's from different providers entirely (I checked to make sure the cables were physically different coming at us via different paths), plus a third fiber optic feed. I was close to adding cable as a fourth. If the Net went out at that place I would have literally hundreds of people pissed within ten seconds. So have redundant feeds, redundant routers, redundant servers, redundant backups. Did I mention that redundancy is important?

Speakeasy

[beeblebrox]

quoth http://www.speakeasy.net/business/dsl/

I can't recommend them highly enough. Pick-up-after-a-few-rings, by-a-person-who-can-talk-dBs-and-DNS grade service, 24/7.

And that's on their residential product.

Here's a question for you

[Colin Smith]

Is it cheaper/better to...

1: Buy an SDSL business service from one supplier, with SLAs, rigorous uptimes and repair times.

or...

2: Buy cheap ADSL services from two or more suppliers but forget the SLA, uptime and repair time guarantees?

I strongly suspect that (2) is the cheaper and more robust system.

Re:Here's a question for you

[EtherMonkey]

2: Buy cheap ADSL services from two or more suppliers but forget the SLA, uptime and repair time guarantees? I strongly suspect that (2) is the cheaper and more robust system.

Except for the fact that both the ADSL lines go over the same copper bundles from the same CO from the same LEC regardless of whether or not they come from the same ISP. Most businesses feel that $600/mo for Internet service isn't worth the price until they realize how they've built themselves into a corner.

A sad fact is that most small and medium businesses will go through this pain and suffering at least once each year for several years before learning better and, in the end, spend more and lose more money trying to do Internet on the cheap than paying up-front to do it the right way. And while it seems sensible and cost-effective to host mission-critical services in-house, the reality is that if they are truly mission-critical and you can't afford proper redundancy, than those services are best off being hosted or deployed at a co-lo center that does provide N+1 redundancy and 24x7 business-class service and support.

Lessons learned:

• Public DNS should almost always be outsourced. This is a security as well as an availability issue.
• Email and eCommerce servers should always be hosted or co-located unless you have N+1 servers and N+1 sites with N+1 Internet connections and reliable failover technology.
• Internet service without an SLA is not suitable for mission-critical applications. This includes consumer-level xDSL, FIOS, and particularly CATV-provided cable Internet.
• Two ADSL's from competing ISP's is not N+1 redundancy.
• Your best options for mission-critical Internet access are (in decending order):
  1. 2x [fractional] T1, each separate carriers.
  2. 1x T1 and 1x SDSL, different carriers
  3. 1x T1 and 1x Business-class ADSL or Cable Internet.
  4. 1x SDSL and 1x Business-class Cable Internet

I've been out of the small/medium business consulting market for a few years now. But when I was consulting I encouraged customers to host or co-lo all mission critical applications and use terminal services (Windows remote desktop) or Citrix for access. The hosting or co-lo center provides all the redundancy and 24x7 service and support, you just pay the bill. The cost was not unreasonably more expensive than hosting these apps in-house when you consider downtime, maintenance and ongoing consulting fees to keep things going.

That's just off the top of my head. I could go on, but then I'd have to send someone a bill.

Re:Don't use a consumer-grade service for buisness

[dbIII]

You're a business. There's no reason a business should be using anything less than SDSL.

How about no choice due to poor communications infrastructure and regulations that prohibit any roll your own solution? If things were really critical a satellite link may be a possibility, but in a lot of places the low end of consumer grade ADSL is as good as it gets - even in state capitals in Australia 15km from the CBD.

Diverse networking is normal.

[Colin Smith]

Using two ISP's is a relatively untapped resource today,

Um... It's pretty much been standard practice since day one. It's how the Internet provides robust routing. All businesses relying on their network should be doing it. Diverse home networks? Depends how important your porn supply is to you.

bah that's nothing

[Danzigism]

try working for an ebay powerseller that lists 500 items a week, where the office is located the bumfuck of delaware, and using a dialup connection shared with 3 employees for 4 years straight.. each person's job requires an internet connection, and the bosses aren't exactly willing to put forth much money for anything tech related.. there has been no broadband offered here, yet my boss's house which is 2 miles away, can get cable internet.. I've looked in to distributing their net connection from home, to the office, but no can do due to line of site and expensive equipment.. same goes with satellite.. the satellite providers charge way too much, for a crappy service.. some even have download limitations that decrease your service to dialup speeds if you overflow your quota bucket.. that's after you spend a couple hundred in equipment, and probably around $100 a month for a business connection..

we're right on the border to where Medicom and Comcast seperate.. and verizon is simply a joke.. I've actually contacted the President of Verizon for Delaware's district, to no avail.. One of those typical, "I'll get back to you on that" phone calls.. For us to get DSL, would require them to spend a few thousand dollars in running new lines underground, as well as special hardware for the fucking FIBER FED PG BOX literally a hundred yards from our office.. Cable companies have also said, that they'll need to dig underground, costing thousands, just to lay some cable to our little warehouse..

I've thought of every possible solution, and they are either too cost worthy, or they simply won't work.. we can't afford to have downtime, and dialup is better than nothing at all.. but I did do the math, and we lose a maximum of a 1000 hours every year in productivity due to waiting for pages to load, uploading high res images for products, and the bulk submission of hundreds of ebay items.. ahh well.. i've definitely gotten used to it, but it makes me wonder how much more money we could make, if we just had a faster internet connection.. I certainly understand that even a crappy satellite investment could help us out big time.. but my bosses are still struggling to pay the monthly bills, so its really out of the question until someone like Verizon, Mediacom, or Comcast can offer a decent $30-70 a month internet connection..

Don't. Want. To. Think. About. It.

[fm6]

I'm a technical writer. I use the web to research the stuff I write about. It's essential. I seem to recall that I was able to do research before I had an internet connection — but I'm damned if I remember how!

Hang on...

[Colin Smith]

The ISP placed their data centre near a major fault line. OK i've heard of bravado but that's taking the piss. "Hahaha mother nature, we have six lines. SIX! Mwhaahaha... Do your worst!" CEO gestures at the ground cackling wildly.

Re:Hang on...

[WebCrapper]

I worked for an insurance company that had a call center that was down for a week due to a car accident. When they moved buildings, they made sure to pull double everything. Double water, power, phones, etc... Spent millions on the extras. They claimed it was the most up-to-date call center in the city. 3 days later, the whole thing was down again. This time: A car ran into the _1_ pole that ran the power, cable and telco lines into the building - some people are just stupid.

I changed strategies on that recently

[CFD339]

I no longer have a T1 to my home office server room. I have a consumer cable modem. I moved my public facing content to a machine at ServerBeach. It's faster, more reliable, and about 1/10 the monthly cost (I live a LONG way from any reasonable POP so a T1 was very expensive).

Now, the stuff clients see is 100% reliable (I have a failover server). The cable modem for my own use works fine -- in fact has been more and more reliable as the cables companies are now trying to compete for phone service and discovering people don't tollerate phone outages nearly so well as cable tv outages.

Re:I'm a Web Developer

[smash]

I suggest you get your own local test environment (IIS box, Apache box, etc) - regardless of the internet connection being up or down, you'll get far more work/testing done if it's hosted locally.

The Horror...

[Benzido]

The internet is super-critical to my job - philosophy. If the internet went down for a day, I would have to go to the LIBRARY!

Well...

[8ball629]

When our T1 (internet and digital phone) went out, we watched the newly released Pink Floyd Pulse DVD if that tells you anything.

Lose?

[Muggins the Mad]

> How much money would your workplace lose if it was out for an hour or an entire day?

For some of us, we'd probably make massive *gains* in productivity.

- MugginsM

Possible fix for malord's Comcast problem

[Brian Stretch]

malord may need a Motorola 484095-001-00 Signal Booster. Check your cable modem's internal webserver at http://192.168.100.1/ and if you do have a weak signal problem like I suspect (see Comcast's support forum and/or the Comcast forum on dslreports.com for how to do the diagnosis) then buy the amp. Yes, you shouldn't have to, but it's your best chance to actually fix the problem. Install the amp at the earliest possible point, before any cable splitters (if you have any).

If Comcast had any brains they'd keep a whole bunch of these in every Comcast service guy's truck and train their people to read the cable modem's signal status page. It'd be a helluva lot cheaper than repeated truck rolls to the same very annoyed customer. Better yet, they'd replace more of their aging copper with fiber before FiOS poaches all their best customers (alas, I'm in SBC/AT&T territory), but that's another rant entirely. Overall I'm reasonably happy with Comcast in my area but I'm still jealous of folks who can get FiOS.

$130 000/hr

[swordgeek]

Yep. A hundred and thirty thousand dollars per hour, 14 hours per day. That's a major player in the oil and gas commodities trading industry. That's why...

Servers are clustered.
Spare desktops are available.
Floor switches are redundant (and on separate power feeds).
Internet service is redundant (through two major carriers).
People have backups who know their job.
All service contracts have specific performance requirements.

If Comcast isn't meeting their stated performance, then they'd better FIX IT NOW! It's their job, after all. Mind you, if they haven't guaranteed anything to you, then they don't have to worry about any more penalty than losing you as a customer.

Get the SLA it in writing, hold them to it, and if they fail, legal action may be neccessary as a last resort.

The military does...

[n3tcat]

The military is becoming so dependant on the internet that when the net goes down, many combat support units are unable to do large portions of their job effectively. Combat support does not include the guys kicking in doors for launching artillery rounds downrange. It's all the guys who make ID cards and fill out insurance forms and fix the soldiers financial problems and such. Without the internet we can't connect to the databases we need to get to in order to modify Soldier's data.

Now this doesn't mean we can't do the job at all. It just means we have to switch back to the old paper and mail methods. This is significantly slower obviously, but it works.

It's interesting to me how the military doesn't do this for money, but rather for this idea that a Soldier's life is at stake. So does that mean that these companies that abandon paper methods don't take their work as seriously as the Army? Or just that the risk of saving money by abandoning these methods is worth it in the long run?

Does a day without net really matter? Or as the parent post mentioned, do months really matter?

It cost me my entire life

[Spiked_Three]

I have been programming for 20+ years. The last 5 or so was done from home. One day I had the brainstorm that since I worked at home I could move somewhere less expensive and with less traffic, so I packed up and moved to east Tennessee.

Guess what I found? They barely have internet here. I had to pay $600 a month for a T1 from Bell South and then found out the infrastructure and/or local workers could not make it run reliably. I had SLAs which were totally ignored. Monthly credits were usually close to the cost. And finally the line went down completely for over a month. I was forced to switch back to dial up, and if I got a 2400 baud connection I felt (feel) special.

Needless to say, I have lost all of my clients, all of my work, they take my car in a month. Moving is not an option for unrelated reasons, but the bottom line is there are still places in the country where internet can not be reasonably obtained (i put satellite in the unreasonable group - it sucks - try VPNing with a dish) and it cost me everything. No this is not made up.