Slashdot Mirror
Redundant Internet Access?
Supp0rtLinux asks: "In order to meet uptime requirements and SLAs, we decided to get redundant T1's with BGP. We already had two Cisco 7200 routers and a T1. After the ISP turned up the additional circuit and we tested everything on our end, all seemed fine. But when the CO lost power and the generator failed, we had no access for 16+ hours. This prompted some investigations which revealed that yes, we did in fact have a redundant T1 with BGP setup and local redundant routers with separate UPS... on our side. However, on their side both our feeds were plugged into the *same* switch which was on the same PDU which happened to be in the same CO and was on the same sonet. And they were charging us for redundancy! Six month later, we have a truly redundant BGP setup. Each feed goes to separate CO's with the primary to the local one. This makes for separate physical switches, separate power, and we have confirmed we're on physically separate sonets. Now, the only true single point of failure is the physical cabling in the street, but in CA that doesn't get damaged very often. To those of you on Slashdot who know what I'm talking about: are your circuits truly redundant? What have your experiences in network redundancy been? How have you gotten past the sales guy to a tech that knows what redundancy really means? Have you been able to prove your redundancy? Have you found yourself paying for something that you weren't really getting?"
I haven't put the "on" to our redundancy just yet, but I can assure you one thing. When I do, two different companies will be providing the circuits.
Having them in two COs, redundant everything, yet linked to the same AS(when it isn't mine) makes me nervous.
Boss: We need redundant connectivity and power.
Sales-Goof: You can have as many people open browsers on as many computers as you want.
For comparison and not a plug, when my boss asked the IBM guy, he pulled out charts and wiring diagrams to explain what they had.
US Democracy:The best person for the job (among These pre-selected choices...)
To those of you on Slashdot who know what I'm talking about: are your circuits truly redundant? What have your experiences in network redundancy been?
I have two homing pigeons.
If Cupid smiles on them, soon I'll have even more redundancy.
Opinions on the Twiddler2 hand-held keyboard?
I worked at a place that was running redundant T1's just as you describe. They might as well have had all the wires running together the whole way.
My issues from there:
1. How do you convince an ISP to bring a feed in from another CO? Distance is a huge problem--they don't want to run it.
2. How do you know what the ISP has on their end, UPS's, generators, etc? Should that be part of the SLA? Or should you demand a tour of their facilities to see where your wire goes?
3. How can you coordinate two seperate ISP's for automatic redundancy? I suppose with a LinkProof box or something. And how do you know they aren't coming through the same telco CO?
4. Should you pay to have them manage the lines and router configurations in a 24/7 scenario? Or does it work well enough to have them do the initial install and then let it run?
5. Finally, what's a resonable cost for this redundancy?
I have some more projects that will be requiring this type of setup. Am interested to hear any opinions and recomendations from experience from fellow slashdotters......
Thanks much!
-m
http://www.invisik.com
Having "redundant" circuits to the same provider is pretty useless. You really need to be connected to two completely separate upstream providers for decent redundancy. If you have mission-critical needs, you want 3.
-Randy
My personal opinion is that trying to reach this level of redundancy for a lot of companies is just not practical and that there are much better approaches.
... or AT&T ... or MCI ... or Sprint ... or Cogent)".
The idea here is to think of your internet connectivity as two different classes of services. You should place your internet reachable servers in a good co-lo. Get BGP lines from two different sources and multi-home the boxes. Don't run your own AS (use the upstreams space) but instead place your servers "close" to your provider's edge routers. In the end, you are BPGing the loop and it is hard for 100ft of cat-5 to fail. In the end, you have to ask yourself "Am I more qualified to keep my BPG up than is Level-3 (or Savvis
In terms of your office, stick to client-only type services. Get two "diverse" connections. This might be a T-1 and a DSL, or a DSL and a cable modem. By using completely different architechures, you can get incredible diversity without spending a bunch of money. You can then IPSEC your local net over the client-only connection back to your addresses in the co-lo and with the help of a little client-side monitoring, auto-switch when a line goes down.
We offer something similar as a part of our hosting offering for users with green-screen (telnet, serial terminal) applications. A client gateway application manages logical "connections" back to our multi-homed central servers walking around BPG router "flaps" and other transient outages that BGP does not even address.
The BACKBONE. If your provider only uses one backbone, there's still a choke point. If the backbone goes down, for whatever reason (it can happen, and has happened), you've got the same effect as being redundant at your end but not at theirs... "theirs" is just further down the line.
There are providers that have multiple backbones, from different providers. I worked for an ISP that at the time had 4 different backbone providers. While there, I saw one of the backbones fail, stay down for several days because the backbone provider dragged their feet in fixing it. Everything else kept working, though, and the only difference was that during absolute peak useage, servers were very slightly slower in responding due to the missing bandwidth.
Being redundant between you and your provider isn't enough... ask if your provider's connection is redundant as well.
Dark Nexus
"Sanity is calming, but madness is more interesting."
If you live in any of the following areas...
:).
# Chicago, IL
# New York, NY # Greater Boston, MA
# Greater Providence, RI # Newport, RI
# Westerly, RI
TowerStream may be something to look into. I use them as our primary connection at the office - they are far cheaper than a traditional T1 ($350/mo for 512k, $500 for 1.5mbit, they can handle around 5GBit max I believe).
True line-of-site is not required, a reflected signal is usually sufficient. An external flat-panel antenna about 6 inches tall and wide is required, however. With ours setup on the roof, we get 0% packet loss, and have had no problems through heavy snow, rain or thunderstorms.
I have occasionally had connection issues, where the wireless modem has needed to be power-cycled. I suspect, however, this is simply due to it overheating
No matter who you order from someone has to do the last mile (aka, local loop). Typically, that's the Incumbent Local Exchange Carrier (ILEC), which is normally one of the baby-bells, or whatever they've become since they've started merging back together.
You might get a line from Sprint that goes through Chicago, and another from MCI that comes from Dallas, but when they get to your town, they hand it off to the ILEC, who runs the last mile.
Even if it was hooked up to a different switch, or was terminaed at a different CO, you still have redundancy problems -- odds are, the lines come into your building at a fixed point, which could be hit by a backhoe.
I know of an ISP that was serviced directly by a CLEC (the city-run cable company pulled fibre to them, besides the copper run from the ILEC...) but they were run on the same poles, so it didn't matter.
The only really redundant systems I know of didn't use wires for one of the components. Typically, they had lines pulled to two different places, through two different COs (in once case, in bordering states, that were on different power grids), and then connected the two with microwave. This way, the second leg completely avoided the ILEC.
It's not cheap, but well, redundancy doesn't tend to be.
In the long run, you have to look at what the costs are going to be, and what sort of losses it's going to prevent, and if the additional benefits are going to outweigh the cost.
Oh -- and typically, even if a CLEC (competitive local exchange carrier) has their own switch, the last mile is still typically handled through the ILEC, which puts you back in the same boat. Even with DSL, it doesn't matter if there are two different DSLAMs, if they're routed through the same CO or SLIC.
Build it, and they will come^Hplain.