Slashdot Mirror
Redundant Internet Access?
Supp0rtLinux asks: "In order to meet uptime requirements and SLAs, we decided to get redundant T1's with BGP. We already had two Cisco 7200 routers and a T1. After the ISP turned up the additional circuit and we tested everything on our end, all seemed fine. But when the CO lost power and the generator failed, we had no access for 16+ hours. This prompted some investigations which revealed that yes, we did in fact have a redundant T1 with BGP setup and local redundant routers with separate UPS... on our side. However, on their side both our feeds were plugged into the *same* switch which was on the same PDU which happened to be in the same CO and was on the same sonet. And they were charging us for redundancy! Six month later, we have a truly redundant BGP setup. Each feed goes to separate CO's with the primary to the local one. This makes for separate physical switches, separate power, and we have confirmed we're on physically separate sonets. Now, the only true single point of failure is the physical cabling in the street, but in CA that doesn't get damaged very often. To those of you on Slashdot who know what I'm talking about: are your circuits truly redundant? What have your experiences in network redundancy been? How have you gotten past the sales guy to a tech that knows what redundancy really means? Have you been able to prove your redundancy? Have you found yourself paying for something that you weren't really getting?"
The local telco will lie their asses off and charge you insanely expensive rates for mediocre service.
Unless you're in a downtown area or a tech park, forget about redundancy.
IMHO, anything facing the public that needs redundancy belongs in a colo.
Conformity is the jailer of freedom and enemy of growth. -JFK
My personal opinion is that trying to reach this level of redundancy for a lot of companies is just not practical and that there are much better approaches.
... or AT&T ... or MCI ... or Sprint ... or Cogent)".
The idea here is to think of your internet connectivity as two different classes of services. You should place your internet reachable servers in a good co-lo. Get BGP lines from two different sources and multi-home the boxes. Don't run your own AS (use the upstreams space) but instead place your servers "close" to your provider's edge routers. In the end, you are BPGing the loop and it is hard for 100ft of cat-5 to fail. In the end, you have to ask yourself "Am I more qualified to keep my BPG up than is Level-3 (or Savvis
In terms of your office, stick to client-only type services. Get two "diverse" connections. This might be a T-1 and a DSL, or a DSL and a cable modem. By using completely different architechures, you can get incredible diversity without spending a bunch of money. You can then IPSEC your local net over the client-only connection back to your addresses in the co-lo and with the help of a little client-side monitoring, auto-switch when a line goes down.
We offer something similar as a part of our hosting offering for users with green-screen (telnet, serial terminal) applications. A client gateway application manages logical "connections" back to our multi-homed central servers walking around BPG router "flaps" and other transient outages that BGP does not even address.
If you're in CA I'm guessing that SBC (Pacbell/whatever you know them as) is the local telco that provides the fiber service to your prem. I think you should be able to get diverse pathing from them. It will cost you some $$$, but is sounds like your organization is willing to pay for redundancy. They should be willing to do diverse pathing to your local CO, or diverse pathing to separate COs. You ought to be able to get strands going out of two separate conduits from your building, and completely separate conduits all the way to your local CO, or another nearby CO. You could have a CO SONET node in your closeset CO as well as a CO SONET node in a nearby CO and feed to your upstream provider from there (dunno if your upstream is PBI, which should definately do this, or another provider, who should as well). That way you can set up a healing SONET ring that will survive (in theory) a fiber cut (yes, they do happen. Even in our lovely CA :P ) or a CO outage (as long as your upstream can feed you from both COs). If you have a large enough netblock you should be able to get a connection from a second Internet provider and run BGP with them. Your problem then will be summarization at close by peering point, which is a complexity that you can get around (at a $$$ cost, of course). Just be aware that CO failures, cable cuts, and peering point failures all do happen, but you can always minimize or mostly eliminate if your organization is willing to make a dollar committment to it.
For the record, I am not an expert on this, but I have a bit of experience under my belt.