Slashdot Mirror
Multihoming Suggestions w/o at Least a /24?
An anonymous reader asks: "I work for a small company who is looking to get a multihomed Internet connection for redundancy.
The logical conclusion would be to get another internet connection to another provider. However, in the case of a primary connection failure, we need to be running BGP to have our internally-hosted sites still accessible to the Internet via the 2nd connection. The problem is that we only have a /28 (16 IPs), which is too small to make it past most route filters, and would then mean that we still couldn't be reached if the primary T1 is down. So, what's our options? (and no, lying and getting a /24 isn't a valid choice)"
The obvious choice is to get a second set of 16 addresses on the other connection, and then make your DNS server send out addresses to whichever connection currently works. Not all services like switching addresses, and sessions break when doing failover, but it might work for you. If you only care about outgoing traffic, load-balancing and failover is fairly easy to do and there are lots of products to help. Again, outgoing sessions will get killed if they happen to use the link that breaks.
Finally! A year of moderation! Ready for 2019?
and get your own /24, or whatever, saying you needed it to multi-home.
but this was back in the 1995-6; things may be different now.
-- www.globaltics.net
Political discussion for a new world
I've seen & skimmed a paper on Cisco's website on doing a hackish form of multihoming by using two WAN links that are NATted against each other. Googling is left as an exercise to the interested.
Of course, the usual question is: what can you afford to have redundancy ?
Because before technical solutions, you might want to review the contract with your access provider to include liabilities. The contract itself might cost more, but it might be simpler than a real redundant solution.
Because unless you know for a fact than your access provider is not reliable and has bad support, playing the redundancy game might be a bit more expansive than "simply" getting a double connection from the internet.
Let's do the excercise: you want a dual internet connection, that's OK, but you surely do not want a single router=single point of failure. So you have to buy another router, most probably the same brand as the one you already have, so to be able to use the (most probably) proprietary high availability solution. Provided your current model supports HA, or you will have to buy a more expensive one ?
Which brings to mind that having a redundant link (with an SLA :-) from the same provider might be an excellent idea, since they are probably aggregating your /28 to other /subnet, your route advertisment won't get lost in their network until it gets aggregated. Just make sure it does not get aggregated on the next hop ;-)
Well, if you are willing to pay for multi-homing, woul'dt it be easier to try to obtain an SLA with only one access provider, SLA including an redundant routing connection, with some redundancy protocol handled
[Pruneau
Set up your servers with a different IP for each route. Set up DNS inside your network so that the DNS server on one interface returns IP addresses that go through that interface, and vice-versa.. with a short expiry time.
If the main link goes down, so does the primary nameserver. The secondary nameserver (on the backup link) then returns IP's that are routed through the backup link.
This should work, but it probably goes against several RFC's..
455fe10422ca29c4933f95052b792ab2
The Radware Linkproof works great for incoming and outgoing connections. If you want redundant firewalls, you'll need two of their Fireproofs too.
Also, Cisco makes a product called CSS that's pretty snappy.
You're going to have to do your own redundant routing in between you and a network that is properly multi-homed with BGP out to the larger internet to make this work like I think you are really wanting it to work.
First, find an upstream ISP that is multi-homed to your satisfaction. Buy some IP's from them and put in a router or two for redundancy.
Next, build two or more tunnels to the ISP over different circuts or providers and run your own small BGP network on private IP's between the router at your multihomed isp and the routers on either end of your connection. Assign the IP's that belong on the multihomed network locally and let your own routers run BGP (or OSPF or whatever else you want to use instead like load balancing) between your LAN and the multihomed network.
It's hackish. It will be fairly expensive. It will also, however, work, let you keep your servers on-site, and give you greater control over redundancy and failover than you'd get with two upstream providers allowing you to use BGP anyway.
In the end, it might work out to be cheaper to do this in the long run since you wont have to pay any upstream ISP for letting you do BGP. You'll just have to pay for colocation somewhere, which could be a lot cheaper.
~GoRK
If the use of BGP is out of the question, there seems to be only one alternative. However, this solution still leaves the ISP as a single point of failure.
The option is Virtual Router Redundancy Protocol (VRRP). A brief description of VRRP, including a diagram, can be found here. Keep in mind that there are numerous other manufacturers that support the VRRP standard, you don't *have* to go with Cisco. Also, remember that with VRRP there is still a single point of failure, the ISP. This means that your ISP had better be a good one.
How about spending money to have a reverse proxy off site, in a colo somewhere, that handles which line to send it down. Clients connect to the advertised IP address for a site from DNS, which is the colo proxy/whatever, and then are either dealt with transparently, like a true proxy, or redirected to whichever line is up at the time.
Its something i have intended to look into for work, as it would jsut be a extension of what we currently use for firewalling anyways, port 80 is redirected from the gateway to a machine behind the firewall. To carryout a port 80 redirect on two publically available ips is probably jsut as trivial, infact as ive been thinking this through, i have tried it with OpenBSD on both ends and apache as the webserver, a rdr on the outside box gets my webpage fine.
Multihome to a single ISP that has multiple redundant backbone connections and do IBGP with them. His summarized aggregate routes will be multihomed on his backbone. You can then peer with him for your smaller subnet. I know it isn't as good as peering with two independant ISP's. Maybe you can connect to the same ISP at two different POPs to alleviate this somewhat.
While your providers are almost certainly supposed to be following address allocation policies from their address providers, you might be able to get them to issue you a /24 nonetheless. You can show a legitimate need for this amount of address space, even though you don't plan to use it for addressing machines--you want to multihome. Try to get your current provider to listen. If they don't, while shopping around for other providers, (which you will need anyway if you want to multihome), make a /24 a requirement of your contract. Renumbering 16 hosts won't be fun, but its not the worst thing in the world at all.
I know it's not strictly necessary to do the redundant hardware/network thing, but why not? It's only a few $20 network cards and a switch.
If you have multiple MX records (one per IP), then you won't lose any email either.
It would be nice to dual-home, but dual IP is a workable solution for the small business.
--Mike--
duh. use a data center with multiple redundant connections.
Well AFAIK you're required to have your own IPs from ARIN before you can get an AS number, and the minimum number of IPs AIRN will hand out is 4096 (and you need to show that you'll use 70%+ of them), then you can apply for your own AS number and use BGP and becomed multihomed. So it's a bit worse then you think. ;) If you really want redundancy get 2 lines from the same provider and make sure they're comming in on different sides of your building.
http://www.rainfinity.com/products/rainconnect.htm l
/24 determined we had already implemented what they could offer.
We evaluated their software, but since we have a
Slashdot? Oh, I just read it for the articles.
Hello,
There is no need to lie. Going multihomed is reason enough to request and obtain a /24 from one of your two providers, despite the fact that your network size only requires a /28. I have performed this exercise for companies of your size many times over, and trust me, any major network provider will give you a /24 if you are switching over to BGP and getting a second connection.
The effect of imposing a /24 or greater limit on BGP routes is that providers need to be more sensative to the needs of companies who, when considering network size alone, can't justify a /24. Thus, going multi-homed is enough of a qualifier by it self to obtain a /24 from an upstream ISP.
-James
I don't see anything noting what your T1 provider's policy/procedure for failure of their service.
Have you asked them?
Based on the information provided, here's my suggestions:
1) Are sattelite connections doable for you? It can be expensive but if you're company needs a quick fix until the main T1 comes back up, this may work for you.
2) Would it be possible to swtich your company's failed T1 through a secondary route to the internet? Most T1 Providers offer this.
3)DSL. I know this may seem wierd but in the event of an outage of service, you need to provide the bare minimum of connectivity to your customers internaly and externally.
Dolemite
Save the World! Use a Quote!
If something's worth doing, it's worth doing right. The main consideration from ARIN, as I recall, is a 70% allocation of existing address space. If you've got /28, two addresses are used for network/broadcast, at least one on a router, and at least one on a firewall.
/28 up into smaller networks, maybe like this, and a larger range is easy to justify-
/28 = /29 - 6 available host addresses
/29 = /30 - 2 available host addresses
/30 = /32 single host (loopback)
/32 single host (loopback)
/30 could be used for NAT translations, or any number of other things.
You do have (a) firewall(s), don't you?
At a minimum, that's already 25% of the existing address space in use, without including any servers you may, or NAT addressing you may be doing.
Break your
This scheme gives you just 5 useable addresses left, assuming you've got two routers and a firewall. The available
With 11/16 addresses used, you're at 68% of your address space. One more host puts you over the top.
Use this type of addressing to get the better of the two ISPs you choose to delegate a routeable block to you.
Obtain from ARIN, an ASN. Inform the other ISP that you want to BGP peer with them, tell them if you want full routes, or summary routes only advertised to you, and give them your ASN and the IP addresses of the router(s) they'll be peered with. Give the first ISP the same info. Both ISP's should give you ASN's and peering info for themselves as well. Here's the catch... make sure you don't turn yourself into a transit area between the two ISPs. Filter your BGP adverts so that you're only advertising routes originated in your AS.
Simple, huh?
Joe
Sig??? I don't need no stinkin Sig!
Radware Linkproof
Magic Radware box... check out the Linkproof or Linkproof LT depending on your needs. Dynamically add or drop providers seamlessly, use your exisiting IP's, handles the failover through a dynamic DNS system onboard.
I have two ISP connections- both look like ethernet to me. One happens to be wireless and the other comes in over a telco circuit, but the handoff is ethernet.
/30 addresses for routing purposes and "my" side has the /25 and /27 blocks they assigned me from their pool.
After much searching and testing I built my router using FreeBSD and IPFW. More on that further down.
Each ISP has given me a block of addresses from their CIDR block. I multihome proxy servers and email servers for inbound and outbound connections. They have one interface with multiple IPs bound. Nothing special there. Their default route is my FreeBSD router.
The freebsd router has mulitple ethernet interfaces. One per ISP and one for my servers. The ISP-facing interfaces have
The default route on the freebsd box is one of the providers.
I use IPFW for egress routing. Packets on the OUT side of the interfaces facing the ISPs are checked for source addressing and either allowed through or pushed over to the proper interface. Works like a charm.
interfaces:
em0 aa.bb.dd.128/25 (my side)
em0 xx.yy.zz.192/27 (also my side)
em1 aa.bb.cc.220/30 ISP A
em2 xx.yy.zz.188/30 ISP B
the rules I use:
ipfw -q add 100 fwd xx.yy.zz.189 ip from xx.yy.zz.192/27 to any out xmit em1
ipfw -q add 201 fwd aa.bb.cc.221 ip from aa.bb.dd.128/25 to any out xmit em2
There's also ipfilter in there handling filtering. IPFW only handles the egress routing.
DNS fills the gaps. I return at least two A records for the hosts I publish.
I used Linux for a short time in this router function but got bored with problematic network drivers.
That Radware device and the one by F5 are doing the same thing, but for at least 5 figures. I looked at them and then opted for this cheaper solution. I just bought duplicate router hardware and just keep a cold spare.
Keep an eye on these: 2002_7 and 2002_3
This applies to web servers as that is my primary concern, but could be used for other types of services.
Setup 4-10 front end web servers and do round robin DNS between them for your site. The 4-10 all need to be colocated on differnet parts of the Internet, differnet providers all in data centers is ideal. These sites can be load balanced/failover protected for more $$$ of course. These are your initial contact points for a client, and they have either your home page or a forward to your currently active site(s). This speeds up your ability to rapidly change from your live datacenter for your real site to your backup datacenter. You run a monitoring / mangament program that's constantly communicating to these servers which is the current live site. (this part of the solution must be VERY secure otherwise you're going to reduce downtime) Very secure and also automated fast, with a safe default to fall back on, like a local static page. Now if your main site gets hit by a 500MB/sec tcp/udp flood on random ports from randomly spoofed ip addresses (like a certian site I manage did 2 months ago), you simply move the current live site to backup site 2, or 3, or 4. If the attack is directed at your dns then it has 4-10 times the total networks to flood and should receive substantialy more attention than were it to take down only 1. If you have 10 front end sites up and one gets taken down then 10% of the people asking for DNS are going to get the wrong site, at least until you update your DNS.
Anyways this is a proposal to a problem made in hind site, but it's one we're working on implementing.
best of luck to you! If there is some poor mans BGP out there I've love to hear about it.
in re-reading this I realize it doesn't seem directly applicable to the needs (cost, etc) but if you were to use 1-2U per colocation facility and less than 1MB/month each you could pay in the few hundreds per front end site, and keep your local facility....
I run Linux on my HP Laptop
My Linux Command of the Day site : LCOD
Why not use colocation instead?
You're considering multihoming a small amount of computers. This has several adverse affects on the internet - mainly that there is another routing entry for your 64 computers.
Does your setup not allow for colocation? It makes more sense anyways because even if you have multiple links to your office - you can still get screwed if there is loss of service (be it a fiber cut or downed phone lines) to your CO.
With colo you need not worry about this.
You know, there are industries where security is paramount, and colocation simply isn't an option if you want any customers.
And just about anybody can get a wireless link these days.
Damn, I'm feeding a troll, aren't I?
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
I know this method will get flamed by quite a few people, but it works very well.
:)
:)
:)
:) But, it's pretty hard to take down 10 servers that can each push out 150Mb/s (dual NIC's bound together with teql).
:)
We want 0 downtime. There's no way to guarantee that any equipment is without failure. Something can/will always break. That's something you have to accept.
voyeurweb.com is located in colo facilities in both New York and Tampa. Each facility has it's own network drop. The size doesn't matter, but for reference, it's 1000base fiber in each location.
We have at least 5 machines in each location. Each machine has it's own IP, and in some cases multiple IP's just to increase it's load (faster machines can handle heavier loads).
You put multiple A records in your DNS. When a customer browses to your site, they get any one of the IP's randomly. Here's what an 'nslookup' returns for voyeurweb.com
> nslookup voyeurweb.com | grep Address
Address: 63.208.2.23
Address: 63.208.2.25
Address: 63.208.2.62
Address: 63.208.2.64
Address: 63.208.2.84
Address: 63.208.2.97
Address: 209.247.59.14
Address: 209.247.59.15
Address: 209.247.59.16
Address: 209.247.59.17
Address: 209.247.59.84
Address: 209.247.59.85
Address: 209.247.59.86
Address: 209.247.59.87
The 209.247.59 IP's are in New York. The 63.208.2. IP's are in Tampa. We're favoring the New York network a little bit, because we have some other specialized sites running in Tampa, and want an equal load between the cities. Right now, we're pulling about 450Mb/s per city at peak time. Just half of our 1000Mb/s drop. We've just added 1Gb/s fiber in Los Angeles, to increase our redundancy. How redundant you make youself is really up to how much the bosses want to spend, and how safe you want your site. Like I said, we want 0 downtime, and we achieve it.
The nice part is, if a machine fails, the client hangs for a few seconds, and then goes off to the next IP. If all the IP's in a city fail, the client can potentially hang for up to 30 seconds, before going to a server that works. Your browser will continue to use any IP that works.
We use a relatively short ttl in our DNS records, so if we decide to shut down all the servers in a city for any reason, within an hour all the traffic stops to them. I've done this many many times now, I'm 100% sure it works.
If we have a known problem (say a server has a hardware failure), you take it out of your DNS, and within an hour, no one is even trying to hit it.
We've done this with pairs of machines, or in the case of voyeurweb.com, up to 25 machines.
It's so simple it shouldn't work. I've been told by quite a few people that it won't work, but then I'll prove to them that it does..
Before we did this (before I was admin), if a machine failed, thousands of viewers would write in complaining immediately. Now, I can take a few machines down for maintaince, and no one notices. If we have a mystery crash at 4am, it's not fatal, we fix it when we can..
If you're a viewer, you probably didn't notice that we shut down all of Tampa for voyeurweb.com, for a week because of provider problems (lack of available bandwidth). You probably didn't even notice when we swapped out all the New York servers.. We were polite with some of them, but got bored with it, and just started yanking cables after a while.. We didn't receive a single Email asking why the sites were down. When a server went down, we did notice an increased load across the rest of them. It's nice having a *LOT* of servers running. If you have 10, you only get a 10% load increase across all the other servers when one goes down.
We don't depend on BGP. We don't depend on expensive load balancing equipment. We don't depend on anything other than the fact that people use browsers, and they resolve IP's through DNS.
In your case, you should have two ISP's providing you bandwidth. Each ISP should issue you a block of IP's from their available pool (like, it's hard to be on the Internet without it).. I'd say, if you want a site that stays up, set up a pair of mirrored servers. Give the first one an IP and gateway of the first ISP, and the second one an IP and gateway of the second ISP.. I could name off over a dozen sites that do this now, but I won't.
If you want to get real fancy, get one machine, put two IP's on it (one from each provider), and have a script monitor each gateway. If one fails, switch to the other. But this doesn't do anything for redundancy if one server should fail.
I hope I've explained this well. I've never seen it well documented anywhere. It's in the BIND documentation somewhere, but they have a real convoluted method of CNAME's and A records, which other documentation says are completely against some of the RFC's, so you shouldn't do it.
Even our evil nemesis does it...
>nslookup microsoft.com | grep Address | sort
Address: 207.46.134.155
Address: 207.46.134.190
Address: 207.46.134.222
Address: 207.46.249.190
Address: 207.46.249.222
Address: 207.46.249.27
Two different networks, splitting the load.
If CNN does it, it must be good.
> nslookup cnn.com | grep Address | sort
Address: 64.236.16.116
Address: 64.236.16.20
Address: 64.236.16.52
Address: 64.236.16.84
Address: 64.236.24.12
Address: 64.236.24.20
Address: 64.236.24.28
Address: 64.236.24.4
BTW, yes we get constant DoS attacks against us.. Sometimes I entertain myself by watching the logs.
To the script kiddie that "took down" one of our machines the other night.. Ummm, you didn't. I was annoyed at seeing the logs, and dropped all traffic from your network.
Serious? Seriousness is well above my pay grade.
We have two broadband links to our small downtown office. Each of these links terminates at an OpenBSD firewall. We briefly looked at what kinds of fail-over models being multihomed in the DHCP ghetto makes available to us (actually our ADSL service provides two static IP addresses, but they still grant you those addresses via DHCP leases).
We can do some cool things with Zebra (OSPF) over our VPN (all our offices / developer's have their networks in different portions of the 10.x space). If segments of our VPN fail, the traffic would find other segments forming a viable route to the final destination. Within the VPN we can pretend that we have a real network.
Fail-over to the outside world is more difficult.
One issue which no-one has mentioned here is stateful firewalling. Even if we can broadcast route information to the two OpenBSD gateway machines, it will break stateful firewalling completely. That said, most of our traffic is not session oriented. A broken HTTP request will fail-over at the application layer. No biggy.
The primary form of session oriented traffic we use is OpenSSH tunneling. The simple solution here is to remove stateful firewalling on the SSH port. I have to doubt that stateful firewalling your SSH ports accomplishes much.
For fail-over of our public web servers we decided against nasty DNS schemes. You don't get load balancing (just load sharing, which is not the same thing), there are issues about DNS record caching, your control is poor, when problems arise your ability to investigate the problems is all in a muddle. We don't tolerate that kind of thing here.
Instead we are directing all public URLs to a highly reliable and well connected public hosting company. That host is returning a frameset document containing a single frame window, with the embedded frame directed at one or the other of our two broadband services. Our firewalls are sending pings to the public hosting company so that the PHP script there can decide which of the two URLs to send out on each request.
There are plenty of extra complications since we are also thunking requests into different SSL compartments and we are using the OpenBSD firewalls to run chroot Apache in proxy mode (which allows them to filter the URLs before they reach the internal content servers).
With the right magic additions to the frameset code we can make the public URL track navigation through our content without revealing the ugliness of our internal fault-tolerant, SSL compartmentalized URLs.
Framesets suck, but not nearly as much as borking the DNS standard.
We are fortunate that our client base is narrow and we can impose client browser requirements. Our traffic volumes are low enough that we can afford the overhead of trampoline packets on each URL access.
We achieved a fairly low standard of fail-over in the end, but the good thing is that we didn't have to involve anyone else, and the fail-over we do have is sharply defined and easy to diagnose.
The internet is not about fault-tolerance and ability to 'survive nuclear attack' for anybody who isn't an ISP, large corporation or government department.
/16'
That idea went out the window with the introduction of CIDR.
There were good reasons for this, primarily the unmanageable growth of route-tables.
IPV6 will never see the light of day because if IPV4 can't be routed economically out to the edge of the network, then increasing the address space by a large factor will not help matters.
There is no way around this except by removing CIDR for a decent proportion of the internet, but route-tables will of course baloon hugely.
So, while 'the people' want to be able to multihome etc. 'the backbones' don't want to have to scale up their routing capacity by a large factor.
Which all boils down to the conclusion that the powers-that-be on the internet have decreed 'thou shalt not multihome unless thy pockets are extremely deep and thou has at least a
This is unfortunately the way it is, and won't change under the current 'internet regime'.
I gots ta ding a ding dang my dang a long ling long
What usually goes down on a T1? Normally it's a router module, the smartjack (most often it seems!), or something at the CO. *ALL* of these can be easily remedied with a second T1 with load balancing and failover. Nothing fancy. We do it here with CEF and it works great. Just run each T1 to a seperate router module.
What would BGP buy you? Well, if you ran your second T1 to another CO and to another ISP POP it would let you survive an entire CO or POP outage. But, how often do those happen with a good ISP? Almost never. How much would it cost to run that T1 to another CO? A ton.
I've already been down this road and found out that a dual T1 setup with something like CEF takes care of the job, unless you want to spend a fortune.
If you do BGP you'll need a bigger router. A small router won't do the job. When I was looking BellSouth required at least a 3640 w/ 128MB as the minimum for any BGP customer.
It's simply round-robin DNS to a bunch of web servers. It's a reasonable technique for your goals, but it's got nothing to do with the original question, which was how to keep his company online when one ISP/connection/link fails, and to both outgoing and incoming traffic of all kinds find the right place to go.
I asked almost the same question on the Cisco-NSP list a while back. Perhaps that discussion would be helpful.