Slashdot Mirror
Working With 2 ISPs For Home Networking?
An anonymous reader writes "This is, I think, a simple question — but one which I can't get the answer to.
As a typical, but perhaps high-demand home user I would like to use 2 separate ISPs. ADSL is pretty cheap nowadays, and 2 x ADSL seems a better value than one fast one — especially in terms of reliability.
If one breaks, at least the other will work.
Using an old box as a router/firewall, how can I configure a system to use two completely separate ISPs in a sensible manner?
Ideally, I'd like the load of my browsing to be balanced, but at the minimum, I'd want some kind of 'fail-over.' If I leave torrents running over night, I'd like the router to use whichever connection doesn't block the traffic — and preferably for it to reset the errant connection.
Ideas?"
ADSL is pretty cheap nowadays, and 2 x ADSL seems a better value than one fast one â" especially in terms of reliability. If one breaks, at least the other will work.
When your DSL is down, it's likely that your neighbor's DSL is down too. Consider cable + DSL, not cable + cable or DSL + DSL.
You can get a "Firebox" VPN/Firewall/Router pretty cheap on ebay. They are running about $75.00US for the Firebox 1200/2. The "/2" part means it has 2 WAN ports and you can load balance across both, it is setup to be redundant, so if one goes down, it moves all traffic to the other automagically. I use one and it works like a champ. There are more expensive solutions, and probably "Roll your own" solutions, but as most of us know, that can provide months and months of aggravation!
"My immediate reaction is "WTF? What kind of moron doesn't make things 64-bit safe to begin with?" Linus
There are little Linux distributions like Brazilfw which run on old hardware and work out of the box with features like QOS, load-balancing, port forwarding, etc. Maybe that's what you need.
pfSense can handle the load balance and failover for you. Then you just need to get two ISPs. Preferably one cable + one DSL but if you can get the two DSL lines on separate circuits, that would work well.
Isn't a dual-WAN router the simplest/cheapest method, whatever you are planning to put downstream of it? http://www.networkworld.com/reviews/2004/0913rev.html
called Clarkconncect (http://www.clarkconnect.com/)
It's basically a CentOs (aka free Red Hat) wich can do multi-Wan. It has a nice web interface fir Firewall, ftp, web and mail server, shell..
No idea if it can reset errant connections, but it can do anything you can on redhat, including using two Wans simultaneously. (chek Clarkconnects forums for multi wan)
up and running within 30 minutes, mine has reached 165 days uptime (Bi-P3 GHz, 2 Go Ram, 4*500Go HDD, 3*Eth 100 (upgraded from a faithfull Compaq Deskpro 400 Mhz "server")- web, mail, and bittorrent dowvnloader (torrentflux-bart) as well as "media server" connected to the xbox with XBMC)
It takes 40+ muscles to frown, but only four to extend your arm and bitchslap the motherfucker
http://lartc.org/howto/lartc.loadshare.html
Check it out.
Most DSL circuits, even sold by different vendors, go through the same facilities and sometimes the same equipment. For example, the local loop is usually the local telco's, no matter who your DSL vendor is. And many DSL vendors resell one of a few wholesale providers (e.g., Covad), so your data on both DSL lines could be going through the same wholesale provider's equipment/facilities. The same may be true of other technologies (e.g., fiber).
In trying to setup something similar, we finally settled on using cable for one circuit and fiber for the other. We know the cable company has its own local loop, and they assured us (FWIW) that they have their own facilities out to their upstream provider (e.g., AT&T, Sprint, etc.). Fiber would be Verizon. We would use DSL, but I'm concerned that it would end up in the same Verizon facilities.
Good luck. There are also routers that do fail-over, but I know that's not what you asked about.
Hotbrick makes a very good load-balancing soho router. They're a bit pricey but they seem to work quite well for exactly what you're describing. Take a look on ebay for their LB series.
I do have to second the suggestion of using Cable+DSL rather than DSL+DSL. Most places where there are multiple DSL providers, they're both operating from the same physical infrastructure with one reselling the service of the other. It's certainly better than one by itself, though.
Even people that believe in pre-destiny look both ways before crossing the street.
Honestly, I think that's not understanding how DSL works very well. In virtually all markets, there's one physical DSL provider, and a few dozen 'ISPs' which cost a little bit more to provide potentially 'unique' services on top. One monopoly for phone (and hence DSL), one monopoly for cable.
Er, the cheapest DSL is what, around $25, $30, for 256k? Double that, and you've got a price for very fast (8mbit or more) cable, including 256-512kbit upstream. Even if you have 2x256k, and the equipment to use it in a decently efficient manner, that's still some 512kbit, and two different IPs.
Only in a few situations can you use the bandwidth of both cooperatively for a single task, and the most common failure is based on when the physical link/line conditions deteriorate, in which case having two ports to the same network isn't going to make any difference at all.
Cable/DSL will provide the potential reliability you'd be looking for, I think. But, as a home user, some 98-99% (even if not 99.97%) uptime isn't good enough? For the additional cost, it's not worth the extra -average- hour per month of downtime you gain 'back'.
If your ISPs downtime is any more than that, you have every right to complain, twist their arm to fix whatever might be causing the problem.
"A Goddess rarely smiles for she is forced by others to be an island unto herself." - Zephiris
A dual-WAN router is the easiest way to go, but I wouldn't call it cheap. A decent dual-WAN router will cost you about twice what it would cost to build a cheap, but decent linux box.
Seriously? Is your network infrastructure -that- unreliable that its actually worth *doubling* your costs for redundancy?
I have had maybe 10-15 hours of internet-only downtime in the last 8 years. Of that, maybe 4 hours affected me (ie I was awake and wanted to use the internet). I've had another 10-15 hours of power fail in the last 8 years, and even with backup power the internet was still down (routers, switches, etc in the upstream path weren't on backup power so keeping my 'modem' up isn't worth beans.
In any case, I can see a lot of situations where it would be worth another $2500 over that period to have had internet access for those couple hours.
If I were running servers (and I am), it might be worth it, but in practice its not worth the trouble. round-robin DNS just means every odd connection attempt fails if one of the links is down, and dynamic dns updates to take the downed link out of rotation would be great except most internet outages are over before dns updates are likely to propogate. So its just not effective.
If I wanted -faster- downloads, that might be worth 2 connections, but that's not what you claimed your objective was. And even then, it usually won't make a specific download faster, but will rather let you do 2 at once at full speed (in the case of a large http or download for example which only uses one connection) which may or may not be what you need. Torrents, using multiple connections, will of course benefit from the extra bandwidth capacity.
If you SERIOUSLY want redundancy, you might want to look at a router that can fail-over to dialup. That will actually stand of chance of being available during a power failure, and might not cost you extra in terms of service, since many ISPs give you some free dialup hours as part of your broadband. And the dialup infrastructure is often separate enough from the adsl/cable infrastructure that you'll be able to connect on dialup while adsl/cable is down.
You just get a Linux box with 2 NICs and start adding static routes :
route add 1.1.1.1 255.255.255.255 eth0
route add 1.1.1.2 255.255.255.255 eth1
route add 1.1.1.3 255.255.255.255 eth0
Etc, etc....
It might seem like a big job, but there's huge ranges of reserved addresses you can skip. Let us know how you get on.
What you want is a "dual wan" router. Which will give you two ways out, by default putting each connection between your local host and a remote host over a single WAN's route, but pool the two WANs so the less-full one gets the whole next connection.
Then you want to look into "bonding", or whatever the router vendor calls their version of it. It usually doesn't work, because the two different WANs usually take very different routes most of the way to the remote host, and the bonding has to accommodate all the hops between on each of the two WAN routes. But sometimes it does work, especially if the routers at both ends of the routes share the same bonding technique.
But you will indeed get immediate uptime benefits. Because if one WAN gives you, say, 99.9% uptime, that's 0.1% downtime, which is still over 31,000 seconds down a year, which is still almost 9 hours. But if you can get connections over either one WAN or the other (each at 99.9%), you can get 99.9999% uptime, which is only about 32 seconds a year, which is unattainable at reasonable prices for a home user.
--
make install -not war
Great, so you googled some shit. Maybe he wants to get some people's experiences with them? What is good or bad?
A witty saying proves you are wittier than the next guy.
It sounds like multihomed routing is what you're looking for. there's a decent intro here:
http://www.oreillynet.com/pub/a/network/2002/08/12/multihoming.html
Women are like electronics: you don't know how damaged they are until you try to turn them on.
Multihoming:
Cable/DSL
http://en.wikipedia.org/wiki/Multihoming
Multihoming caveats:
http://en.wikipedia.org/wiki/Multihoming#Multihoming_caveats
Get matching NIC cards.
~hylas
I have quite a bit of experience with this, as I use two consumer ADSL circuits to provide very reliable 'net services at my office.
To an extent you either get to use two different services (for reliability) or combine them into one service for improved performance. Not both.
If you're going for reliability, you'll be using two different providers. That eliminates the use of multilink PPPoE to bond the two services into a single logical service with a single public IP address. It also eliminates ATM channel bonding, which is the other way to achieve the same end. This isn't such a great loss as you might think since I've *NEVER* found a provider (at least here in Australia) that knows what either is, let alone supports even one of them.
So, you're stuck with two ADSL circuits, each with separate PPPoE connections (or direct IP over ATM links; either way) and separate public IP addresses.
This sucks. You can't even load balance across them properly without the cooperation of a router/proxy on the other side of your ADSL links.
Load balancing your transmissions on a per-packet basis is obviously hopeless because any sane ISP has egress filtering based on source IP address, and even if they don't you'll still get replies back on the official source IP (so you won't gain much). SNAT won't help because if you SNAT some packets in a connection the recipient will have no idea they're part of the same connection as the unmodified packets leaving on the other connection. The only way that packet-level load balancing across multiple links with different IPs will work is if you're only talking to an endpoint (probably a VPN termination point) that is aware that you're using multiple connections and can combine them. You can use tricks like multilinked PPTP for this, or iptables trickery on each end. In any case, you're going to need access to a server with enough bandwidth to service both connections that's willing to route traffic for you. You probably don't have this.
So, packet-level load balancing is out. What's left? Connection-level, and per-protocol.
Connection level load balancing works well for some services. Outgoing SMTP, for instance, is well suited to being randomly allocated between multiple ADSL links (if you're unfortunate enough to have users who think that 100MB attachments are a good idea). Unfortunately most home user services like HTTP web browsing are not. You'll find that websites like to store session data with your IP address, so if you do connection load balancing with HTTP you'll find that websites keep on forgetting your login. To work around this you need to use "sticky" load balancing that remembers which connection was used to talk to a given host - but that, of course, reduces the benefits of the load balancing.
In the end, all you can really do is a bit of sticky connection-level load balancing when establishing new outgoing connections for some protocol types. If you want more than that, you need to do ugly things like say "all FTP connections go out ADSL1, and all SIP and other VoIP connections go out ADSL2" etc.
Personally, I don't bother even with that. I have both ADSL services listed as MXes for the company's DNS, so if one is down we still get mail. The A record points at a colocated server elsewhere on the Internet, so that's not a worry, but if it didn't I'd have to use some sort of ISP-level or colo load balancing to reroute traffic down whichever link was currently available.
Outgoing connections just all use the primary link when it's up, and fail back to the secondary link if/when the fast one is down. The secondary link is the primary MX, so when both links are up mail will tend to come in one link and everything else in the other.
If I wanted more than this, I'd probably have to route everything through another server colocated at an ISP or peering point. Unless I could get free traffic between it and both my ADSL circuits this would get expensive fast - and it'd also reduce the benefits of the redundant ADSL links
God, not another person saying this.
Slashdot articles aren't just posted for the question, but for the discussion. Yes, anyone can find an answer to anything they want with Google+Wikipedia+etc.
The point here is that maybe someone will take an interest in it that never thought of it before or cared enough to dig around Google.
Obviously from the author's point of view, multiple viewpoints by the readers would be helpful. However from the Slashdot mods (and community in general) it's an interesting enough topic to read on their own.
YES
I live in the Rockies on the western edge of a mountain ridge at 10k ft elevation - in other words a lightning magnet. I'm a full-time telecommuter for a multinational, & I work daily with people from 5 different time zones. Teleconferences, webex's etc. are my daily work life. Loss of connectivity to our source code repository can be a serious problem.
EVERY time there's lightning with 1/2 mile of here my phone & DSL go out. Last year I was out 7 different times for more than 24 hours. I lose track of the number of times I'm out for just a few hours.
I have a secondary ISP - WisperTel, a wireless WISP - that's a lot less reliable than DSL. Latency is bad, it's down a couple times a day at least, although usually for short periods.
To top it all off, I'm outside of cell phone coverage... and I have 3 DIFFERENT carriers. I'm only 1/2 mile to the nearest coverage, so I can drive or walk to make the necessary calls when both ISPs are down. This is fun when there's 3 ft of fresh snow on the ground, and it's -10F. Thank goodness for snowshoes... (Last year alone both were down at the same time 3 different times).
If I could also get cable here I probably would... although I do hate Comcast with a passion.
Stupidity... has a habit of getting its way.
There are four main reasons that DSL goes down
I've had DSL fail four times in the last 10 years. One was my DSL router. Two were when phone company installers working on boxes down the street disconnected me by accident. One was a billing problem (but that was when my ISP was providing beta service, and they mixed up things between my home account and work lab, and I was customers #1 and #2 in the western half of the country :-) Some of these can cause both circuits to fail, some can't - and backhoe events are pretty rare. On the other hand, cable's more likely to have common failures than DSL is, unless you're one of those rare people with two cable providers, because there's more shared infrastructure between the two circuits.
Even so, I'd recommend going with two different providers because they're going to have different performance issues and probably different policies. If it won't interfere with your cable TV service, I'd recommend cable and DSL - cable's usually faster, though more likely to be flaky, and more likely to have obnoxious limitations on your service like not letting you run a web server at home or giving you 20 Mbps of download speed with a monthly download cap that limits you to an average of 50kbps if you use it 7x24. DSL is more likely to be reliable (because infrastructure gets fixed along with lifely telephone service as opposed to television), probably slower depending on your distance from the telco, and you usually have a choice of dozens or hundreds of ISPs if you don't like the policies or pricing your telco offers.
One obvious way to mix the two services is to have a DSL with a static IP address, and do most of your own downloading from the cable modem. You'll need some kind of router to deal with keeping track of the two services, and some kind of firewalling, so you probably want to use an OpenBSD to do that and whatever your favorite Linux, Mac, Windows, or game boxes behind it. (I'm picking OpenBSD because it's usually the best at security and firewalling and at least OK at routing, and you probably won't be putting anything requiring fancy hardware drivers on your firewall.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Let's start at the bottom of the OSI stack - physical layer. The wires from your house to the telco office are usually physically separate until they hit the first active device, which might be a Subscriber Loop Carrier in a big green box down the road, but is more likely to be copper all the way to the telco office. They're bundled into bigger and bigger cables (e.g. 24-pair, 50-pair, etc.) There are common-mode failures here - backhoes, wet cables, cars crashing into the telco box - but one of the most common failure modes is "technician mistakes", which usually only take out one wire pair at a time.
At the telco office, your wires get connected to a DSLAM which provides Layer 2 service (DSL is usually ATM underneath.) If both ISPs are using telco DSLAMs, then it'll probably be the same DSLAM box, but if one of your ISPs is using Covad and the other one's using telco, then you're on different DSLAMs. Some DSLAMs have integrated routers, but back when I was working more directly with this stuff there'd typically be an ATM network connecting the DSLAM to some regional concentrator network. The ATM network might have common-mode failures such as port cards, but it's mostly carrier-grade equipment with diverse physical routing.
Eventually you get to a router for Layer 3 service. If your DSL provider uses a telco DSLAM and forces you to use PPPoE, there's a good chance that you're tunneled through a telco router, but eventually you'll hit a router actually managed by your DSL provider. And from there on out to the Internet backbone, everything's basically diverse.
I don't know how Verizon does FIOS - the fiber system's obviously diverse from the copper+DSLAM system, but there might be more common infrastructure upstream or they may use different tools to concentrate it (e.g. FIOS might be using routers while DSL might be on ATM.) If you're using Verizon DSL as opposed to a third-party ISP or an ISP using Covad, you'll probably hit the same Internet peering points, so you could be susceptible to problems like "Cogent decides to have a peering fight with Verizon this time", but on the other hand your ISP might have Verizon as their upstream provider so it's a bit hard to tell. That layer's certainly much more reliable than 10 years ago.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
look for the Linux Advance Routing Howto
Somewhere in that site it talks about some of the problems of having 2 IP addresses, like confusing game servers and the like, but with a bit of tweaking you could get it functional. I don't think this solution explicitly provides failover functionality, but I suppose that could be scripted in somehow.
pfsense is a nice turnkey solution for this too, if you're not into spending a couple weeks solid trying to make your debian or lfs distro act like a router.
db
I am literally 3000 tokens away from the chaotic crossbow --Stephen
To do this properly with load sharing and immediate failover, at the moment the professional solution would be that you should
- get business class connections and
- run BGP over both links.
If you don't already know what BGP is, this solution is probably too complicated for you. Worse, the global BGP routing table is a shared expense, and your extra route would impose a (slight) extra cost on literally every other ISP running BGP. (The business class connections are because you will need several static fully routable IP addresses to do this, plus run BGP, and that requires more than a consumer class connection.)
There is a lot of discussion at the moment about this at the IETF, and people are working on something called LISP (no relation to the computer language), which would provide true multi-homing without the bother of running BGP and adding to the global routing tables. Things like immediate failover and load balancing should follow more or less automatically.
There is a lot more information available at Lisp4.net. I have heard of some initial testing, but in my opinion this is still a ways from commercial use.
Can't load balance hosted services without a remote router? Round robin DNS with short TTLs, with a script to remove an IP if a link goes down.
Outgoing TCP connections are OK when using Linux:
http://lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS
If you buy an off the shelf solution from the likes of F5 there's even more control.
I'm sorry if I haven't offended anyone
OK, so you have two routes to the internet. One packet departs, but is returned by the other route. How to glue those together is a very non-trivial problem.
Sprint tried that in 1997-2001 time frame with bonded T1 & T3 services. The bonding never worked for persistant connections, and only slightly better for transiant connections. UDP worked best. And that was using a routing system that understood it was bonded, not one completely unaware of another route.
These days $DAYJOB uses OC3's and SONET rings for Internet, so there may have been advances I'm unaware of, but back then, it really, really sucked. Off the cuff, I'd say use Linux and the Zebra package on a old computer, and try that, but no promises. Personally, I don't think it will work well.
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
I produce a system that can do this. It's called Broadbond.
You can bond several ADSL lines, even from independent providers, and it will deliver the combined upstream and downstream bandwidth of the two. All traffic is load balanced across the two lines and can also be transparently compressed. The throughput of the lines is automatically measured to determine the optimal load balancing. Differences in latency on the two lines are compensated for.
The catch (there's always a catch!) is that you need to have a partnering system co-located with an ISP to handle the far end of the tunnel -- although I can also provide this if you would prefer.
The system is available as a software package that you can license to run on Linux or OpenBSD and also pre-installed and pre-configured on a couple of small embedded Linux boxes -- very low power (under 5W), no moving parts, good for up to 90Mbit/sec.
I bond two ADSL lines to my office, 4.4Mbit and 9.6Mbit, and I get around 13.5Mbit on file transfers.
If you're interested, contact me (details on the broadbond.org web page).
I have DSL and cable. I also have a D-Link DL604 load balancing router. It sucks.
The router seems to think that as long as the physical ethernet connection is up, the provider is up. It tends not to detect network failure. There are ways to set up a periodic monitor of some host to detect if the network is up, but it does not seem to work properly.
What I want from this thing is:
Lock SMTP to one port and thus one provider. My AT&T DSL SMTP server will not accept mail from my Comcast account. (this is correct behavior for anti-spam). The DL 604 does this correctly.
I want the router to send any new connection for a naive (not currently in routing table) external network to both providers. I want it to measure the response time ( over a number of packets ) and then lock the route to the network which provides the best performance. It can periodically re-test the routes - perhaps every 5 minutes or so. This should address the problem of non-neutral peering between various providers. It is not always true that the higher bandwidth cable connection is the best connection to where I want to go. If I am accessing a client's machine who is on AT&T DSL, my DSL connection may be faster than my cable connection. I want the router to deeply inspect the traffic and be able to detect if a session breaks on a particular WAN port, and try the other. I also want it to quickly recognize when all sessions on a particular WAN port break and switch to the alternate port, while testing the original port.
I want built-in diagnostics that can show me how often a provider drops the ball, shiny graphs of bandwidth and latency etc. It would be cool if the router would allow me to see what the instant connection graph between my LAN and external networks looks like. ( which of my hosts connect to which external domains at the moment ).
I would like to be able to see graphics of IP address / port scans.
I want the router to be able to do some intrusion prevention, particularly if no one is using my network at the moment - someone tries to scan - shut the thing off for a while. ( do I care if I DOS myself if I am not using the net? NO! )
There is a hardware provider http://www.routerboard.com/ that can provide multi-wan multi-lan and wireless router hardware for cheap. They also have software but nothing that does all the tricks I want...
Coders, here's a base spec, send some bits!
OZ
enough is too much
IOS supports unequal cost load balancing with various routing protocols like RIP. You can do per packet or per destination. You can get a used 3640 for fairly cheap and throw in a 4 port ethernet network module and use it as a WAN router. If you needed rendundancy, get a 2nd one and use HSRP. You'd also need at least 2 switches and have a trunk going from switch to switch as well as to both routers. Sounds complex but is really easy to implement with a little bit of networking and IOS knowledge. All the people recommending DSL + Cable are right, DSL + DSL = not redundant.
The Japanese definition of "rural" is nowhere near the definition of rural here in the US. this is because they have an ungodly amount of people for the land they inhabit.
Basically, what I am saying is the Japanese idea of rural is, at best, like a marginally populated suburban neighborhood in the US.
Here are some raw numbers to better illustrate my point (from this study, year 2000 numbers):
Japan total rural area (sq km): 273,646
Japan total rural population: 13,498,527
Japan rural population density (people/sq km): 49.32
US total rural area (sq km): 8,423,867
US total rural population: 54,936,968
US rural population density (people/sq km): 6.52
SEE THE DIFFERENCE? It's almost an order of magnitude! And the urban numebrs show a 3x difference between the US and Japan; closer, but still nowhere near each other.
Of course we have infrastructure problesm here in then US, and they largely don't; it just comes with the territory.
Man is the animal that laughs.
And occasionally whores for Karma.