Affordably Aggregating ISP Connections?

An anonymous reader writes "Has anyone setup a system to aggregate multiple ISP connections to form a high bandwidth site-to-site link? Load Sharing SCTP looked interesting, but it doesn't look like it has been widely adopted. Multi-Link PPP appears to be more widely supported for clients, but I can't find any good guides for setting up both sides of the connection for a site-to-site link. The hardware solutions I've found are expensive for a small business. Does anyone have experience using hardware solutions from Mushroom Networks (Virtual Leased Line, p2 of this document), Ecessa (site-to-Site Channel Bonding), or others?"

Comments

Re:Peering

[Annymouse Cowherd]

I think that the poster was intending to agreggate a cable, DSL, and satellite link to make a more reliable connection, not get multiple ISPs on one line.

Bonded VPNs

[Anonymous Coward]

I have bonded 2 IPSec VPNs running over 2 ISP's to create a bigger (and cheaper) site-to-site link on the cheap.

http://www.zeroshell.net/eng/faq/vpn/

Read Point 5 in the link

What are you really trying to do?

[BadAnalogyGuy]

What you have presented us with here is a "B C" problem. You want to achieve C, so you ask us how to do B. Unfortunately, you never specify what A is, so the best we can do is give you some pointers for B which are probably going to be irrelevant and useless to what you are really trying to achieve.

Most of the comments will probably be about trying to figure out what your A problem is. To that end, why don't you just get a faster line in the first place and forget about this line aggregation stuff you're asking about?

Re:What are you really trying to do?

[KingSkippus]

Unfortunately, you never specify what A is...

As the other poster noted, it's not always easy to just add more bandwidth. Where I live, the absolute fastest DSL line I can get is 1.5 Mbps. Fortunately, my cable company offers faster options, up to 22 Mbps. If they didn't, I'd be screwed if I actually wanted a decent connection relatively cheaply.

Also, one nice thing about having multiple links over multiple ISPs multiplexed together is that you have redundant links. If one ISP is having problems, you still have some bandwidth, which is generally better than no bandwidth at all. I'm assuming that the submitter would like features like automatic failover, so that if one link goes down, all of the traffic will defer to the other link until it comes back up.

Where I work, we have this type of setup with most of our big plant sites, although with the bandwidth we're talking about, it's definitely high-end business-class (read: expensive as hell) service, and because we don't want sites to become dependent on the higher bandwidth, we leave our secondary circuits idle unless needed. It would be nice for there to be a solution that offers the higher bandwidth and redundancy of a multiplexed connection cheaper than it would cost to only be available to huge MNCs.

Re:What are you really trying to do?

[TheRaven64]

But that's the grandparent's point. What does he actually want from this setup? There are a lot of factors that will affect the best solution. Does he want:

• Individual connections to be faster?
• Total throughput to be more?
• Overall reliability to be better?
• Transparent fail-over if one connection goes down?

If he wants the last one, does he want:

• Existing connections to continue working?
• New incoming connections to keep working?
• New outgoing connections to keep working?

Some of these are trivial, some require a little bit of client-side configuration, some require additional support from the ISP. Without knowing what he actually wants to achieve, it's impossible to make a recommendation. You can do all of these things relatively easily with a stock OpenBSD install on your router, but exactly which ones you want depends a lot more on the requirements. For somethings, you want to run a VPN between the two sites with packets sent over some of the link with the most bandwidth. For others, you could get away with just a couple of routing rules. If you want more than just the two sites and you want existing connections to work then you need the ISP to support updating the routing tables when their link to you goes down.

Have experience with Ecessa...

[Anonymous Coward]

We've been using 2 Powerlinks from Ecessa (back when they were Astrocom). They work really well, and the price is tough to beat. We have one in our Dallas branch (with a T1 and business cable ISP) and one at our home office in Baton Rouge (a dual bonded T1 and business cable). They are channel bonded with each other, so the site-to-site VPN is more stable. They made my life a lot easier!

Re:Don't be so cheap

[adolf]

All of them?

Um, yeah: Whatever you say, kid.

We usually just use a Roadrunner account for the main office, just like all the other small business out there. It's faster and cheaper than a T1, and has better reliability than the PRI that handles our phones. (We also have a freebie account with a local WISP that we do some business with for manual fail-over, but we haven't had to use it in years.)

tomato

[angelbunny]

The cheapest way to do this is use the mlppp version of tomato on a wrt type router. You can check it out here: http://fixppp.org/

Re:tomato

[Anonymous Coward]

Only works if you've got DSL, and then again only if they use PPPoE. Then the remote DSLAM needs to support mlppp as well.

I would suggest OpenBSD + PF and just load balance the connections using PF. Takes all of 10 lines of code to get it up and going and is well documented. This won't aggregate your bandwidth, however if you have multiple streams open, it'll bounce those between two or more connections. I've personally done 4 lines like this (2x adsl2+ and 2x DOCSIS 2) and hit about 95% utilization across all lines.

Also with PF, both lines don't need to be the same speed, or even with the same provider, which gives you some additional fault tolerance.

pfSense

[adairw]

Unless you can get your ISP to bond several connections together about the best you can do is load balancing across multiple connections. I use pfsense (http://www.pfsense.com) as my router/firewall VPN solution that's free, you only supply the hardware to run it on. with it you can load balance and fail over to 2 or more connections automatically. Specif connections can even be setup to have certain traffic routed over them while all other traffic gets load balanced round robin style. there are of course other free *nix distros out there that will let you do the same type of stuff however I and many others have found pfSense to be far batter than most. AW

Re:pfSense

[angelbunny]

I use pfSense too for my multi wan needs and it really is a wonderful distro imho.

However, there is a difference between grabbing a bunch of wans and throwing them together vs linking them together like one giant pipe. I'm not completely sure what the author is trying to do but if this person wants the multi lane freeway approach instead of multiple separate parallel roads than pfSense does not currently support protocols such as mlppp and may not be what they are looking for.

However, if the person wants to hack it a bit then theoretically it could be done since pfSense is built on freebsd (i believe) which does support mlppp. However, one might ask why do that when you could install freebsd on the box instead of pfSense and do the same thing with the same amount of work?

Talari Networks?

[Anonymous Coward]

Have you looked at what Talari Networks (http://talari.com/) is doing? I'm pretty sure their products do EXACTLY what you're talking about. Might be pricy for you, but it should do the trick.

printers/newpapers

[shareme]

Your local newspaper or medium sized printer will have such a setup. Buy their IT staff diner to get the information..

Re:printers/newpapers

[bendodge]

Even in really depressed metros, diners are still pretty expensive compared to T1's.

Multiple bonded connections

[davidwr]

In theory, you can bond multiple DSL, multiple cable, multiple T1, or even multiple dialup connections from the same vendor.

Even if you are in a small town where the best service you can get is 1Mbps DSL, if you've got enough wires running from the neighborhood box to your house you can ask for 2 or 3 or more separate DSL lines and see if the local telco will support bonding them.

Even 15 years ago some telcos offered on-demand, 0-24 circuit, bonded dialup. The idea was a business would use it as up to 24 voice circuits during times of the day they talked a lot and up to 24 modem/data circuits when they needed them, typically at night for batch data exchange. It was sold as an alternative to T1 or ISDN, the first of which was very expensive and not available in all areas, and the latter of which was expensive and roughly the equivalent of 2 phone-or-data lines.

DSL and later cable internet made this pretty much obsolete, at least in technically advanced areas.

Re:You can't do what you want to do

[Anonymous Coward]

Sure you can.

http://www.openbsd.org/faq/pf/pools.html

One simple example. Plenty of other options available with other software. As long as you load-balance per connection instead of per packet there aren't many issues with this, and those often don't apply outside of special use cases.

Dreytek

[Rainwulf]

The higher end dreytek business modems support at least two aggregate DSL links. The real question is, do you want a wider pipe, or a faster pipe. One is easy, the other not so easy. Bigger trucks in your tubes, or faster trucks in your tubes :) (sorry couldnt resist that analogy)

Re:You can't do what you want to do

[hcetSJ]

a fucking book on how routing works

Now there's a fetish you'll only run across on Slashdot.

Re:You can't do what you want to do

[Penguinoflight]

It is possible as long as you have control of both endpoints. The routing book is probably still a good idea.

Ask Willie Nelson, he uses Mushroom Portabella

[Anonymous Coward]

Wired has an article on Willie Nelson's setup in his tour bus running, http://www.wired.com/epicenter/2009/10/willie-nelson-broadban/ "Willie Nelson has tossed the satellite dish off the back of his corn-powered tour bus in favor of a little box that fuses wireless data cards from a variety of networks into a single connection."[Mushroom Networks PortaBella 141]

Need More Infos

[LoudMusic]

Sounds like you're trying to take a DSL, cable, and possibly a T1 or other technology and trunk them for combined throughput. That isn't possible because you'd have packets in the same stream taking different routes and TCP/IP doesn't allow for that, that I know of. I don't think any technology allows for that. For example an 8mbit DSL, 6mbit cable, and a T1 can't be combined to make a 15.5mbit connection. I suppose the same would be true if you were wirelessly connected to multiple networks.

You can, however, use all three gateways independently with a variety of load balancing software so that when a new request is made from any host it is routed through the gateway with both the quickest response time and the most bandwidth available. I'll let you look that up on your own - there are lots of free options. The problem is that the load balancer needs to be smart enough to not fuck up your active sessions. If you were communicating with a host via one route, went idle for a bit but didn't end the session, then sent more data via another route the host on the other end will most likely (if written correctly) not accept your new packets.

The way we handled it at "The Geek House" with three internet gateways was to just permanently assign gateways based on the role of the host, and made sure not too many were on the slower gateways. It's not perfect, and certainly could have been geekier, but it worked and we didn't have to worry about shit breaking in the middle of a frag fest. And if one gateway was down the hosts configured with that gateway just had to change their gateway.

Re:Need More Infos

[mindstrm]

"TCP/IP doesn't allow for that, that I know of"

It sure does - it doesn't care what route the packets took - just that they got there. THe problem is if you split the stream over 3 links with varying latency - you won't see the performance gains you wan t- it'll more likely hurt.

If the goal is to end up with a virtual point-to-point link between two offices using multiple ISPs, you can certainly leverage multiple connections to do that. It also depends on the nature of the traffic.

You can set up multiple VPN tunnels and then run whatever protocol you want - you could do MLPPP - but that'll get ugly if the links don't have very similar characteristics.

The solution you mentioend in the end - Iv'e found that' susually the best - you can get most common *nix systems to do some kind of weighted load balancing of outgoing sessions... whether it's per-source, per-destination, per-protcol, or based on any other weird usage combination you had.

For an office situation Iw as once in - we had 1 2mbps and 1 x 4mbps lines (from separate providers) and a very high latency 1Mbps satellite connection.
I gave them a web page that had four buttons on it.
The first was "normal operation - 2MB + 4 MB". TCP sessions would be randomly routed over one orhte other, with double rpeference given to the 4 meg line.
The ohters were "ISP1, ISP2, and Satellite" respectively. At the push of a button the routes would flip, the state tables would flush, and everything would work. For practical puproess, it worked really well.

There is no magic way to simply aggregate bandwidth from separate providers over consumer connections with meaningful results... not like bonding multiple direct lines or anything like that.... 2 + 2 won't equal 4.... but depending on the use case, it can be just about as good.

Only Half a dozen BSD and Linux Appliances...

[thatkid_2002]

I'm not one to yell at noobs but I really can't imagine timothy did more than a Bing search because I see that pfSense comes up on the first page of results on Google when you query "multi wan".

PfSense is probably the go for this, but you are free to choose any other BSD or Linux based distro which gives you a nice pretty point and click web interface out of the box and good online documentation on how to use the features.

Hell, you don't even actually need physical hardware for this provided that you have two NICs available and a virtualization capable server.

Mac OS X?

[appleguru]

Admittedly, I have no idea if it works, nor do I have any idea how it decides to load balance between the connections.. But I ran across the feature the other day and it looked pretty cool.

In Mac OS X you can create a new "Aggregate" network device from any other devices and, in theory, do exactly what your describing. Again, I just ran across this the other day in Network Preferences and have no idea if/how it works, but it might be worth a shot (especially since it seems a lot easier to configure than a roll your own router with dd-wrt or tomato, though those likely offer more fine-tuned configuration).

Re:You can't do what you want to do

[Jah-Wren Ryel]

Wow. I'm not the AC but after that response I fully agree with him.
Your use of selective quoting is amazing, you got some big-ass internet cojones to ignore the rest of the very same sentence that you quoted.

...to form a high bandwidth site-to-site link.

LISP Routing

[paul248]

Some people (Cisco, etc.) are working on developing the Locator/ID Separation Protocol as a core component of the Internet infrastructure.

If that ever takes off, you'll be able to buy a Provider Independent IP address block, advertise it through multiple ISPs (even Cable/DSL), and transparently load balance your upstream and downstream traffic across them, without bloating the core BGP tables.

The downside is, you'll have to use an MTU that's smaller than 1500, but I'd say it's a fair trade.

Advanced Routing Howto

[flyingfsck]

The Advanced Routing Howto on tldp.org - nuf sed.

Re:Peering

[hardwarefreak]

I know linksys has a couple routers (both the RV042 and RV082) that supports 2 incoming broadband connections with link aggregation (or it can use it as failover) if you used two of these and set up a VPN it would be fairly cheap/easy (under $500 easy) I just looked on their site but since the Linksys business stuff is now buried in Cisco's crappy site, i was unable to find a link. I've seen them at Fry's plenty of times. I've used several of them and they tend to be fairly stable.

I looked into the RV082 a while ago and found that you can get reasonably close to doubling your _outbound_ bandwidth, but not inbound. Bonding the inbound links would require both WAN lines be provided by the same ISP, so they could configure round robin across your two links.

The RV082 is a great little SOHO router and does pretty good load balancing/aggregation of outbound traffic. The OP seems to be looking for true bi-directional link aggregation of dissimilar ISP WAN links (cable/dsl or two of one of these from different providers). This is simply not possible, because there must be intelligence on the other end of your links round robin'ing the traffic between them, just like your RV082 is doing in this case.

In short, this is a great inexpensive product to double your outbound and provide redundancy. Keep in mind you'll need to do some creative things in DNS and with port forwarding on the Linky as you'll have two different public IPs on those WAN links. WRT hosting a mail server, you'll need two MX and A records, one for each public IP on each WAN link. You'll also need duplicate records for all your servers, whether WWW, ftp, etc.

Setting up _inbound_ redundancy is not simply clicking a radio button as with outbound redundancy. Remote hosts have to be told how to reach you. This means advertising both routes. Since you aren't paying an ISP for this redundancy, and you're doing it on the cheap yourself, you'll have to mangle DNS to get the inbound redundancy.

If you're looking for merely link aggregated high bandwidth site-to-site, I'm not sure if this Linky will do so with the VPN feature. You can sure try it. You can also use the little brother RV042 for a little less money, although neither is terribly expensive.

Re:Peering

[unity]

I've been using the hotbrick LB-2 for years to aggregate dsl and cable lines. Works like a charm.
http://www.hotbrick.com/produto.asp?tipo=3&catpro=2

I thought they had up to a 4connection version, but I don't see it anymore.

Re:Pering

[lamapper]

Its long, at least read about Greenlight in N.C. and learn!

I am 100% positive you could do this with hardware that will run the DD-WRT, here is a list of DD-WRT supported devices, they have a search link, but I find that it does not work very well if you do not know the name of the router / firewall that you are looking for. So use the list and find a supported device.

You would need two of them and two different providers. You could even get a third one and do some special VLAN stuff to put some ports on all three on the same virtual network., many options.

These devices are very light weight, therefore shipping is next to nothing. The Linksys WRT54Gs' were great routers for the DD-WRT software. Costing over $75 when they first came out, dropping to $69 for years and finally hitting $15 or $30 when the stores were unloading them to bring in the new Linksys routers (none of which will support the DD-WRT software, except one that runs Linux). NOTE: there are BETTER routers than the WRT54G to run this software. The WRT54G will ONLY run the Micro version of the software. Do yourself a favor and get one that will run the Mega version of the software! (They cost less than $100 per and well worth the price.)

Linksys (Cisco) begin removing DD-WRT compatible firewall/routers from store shelves, replacing them with devices that are NOT compatible with the DD-WRT software in 2007/2008.

Get two DSL lines ($13 - $19 each), add in a NAT and a couple of these routers, probably need to do some secure tunneling to avoid the DNS of the Cable / DSL Companies and voila you are good to go. Your DSL speed will vary based on distance, but even far away you can get 1.5MB down and 384Kbps up. If closer you can get 3Mb down and 768Kbps up. (That is faster than 98% of Americans with Cable Modems because of throttling of service by Cable providers.)

Could you run the second DSL upstream over the first one? Thus saving the cost of a second telephone line, you would lose the redundancy that two telephones would provide, but save around $13 per month on a second phone line...probably better just to get the two lines, you total cost of ownership (TCO) will still be less than $60 per month and you will have redundancy. If one service gets stupid and starts throttling, drop them and get a different one. Politicians help us if they all throttle!

Solves allot of problems related to Cable companies throttling back service if you can create a secure VPN that their Deep Packet Inspection and/or Bandwidth shaping (throttling) service might have a harder time restricting (throttling). Granted they would still throttle you back by your IP address or MAC address of Cable Modem. Again, they do that now anyway.

A friend of mine was pissed that he was throttled back to less than 100K down and 0K up 85 - 95% of the time. He went on and paid his cable company the $10 burst / protection racket money / "give me a little more of what I am already paying for money" extra fee. Keep in mind that they were promising up to 8MP and delivering less from day one. He said he got a letter in the mail that they would be rolling out a new service in his area, the day after they started using that service, his bandwidth was throttled to next to nothing. (0 Kbps upstream, consistently less than 20Kbps). (There were 1 GB, 2GB and 3 GB ~ 1 second spikes ONLY, unless he was downloading a Linux distro, then he got 3GB - 4GB sustained with a 1 sec 6GB spike) He is convinced that they throttle him back because he uses Skype VoIP service (uses P2P packets) in a vain attempt to get him to switch to the Cable companies VoIP service. At less than $100 per year, Skype blows away any telco/Cable company offering.

Guess what his speed was after the switch over....Yep less than 100K (down) and 40K upstream 95% of the time. When he is throttled back to 0Kbps like I am, t

Linux Iproute2 is all you need

[SectoidRandom]

A few years back I did this with a colleague, we actually investigated 3 solutions; 2 commercial and one linux script based, in the end the one that won easily was the Linux script.

Basically using iproute2 and some nice scripts gives you the ability to load balance your outbound packets and then using some relatively simple scripts to monitor each remote peer for automatic failover.

A quick google turns up this blogger who sounds (from a quick skim) like he's doing the same thing: http://blog.taragana.com/index.php/archive/how-to-load-balancing-failover-with-dual-multi-wan-adsl-cable-connections-on-linux/

Unfortunately I can't remember the commercial solutions we tested (this was 4-5 years ago!), but although they did exactly what you wanted perfectly, our problem was that we were doing this for a managed services company who ran 150+ IPSEC VPN's over those (at the time) 3 bonded ADSL connections, needless to say the commercial solutions had never imagined anyone trying to statefully balance that many VPNs! However with some tweaking (to be honest a LOT of tweaking) we got the Linux solution working a treat, even with nearly seamless failover.

Google is your friend on this one.

Re:You can't do what you want to do

[Jah-Wren Ryel]

What difference does that make?

It makes all the difference in the world. All you need is the appropriate device at each site - not at the ISP. Set up a VPN tunnel across the multiple links that terminates at the other site and you can aggregate at the packet level just like any of the site-to-ISP aggregation methods. The only case where the ISP has to support link aggregation is where it is site-to-internet-at-large, not site-to-site.

If so, the internet cojones apparently don't require intelligence.

Considering that it now appears you've been proclamating without investigating, it is quite appropriate that you would say that.

Re:You can't do what you want to do

[Jah-Wren Ryel]

An ISP provides a connection to the internet, by defintion. So, "site-to-internet-at-large" is what was the topic of discussion.

That's some funny ass shit dude.

The OP said site-to-site link and you think he meant not site-to-site link!

You crack me up. Are you stoned or just high on your ego defense mechanism?
Been fighting for peace too?
Fucking for virginity maybe?

Re:You can't do what you want to do

[Jah-Wren Ryel]

What's funny is how you keep ignoring the original premise and want to infer based on subsequent statements

Subsequent statements in the same sentence that serve to clarify his intent.

You just keep right on denying the obvious dude, safe and warm in your little house of meaningless semantics

Packet reordering / VPN Bonding

[scamp]

Obviously direct aggregation isn't possible, as each line will have a different source IP. What works is load balancing, but load balancing sucks. If you do per-TCP-connection load balancing on multiple lines, lots of sites will give you problems, as multiple requests for the same session are coming from different IPs. Online banking doesn't like this, ads-supported sites often don't like this (as the ad was loaded from a different IP). So this leaves you with per source-host load-balancing, and this only makes sense if there are lots of people who are two share the lines.

Doing real aggregation (bonding) requires a remote endpoint obviously, located in a datacenter somewhere for example. Problem: There is no standard protocol that works for a combination of different lines, Multilink-PPP will only work for several identical lines from the same ISP (ideally using the same clock source at the DSLAM etc). Why is that? That's because if you use multiple lines, they will have different latencies / round trip times. And if you bundle those, this means that TCP packets will overtake each other in-flight. So in the end whoever is receiving the re-assembled stream will get it out of order. And TCP can not differ between reordered and lost packet - if an unexpected (too high sequence number) packet is received, it is dropped. And this can not be solved by buffering at the router/PPP-device, because this buffering would interference with TCP windowing. In the end most of your aggregated bandwidth will therefore be eaten by retransmissions.

So, people may tell you to try this and that, but in the end everyone who has ever REALLY tried it himself will tell you: Forget about it, the performance will always be really bad (unless you have multiple identical lines).

There is a small german startup I work for which has solved the problem by creating a new bundled VPN protocol running on the way between the router in your office and the one in the datacenter, basically running a man-in-the-middle attack on TCP to get rid of the packet reordering in-flight. See http://www.viprinet.com/ for the available products and background info on how it works. Pricing starts at ~1000 USD, but obviously you'll need two boxes - probably not what you'd call "affordable". And sadly we do not yet have distributors inside the USA.

Re:You can't do what you want to do

[Jah-Wren Ryel]

Your meaningless semantics really are meaningless - they certainly aren't details that make a difference to solving the actual problem as stated.

As someone who has done precisely what the guy asked for, as previously described with a VPN, this 'not a tech' laughs at your continued denial of the obvious.

PS, this "not a tech" has 20+ years of tcp/ip stack and other misc internals experience, he knows exactly what he's talking about.
By your own demonstration in this thread, you don't.