Affordably Aggregating ISP Connections?

An anonymous reader writes "Has anyone setup a system to aggregate multiple ISP connections to form a high bandwidth site-to-site link? Load Sharing SCTP looked interesting, but it doesn't look like it has been widely adopted. Multi-Link PPP appears to be more widely supported for clients, but I can't find any good guides for setting up both sides of the connection for a site-to-site link. The hardware solutions I've found are expensive for a small business. Does anyone have experience using hardware solutions from Mushroom Networks (Virtual Leased Line, p2 of this document), Ecessa (site-to-Site Channel Bonding), or others?"

Comments

Bonded VPNs

[Anonymous Coward]

I have bonded 2 IPSec VPNs running over 2 ISP's to create a bigger (and cheaper) site-to-site link on the cheap.

http://www.zeroshell.net/eng/faq/vpn/

Read Point 5 in the link

Have experience with Ecessa...

[Anonymous Coward]

We've been using 2 Powerlinks from Ecessa (back when they were Astrocom). They work really well, and the price is tough to beat. We have one in our Dallas branch (with a T1 and business cable ISP) and one at our home office in Baton Rouge (a dual bonded T1 and business cable). They are channel bonded with each other, so the site-to-site VPN is more stable. They made my life a lot easier!

tomato

[angelbunny]

The cheapest way to do this is use the mlppp version of tomato on a wrt type router. You can check it out here: http://fixppp.org/

Re:tomato

[Anonymous Coward]

Only works if you've got DSL, and then again only if they use PPPoE. Then the remote DSLAM needs to support mlppp as well.

I would suggest OpenBSD + PF and just load balance the connections using PF. Takes all of 10 lines of code to get it up and going and is well documented. This won't aggregate your bandwidth, however if you have multiple streams open, it'll bounce those between two or more connections. I've personally done 4 lines like this (2x adsl2+ and 2x DOCSIS 2) and hit about 95% utilization across all lines.

Also with PF, both lines don't need to be the same speed, or even with the same provider, which gives you some additional fault tolerance.

Re:What are you really trying to do?

[Anonymous Coward]

To that end, why don't you just get a faster line in the first place and forget about this line aggregation stuff you're asking about?

A lot of people don't realize that in many places in the US "getting a faster line" just isn't an option. When you get out of the large metro areas, the connection options start going down considerably until you may be left with satellite as your main option and iffy terrestrial wireless (or, gasp, dialup) as your backup options. When that happens, I don't blame anyone for trying to tech the tech for better bandwidth.

Talari Networks?

[Anonymous Coward]

Have you looked at what Talari Networks (http://talari.com/) is doing? I'm pretty sure their products do EXACTLY what you're talking about. Might be pricy for you, but it should do the trick.

printers/newpapers

[shareme]

Your local newspaper or medium sized printer will have such a setup. Buy their IT staff diner to get the information..

Multiple bonded connections

[davidwr]

In theory, you can bond multiple DSL, multiple cable, multiple T1, or even multiple dialup connections from the same vendor.

Even if you are in a small town where the best service you can get is 1Mbps DSL, if you've got enough wires running from the neighborhood box to your house you can ask for 2 or 3 or more separate DSL lines and see if the local telco will support bonding them.

Even 15 years ago some telcos offered on-demand, 0-24 circuit, bonded dialup. The idea was a business would use it as up to 24 voice circuits during times of the day they talked a lot and up to 24 modem/data circuits when they needed them, typically at night for batch data exchange. It was sold as an alternative to T1 or ISDN, the first of which was very expensive and not available in all areas, and the latter of which was expensive and roughly the equivalent of 2 phone-or-data lines.

DSL and later cable internet made this pretty much obsolete, at least in technically advanced areas.

Re:You can't do what you want to do

[Anonymous Coward]

Sure you can.

http://www.openbsd.org/faq/pf/pools.html

One simple example. Plenty of other options available with other software. As long as you load-balance per connection instead of per packet there aren't many issues with this, and those often don't apply outside of special use cases.

Only Half a dozen BSD and Linux Appliances...

[thatkid_2002]

I'm not one to yell at noobs but I really can't imagine timothy did more than a Bing search because I see that pfSense comes up on the first page of results on Google when you query "multi wan".

PfSense is probably the go for this, but you are free to choose any other BSD or Linux based distro which gives you a nice pretty point and click web interface out of the box and good online documentation on how to use the features.

Hell, you don't even actually need physical hardware for this provided that you have two NICs available and a virtualization capable server.

Re:Need More Infos

[mindstrm]

"TCP/IP doesn't allow for that, that I know of"

It sure does - it doesn't care what route the packets took - just that they got there. THe problem is if you split the stream over 3 links with varying latency - you won't see the performance gains you wan t- it'll more likely hurt.

If the goal is to end up with a virtual point-to-point link between two offices using multiple ISPs, you can certainly leverage multiple connections to do that. It also depends on the nature of the traffic.

You can set up multiple VPN tunnels and then run whatever protocol you want - you could do MLPPP - but that'll get ugly if the links don't have very similar characteristics.

The solution you mentioend in the end - Iv'e found that' susually the best - you can get most common *nix systems to do some kind of weighted load balancing of outgoing sessions... whether it's per-source, per-destination, per-protcol, or based on any other weird usage combination you had.

For an office situation Iw as once in - we had 1 2mbps and 1 x 4mbps lines (from separate providers) and a very high latency 1Mbps satellite connection.
I gave them a web page that had four buttons on it.
The first was "normal operation - 2MB + 4 MB". TCP sessions would be randomly routed over one orhte other, with double rpeference given to the 4 meg line.
The ohters were "ISP1, ISP2, and Satellite" respectively. At the push of a button the routes would flip, the state tables would flush, and everything would work. For practical puproess, it worked really well.

There is no magic way to simply aggregate bandwidth from separate providers over consumer connections with meaningful results... not like bonding multiple direct lines or anything like that.... 2 + 2 won't equal 4.... but depending on the use case, it can be just about as good.

Re:You can't do what you want to do

[Jah-Wren Ryel]

Wow. I'm not the AC but after that response I fully agree with him.
Your use of selective quoting is amazing, you got some big-ass internet cojones to ignore the rest of the very same sentence that you quoted.

...to form a high bandwidth site-to-site link.

Advanced Routing Howto

[flyingfsck]

The Advanced Routing Howto on tldp.org - nuf sed.

Re:Peering

[unity]

I've been using the hotbrick LB-2 for years to aggregate dsl and cable lines. Works like a charm.
http://www.hotbrick.com/produto.asp?tipo=3&catpro=2

I thought they had up to a 4connection version, but I don't see it anymore.

Linux Iproute2 is all you need

[SectoidRandom]

A few years back I did this with a colleague, we actually investigated 3 solutions; 2 commercial and one linux script based, in the end the one that won easily was the Linux script.

Basically using iproute2 and some nice scripts gives you the ability to load balance your outbound packets and then using some relatively simple scripts to monitor each remote peer for automatic failover.

A quick google turns up this blogger who sounds (from a quick skim) like he's doing the same thing: http://blog.taragana.com/index.php/archive/how-to-load-balancing-failover-with-dual-multi-wan-adsl-cable-connections-on-linux/

Unfortunately I can't remember the commercial solutions we tested (this was 4-5 years ago!), but although they did exactly what you wanted perfectly, our problem was that we were doing this for a managed services company who ran 150+ IPSEC VPN's over those (at the time) 3 bonded ADSL connections, needless to say the commercial solutions had never imagined anyone trying to statefully balance that many VPNs! However with some tweaking (to be honest a LOT of tweaking) we got the Linux solution working a treat, even with nearly seamless failover.

Google is your friend on this one.

Re:What are you really trying to do?

[TheRaven64]

But that's the grandparent's point. What does he actually want from this setup? There are a lot of factors that will affect the best solution. Does he want:

• Individual connections to be faster?
• Total throughput to be more?
• Overall reliability to be better?
• Transparent fail-over if one connection goes down?

If he wants the last one, does he want:

• Existing connections to continue working?
• New incoming connections to keep working?
• New outgoing connections to keep working?

Some of these are trivial, some require a little bit of client-side configuration, some require additional support from the ISP. Without knowing what he actually wants to achieve, it's impossible to make a recommendation. You can do all of these things relatively easily with a stock OpenBSD install on your router, but exactly which ones you want depends a lot more on the requirements. For somethings, you want to run a VPN between the two sites with packets sent over some of the link with the most bandwidth. For others, you could get away with just a couple of routing rules. If you want more than just the two sites and you want existing connections to work then you need the ISP to support updating the routing tables when their link to you goes down.

Re:You can't do what you want to do

[Jah-Wren Ryel]

What difference does that make?

It makes all the difference in the world. All you need is the appropriate device at each site - not at the ISP. Set up a VPN tunnel across the multiple links that terminates at the other site and you can aggregate at the packet level just like any of the site-to-ISP aggregation methods. The only case where the ISP has to support link aggregation is where it is site-to-internet-at-large, not site-to-site.

If so, the internet cojones apparently don't require intelligence.

Considering that it now appears you've been proclamating without investigating, it is quite appropriate that you would say that.

Re:You can't do what you want to do

[Jah-Wren Ryel]

An ISP provides a connection to the internet, by defintion. So, "site-to-internet-at-large" is what was the topic of discussion.

That's some funny ass shit dude.

The OP said site-to-site link and you think he meant not site-to-site link!

You crack me up. Are you stoned or just high on your ego defense mechanism?
Been fighting for peace too?
Fucking for virginity maybe?

Re:You can't do what you want to do

[Jah-Wren Ryel]

What's funny is how you keep ignoring the original premise and want to infer based on subsequent statements

Subsequent statements in the same sentence that serve to clarify his intent.

You just keep right on denying the obvious dude, safe and warm in your little house of meaningless semantics

Re:You can't do what you want to do

[Jah-Wren Ryel]

Your meaningless semantics really are meaningless - they certainly aren't details that make a difference to solving the actual problem as stated.

As someone who has done precisely what the guy asked for, as previously described with a VPN, this 'not a tech' laughs at your continued denial of the obvious.

PS, this "not a tech" has 20+ years of tcp/ip stack and other misc internals experience, he knows exactly what he's talking about.
By your own demonstration in this thread, you don't.