Load Balancing Using Multiple PPP Links?

M@W asks: "I currently run a Linux box (Slackware) in the office to handle e-mail, Web serving, SQL serving, dialup, print and fileserving, and it is connected netwise via a permanent modem. We're shortly opening another office a few kilometres away, and will need network connectivity. Since ISDN is so costly, I was planning on running two or more 33.6k modems at each site. To distribute the network traffic properly between the links do I need to be running something like EQL or EQL-plus? Since both ends will be our private networks, can I just assign IP's to all modems and set up the routing tables to use all links? Has anyone else done something like this? (Another option would be to use cable modems and a Virtual Private Network link, as they're only $59AUD per month (flat rate), but they don't allocate static IP's)"

Comments

Multilink PPP and/or BGP?

[Vito]

Coincidentally enough, I'm actually looking to do this with 2+ analog cell phones with modems attached, so I can transmit a live voice Ogg Vorbis stream in the field to a relay station at high quality. ;)

How about setting up Multilink PPP? That's what it's designed for. Have two or more modems at your new office dial into your current office using the MP patch to pppd, and poof, instant channel bonding type stuff.

The other suggestion someone had (on the linux-router mailing list, actually) was to use BGP to split up your data between two or more links (multi-homing). I don't know how that works, though.

--Vito

Re:Multilink PPP and/or BGP?

[justin42]

BGP is way too complex too use.

Re:Multilink PPP and/or BGP?

[h2odragon]

The "MP patch to pppd" linked above works; with a bit of effort, I had a 2.2.15 version of it running last week... and then discovered that it wasn't going to be useful for what I needed it for. USR Courier ISDN modems don't do channel bonding without MPPP, unlike the motorola's used to.

If you use the "multipath routing" feature of 2.2.x kernels, you can set up two or more routes to an address, and they'll all be used for outgoing traffic. There's some magic so that a single TCP connection only uses one link during its lifetime, but for most purposes it should work well.

The EQL driver is odd, I've never used it, but it's my understanding that it only works between linux and linux or a Lucent Portmaster 2e (not 3+). There's been a "True Link Equalizer" since 2.2.9 or so, I think, but that's about all I know about it; whether it's functional, how it's used etc I have no clues about.

Re:Multilink PPP and/or BGP?

[anticypher]

You have several options.

If you are going with multiple analog modem connections, look into MultiLink PPP, RFC1717. There are a number of PPP implementations that offer it. I've seen it on some linux boxes, and I'm sure I've used it on an OpenBSD box once.

If both of the sites are served off of the same central office, get a pair of DSL modems, and you should be able to get around 512kbps between the two sites. The DSL modems should not set you back more than the 4 or 6 analog modems you would have bought anyways.

Then ask the phone company to install "an unloaded dry copper pair" to be connected between the sites.

If you tell them it is for data lines, you'll pay more for higher speed lines. If you say you are using it for an alarm system, that is probably the cheapest tariff you can pay, but be aware that if the phone company has no sense of humour and finds you shipping high speed data across it, they'll either bill you or cut your service.

And stay away from BGP, it can do load balancing, but by the time you learn enough to implement it correctly you'll be too valuable to stay in your current job :-)

the AC

DSL

[justin42]

DSL, if available, might be a good option.

Both sites could have DSL and run VPN software. It would be plenty fast, seeing that both lines will be locally routed at the CO. Both sites would also have internet access.

A slightly cheaper solution is to get an untapped line from the phone company between the two sites, and attach DSL modems to that. They would then be directly connected (security and speed reasons)

Re:DSL

[lingenfr]

Pertaining to the last section, when we order them from the phone company, we are looking for 'dry pairs'. The significance is that there is end to end connectivity without going through a CO. We used such a solution with 56K frame relay. Worked like a champ for $50 a month. If Internet access for the second office is also an issue, I concurr with my learned contemporary and say that DSL or otherwise the cable modem combined with a VPN is a good choice.

Having tried multilink, I was disappointed and thought that the pain to gain ratio was a bit too high.

I don't think that BGP will give the results that you are looking for in terms of aggregating bandwidth. It would facilitate fail over. But... I could be wrong. Cheers.

Re:DSL

[Robert S Gormley]

DSL is not available (for a few more months) in Australia. :)

But yes, would be nice ;)

Solve your static IP Problem

[Moeses]

You could use a service like http://www.dynip.com to give you a URL that always points to your dynamic IP address. Then you could use the cable modem option.

RFC1149

[YASD]

Well, if it's only a few km away, you could try the link proposed in RFC1149.

------

Load Balancing Options

[PatJensen]

You have a few different options for load balancing multiple connections on an IP network. You can use multilink PPP, or eql which is the Linux kernel load balancing device or you can use routing options such as multiple default routes or BGP.

Using BGP or OSPF or multiple default routes would only give you the bandwidth of up to 1 of your interfaces, and would spray packets out of each interface in a 1-goes-here 1-goes-there manner. This probably wouldn't be what you are looking for voice traffic. Using Linux eql you can setup bonding to aggregate the speed of all connected modems so that with 2 28.8k modems, you would get a totally of 5-6k/s on your transfers. This is done at the interface level versus the network routing level and sounds like it would be right up your alley.

Re:Load Balancing Options

[pleitner]

Using BGP or OSPF or multiple default routes would only give you the bandwidth of up to 1 of your interfaces, and would spray packets out of each interface in a 1-goes-here 1-goes-there manner.

Not quite correct. There is the concept of an mroute cache that alleviates this problem. Basically if you route a packet out over link0 and another packet comes along it first looks up the mroute cache. If it finds a match (which it will in this case) it will send it out along the same path as the previous packet. If it doesn't find a match, then it will pick either of the links as you suggest.

This probably wouldn't be what you are looking for voice traffic.

This is not what you are looking for with any traffic. The mroute cache described above is used to assist the receiving end so that it is less likely to receive packets out of order and stop problems such as weirdo retransmission problems due to one link being significantly slower than the other (assuming multihoming). In general mroute cache is a good thing.

Also, the only real reason you would use BGP for something like this is if you were multihomed (it would be an absolute joke trying to receive any kind of routing table over modem links anyway). Multilink PPP (on slow links) to the same upstream is best taken care of with static routes.

Multilink PPP

[zanzar]

I know you can do this with pppd under FreeBSD, and I bet you can do it under linux, as well. With FreeBSD, at least, a fairly simple set of options in /etc/ppp/ppp.conf will setup multilink ppp. Any number of connections could be shared this way, so if you wanted to have three or four modems on each end being split equally, it would work as easily as two. You shouldn't have to worry about routes, etc., any more than you do with PPP in a normal scenario.

If you haven't given FreeBSD a chance, yet, you're cheating yourself. It can do just about everything that linux can do, but usually it's easier to setup. :)

Re:Multilink PPP

[adolf]

Alternatively, there's a program called mpd which does the multilink PPP thing fairly well, along with demand-dialing of links. It can be found in the FreeBSD ports collection.

I've been using it with a pair of 28.8 modems on a slow 8-meg 486 for a month or so, since I've moved away from civilization (well, close enough - I had 128k ISDN). My only complaint is that the thresholds for bringing up/tearing down connections are somewhat inflexible, and the default modem scripts are the horribly complex, all-things-to-all-people type that even AOL doesn't use.

If rubber bandwidth/line usage is not a concern, the standard userland PPP daemon (ppp) will probably work fine, and much more easily.

For all-in-one hardware solutions, as others have said, DSL modems and a dry pair would probably be optimal.

Or, VersaNet makes a box which contains two or four 56k modems and some routing hardware (http://www.versa-net.com), which also has some niceties like STAC compression that just don't exist in the world of free software. A pair of these, connecting to eachother at 4x33.6k would approach the speed of 2B ISDN, which isn't bad.

3com makes a similar beast as well, but having had insurmountable trouble with their ISDN OfficeConnect product and returning it at a loss, I can't recommend any of 3com/USR's small dialup routers.

In today's world? RFC2459

[hodeleri]

QOS is a must! RFC 2459 - IP Over Avian Carriers with Quality of Service

--
Eric is chisled like a Greek Godess

ppp not pppd

[hodeleri]

pppd is the kernel-land ppp interface, the user-land version is ppp, which uses the tun interface (and the /etc/ppp/* files)

ppp under FreeBSD has an excellent manual page and is very easy to configure.

--
Eric is chisled like a Greek Godess

sch_teql

[twl]

the true link equalisation module does work, but each destination address has to have its own netcard (ie physical device). perfect if you've got, say, 2 dsl lines and you want to provide both fault-tolerance and load-balancing of outbound traffic.

Static IP's

[rips]

To use EQL each modem must have the *same* IP and the remote site must also support EQL (such as Livingston's Portmaster series).

Equal-cost multipath routing (is that the right term?) will work on outgoing data but obviously you don't have access to your ISP's incoming data routing.

You're probabily best to talk witk some prospective ISP's before slashdot. It all depends on the equipment they use.