Slashdot Mirror

← Back to Stories (view on slashdot.org)

Linux and Multiple Internet Uplinks: a New Tool

Posted by Soulskill on from the fault-tolerance-for-all dept.

New submitter Alessandro Zarrilli writes: Linux has been able do multipath routing for a long time: it means being able to have routes with multiple gateways and to use them in a (weighted) round-robin fashion. But Linux is missing a tool to actively monitor the state of internet uplinks and change the routing accordingly. Without it, from a LAN perspective, it's like having a RAID-0: just one uplink goes down and all of your LAN-to-WAN traffic goes down too. Documentation and examples on the subject are lacking; existing solutions are few and deeply integrated in firewall/routing specific distributions. To address these issues, a new standalone tool was just released: Fault Tolerant Router. It also includes a complete (iptables + ip policy routing) configuration generator.

80 comments

  1. Just one question... by Anonymous Coward · · Score: 0

    Is this APK host file compatible?

    1. Re:Just one question... by omnichad · · Score: 1

      As much as it's medium rare steak compatible. They're compatible in the fact that they're completely independent (and unrelated) of one another.

  2. Not RAID-0 by Anonymous Coward · · Score: 0

    RAID 0 has zero redundancy.
    RAID 1 does not fit either as that would just mirror traffic and double costs.

    time to stop using storage words and learn networking.

    Load Balancer

    1. Re:Not RAID-0 by jshazen · · Score: 1

      Actually, the analogy is quite apt.

      As TFS talked about configuring it (load sharing with no monitoring), they would be getting twice the bandwidth, with the same drawbacks as RAID0: If either connection goes down, you have a (mostly) unusable system (because with no monitoring , half your packets are still going out the broken link).

    2. Re:Not RAID-0 by Ultra64 · · Score: 1

      >RAID 0 has zero redundancy.

      That's what he said.

      "just one uplink goes down and all of your LAN-to-WAN traffic goes down too."

  3. overusing "perspective" = douche by Anonymous Coward · · Score: 0

    Like last year's "up-speak", the new linguistic laziness fail is to put "perspective" at the end of everything. It's a clear sign that your speaker doesn't know what he's saying, and is padding his words to fill space and time, waiting for the other party to provide the clarify he lacks.

  4. Strange by ledow · · Score: 5, Interesting

    Strange.

    I was using routing patches to Linux nearly 7 years ago to do this (admittedly it wasn't in the stock kernel, but the patches weren't huge)... you were able to specify multipath and multiple gateways and if one route went down, the others were prioritised and would take over, and also your upstream etc. were balanced properly and took account of failing routes automatically without any kind of daemon etc. running.

    I ran a school off multiple ADSL and even 3G connections with it - the only manual maintenance I ever had to do was to put the ADSL modems onto a SMS-controlled relay (SMS came in on the same 3G stick!) because our ISP would often give us "dead" sessions if they'd had problems (where you'd get PPP and an IP and a remote gateway but couldn't do anything across them) and we were then able to manually reset if necessary. My bursar and I used the system for five years like that, only ever resetting it to enable VPN when all the upstream routes had got dead sessions, and that less than once or twice a year.

    And, no, we didn't have to do much. It was a stock Slackware install with one set of patches to a (2.6?) kernel to enable the multipath routing etc. Pretty well advertised at the time, one plain page of simple patches (I remember porting them myself to a newer kernel version, just before the new diffs came out), I'll try and dig it up.

    And "RAID-0 for upstream"? Bollocks. It "just worked" whatever interfaces were up (proven by it would even include the 3G PPP interface whenever it came up, and that only came up when we manually instructed it to connect as it cost money).

    Not saying this isn't good software, but it's by far not the problem the summary purports it to be, not a first by any means, and certainly not "new".

    1. Re:Strange by ledow · · Score: 4, Informative

      Think it was:

      http://www.ssi.bg/~ja/

      Seems to still be updated.

    2. Re:Strange by Anonymous Coward · · Score: 0

      Dicedot fails it.

    3. Re:Strange by TopherC · · Score: 1

      I wonder how this is different from channel bonding / link aggregation? I looked into this a few months ago and don't remember all the details but there's a "bonding" kernel module, which can run in some modes entirely in kernel space, or in a user-space-assisted mode. There is a round-robin mode but there are several others that include fault tolerance and load balancing. LACP can be used in cooperation with other network elements including switches if you want something that spans a local network.

      I had limited success with this myself, so I wonder what new technology the Fault Tolerant Router brings?

    4. Re:Strange by ledow · · Score: 1

      LACP would, indeed, fulfill the purpose but relies on you being able to obtain LACP support on upstream connections from your ISP. LACP must be enabled and known about on both ends for it to do anything.

      It's not always true that you could get support on upstream connection, but they are many, and multiple, types of bonding that provide similar facilities.

      However, in terms of being able to get disparate connections that can be conjoined without specific support on the other end or high-end hardware, there are fewer - but non-zero - ways of doing that too.

    5. Re:Strange by msauve · · Score: 4, Informative

      Link aggregation works at layer 2 (e.g. Ethernet). Basically, make multiple cables between 2 devices look like a single one.

      Multipath routing works at layer 3 (e.g. IP), you can send outbound packets to multiple routers for further forwarding. It works when there are "real" routes between the hosts (i.e. not behind NAT).

      This is one step beyond that, since it also does connection tracking and will work with outbound port NAT, so you can have a private network connected to multiple ISPs.

      --
      "National Security is the chief cause of national insecurity." - Celine's First Law
    6. Re: Strange by bbn · · Score: 1

      Linux has been able do multipath routing for a long time: it means being able to have routes with multiple gateways and to use them in a (weighted) round-robin fashion. But Linux is missing a tool to actively monitor the state of internet uplinks and change the routing accordingly

      No the author is lacking knowledge of such tools. There are at least three major implementations of common routing protocols for Linux.

      It is actually surprisingly simple. You activate OSPF on your Linux box and it will add routes on working links and remove them again if the path is no longer viable. Works both ways too, so your routers will not try to deliver traffic to your box on a dead link. Instead the routers will route around the problem.

    7. Re:Strange by hwk_br · · Score: 2

      The site is now at http://ja.ssi.bg/ and it works very well, specially with dozens of users. The "nano.txt" file explains it all, with examples.

      --
      \m/
    8. Re: Strange by Macfox · · Score: 1

      From reading the details on the site, it's more sophisticated. It interacts with iptables, which traditional routing protocols do not.

      --
      Area51 - We are watching...
    9. Re:Strange by Alessandro+Zarrilli · · Score: 1

      If I'm not wrong, ja kernel patches detect a dead route by pinging the interface gateway. In my experience, in my environment (Italy), this is not enough to detect a dead gateway: we often get dead DSL links or very disturbed radio links. In both cases the interface gateway is up and running, so the uplink would not be detected as down. And this unfortunately doesn't happen here "less than once or twice a year": we don't have a very nice internet infrastructure here in Italy, in some rural places it can happen once a month. About the RAID-0 analogy, it was just to let people better understand using a concept they are familiar with: I'm confident many Slashdot readers know what a RAID-0 is. And I still think the RAID-0 analogy is pertinent. If you don't have a mechanism to exclude dead uplinks, it's like losing a disk in a RAID-0 array: all your data is lost! In this case (almost) all of your LAN-to-WAN traffic is down. Actually you had a mechanism to exclude dead uplinks: ja kernel patches, they worked in your environment but, as I said before, not in mine.

    10. Re:Strange by Alessandro+Zarrilli · · Score: 1

      Channel bonding is a different thing, but you need upstream support to let it work. It's for example what these guys are doing (letting you connect to their server on the cloud): http://speedify.com/

    11. Re: Strange by Alessandro+Zarrilli · · Score: 1

      Does OSPF work also with cheap DSL routers Telecom Italia or Vodafone provide us? I doubt it. Unfortunately we're not in Silicon Valley here... ;-)

    12. Re: Strange by Alessandro+Zarrilli · · Score: 1

      Exactly! I wanted to create a well documented and fully functional system targeted to the average Linux guy, not to a network specialist. It cares about everything: creating iptables configuration, setting ip policy routing rules, monitoring the uplinks, warning the administrator.

  5. Ruby?? by Schlopper · · Score: 4, Insightful

    I do not want to install Ruby on my firewall/gateway along with all of its douchebaggy dependencies and gems/crystals/unicorns/whatever-the-fuck-they're-called. This is networking, not some hipster web 2.0 app.

    1. Re:Ruby?? by Anonymous Coward · · Score: 0

      It is web scale. Don't you want to be web scale? What is wrong with you? /s

    2. Re:Ruby?? by Anonymous Coward · · Score: 0

      I didnt look, but wow. oh how standards have fallen

    3. Re:Ruby?? by John+Bokma · · Score: 4, Informative

      Ruby is a scripting language like Perl or Python. Rails is the "hipster web 2.0 app", which is a library (framework) for Ruby.

      --

      Perl Programmer for hire
    4. Re:Ruby?? by Hulfs · · Score: 3, Informative

      It's really a glorified bash script..and it doesn't actually require any ruby gems to run.

      It's just pinging a configurable IP to test specific outbound connections and when an interface goes up or down it resets the routing table...nothing especially fancy.

    5. Re:Ruby?? by marcello_dl · · Score: 1

      Then, mr hardtoplease, wait for the systemd based implementation, it is right around the corner (holding a knife).

      --
      ---- MISSING MISCELLANEOUS DATA SEGMENT --- [sigdash] trolololol
    6. Re:Ruby?? by Anonymous Coward · · Score: 2, Informative

      It's really a glorified bash script..and it doesn't actually require any ruby gems to run.

      https://github.com/drsound/fault_tolerant_router/blob/master/fault_tolerant_router.gemspec

      At this time it installs at least one gem, and it looks like it's for sending mail (notifications). And that gem might have dependencies of its own.

      Given time, and enough feature creep, this project very wall may require more gems down the line.

    7. Re:Ruby?? by Hulfs · · Score: 1

      At this time it installs at least one gem, and it looks like it's for sending mail (notifications). And that gem might have dependencies of its own.

      Given time, and enough feature creep, this project very wall may require more gems down the line.

      D'oh...You're right..I scanned the gemspec and missed the runtime one below the dev ones.

    8. Re:Ruby?? by The+Raven · · Score: 1

      Ruby is an excellent scripting language, equivalent to Perl or Python... better in some ways, worse in others. Nobody bats an eye if something requires Perl, even though CPAN Gems are essentially equivalent.

      Ruby != Rails.

      --
      "I will trust Google to 'do no evil' until the founders no longer run it." Hello Alphabet.
    9. Re:Ruby?? by Alessandro+Zarrilli · · Score: 1

      It's not aimed at embedded systems, so dependencies are not a problem for me. Then, installing dependencies with bundler it's a joke. It's clear you're not a Ruby fan! ;-) I agree it would have been better to write it in C, but I'm a little rusty on C, today I almost develop everything on Ruby, so it was a natural choice for me.

    10. Re:Ruby?? by Alessandro+Zarrilli · · Score: 1

      Yes, you could write it in Bash. I extensively used Bash to do shell scripting in the past, but it doesn't give me the same "programming pleasure" as Ruby does. Personal tastes.

  6. remote video streaming by bigmo · · Score: 1

    I am unclear if this would work for a "single stream" like a video webcast source in a remote location using multiple cellular links like jetpacks. I can easily see how any given network request would go to one link or another depending on availability, but I'm assuming that this would not be able to take a network video stream and parse out portions of it to the various uplinks and then at the far end put the stream back together (in order to get either extra bandwidth or extra reliability) to forward on to a server. Teradek and other companies have specific hardware & software to do this. Can this fault tolerant router be made to do something like this, either with or without additional software?

    1. Re:remote video streaming by Anonymous Coward · · Score: 0

      Depends on your transport protocol for the stream. UDP (most common) and TCP based streams, probably not, unless you were only multihoming over a VPN link and presenting a single IP at some remote end location. An SCTP stream could do it as it supports multihoming natively. That's just pretty rare. I think WebRTC is somewhat based on SCTP but without the multihoming capabilities yet. But if it get's that then yeah, multipath video streams galore!

    2. Re:remote video streaming by swb · · Score: 1

      I think the logic for streaming would just be too complicated over multiple links. You might make it work if you had a proxy that knew about the links and had some way to choose paths, but you'd still have to work out the application layer bit to negotiate the paths with the streaming source so it could send them down multiple paths and then there would need to be some kind of ordering and link balancing on top of it.

      I think the only way to make it work with a stream is something like you've outlined -- an MPPP session over the individual links to a remote MPPP server whose WAN side is is the IP initiating traffic. MPPP should be able to handle the link balancing and stream order.

    3. Re:remote video streaming by Alessandro+Zarrilli · · Score: 1

      I would use OpenVPN. One VPN going from each uplink to your data center. Then use channel bonding on the OpenVPN interfaces on both sides: remote location and data center. Of course you would need a modified version of Fault Tolerant Router to detect if an uplink goes down, otherwise your "channel bonded" connection would go down too...

  7. marking packets by Mirar · · Score: 1

    I would be happy enough if I figured out how to mark the packets so that my routing actually works. I already have a shell script to switch connection from fiber to 3g, but incoming connections only work on one at a time. However, I'm not willing to run Ruby on my router to solve that issue...

    1. Re:marking packets by Anonymous Coward · · Score: 0

      Linux is open source. Write your own kernel module for packet handling needs.

    2. Re: marking packets by Anonymous Coward · · Score: 0

      not everyone who uses linux is a developer, prick.

    3. Re:marking packets by Alessandro+Zarrilli · · Score: 1

      No, you needn't Ruby on your router if a working packet marking solution is all you need. Just run Fault Tolerant Router in demo mode on... Windows, if you wish! Grab the command output and use the same commands on your router, using Bash or whatever you like.

  8. Zebra by BitZtream · · Score: 1

    That's one of the points of zebra and the suite of tools that it brings to the table.

    The kernel shouldn't do more than it already does unless you want to move the kernel into systemd as well.

    http://en.wikipedia.org/wiki/G...

    Zebra is not, by any means, new.

    --
    Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
    1. Re:Zebra by BitZtream · · Score: 1

      Oh hell, this story is nothing more than a slashvertisement for some guys new pet project, the first commit is barely 2 weeks old.

      Could you people use google and a cloupon before you reinvent the wheel using a shitty scripting language, please.

      --
      Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
    2. Re:Zebra by BarbaraHudson · · Score: 1

      It would be nice if we could get rid of stories like this and all those stupid kickstarter "gimme money to build this cool new thing" ads.

      --
      "Transparent" is a shit show that trades on every stereotype going. A man in drag is NOT a transsexual.
    3. Re:Zebra by mattventura · · Score: 1

      unless you want to move the kernel into systemd as well.

      Shhhhh...you're giving them ideas.

    4. Re:Zebra by Alessandro+Zarrilli · · Score: 1

      First commit is 2 weeks old because until then the program was internally developed and used, as stated in the README. 2 weeks ago it was put on GitHub and released as GPL.

  9. Done in bash since the 2000s by broknstrngz · · Score: 1

    But hey, it hasn't been on Github until now. Why the hell is this news?

    1. Re:Done in bash since the 2000s by Anonymous Coward · · Score: 0

      If it has a github and title is catchy, it's gotta be special and you better be special enough to appreciate. Even if it's bloated hipster ruby.

  10. OpenWRT with mwan3 by AlreadyStarted · · Score: 5, Informative

    OpenWRT package mwan3 has similar functionality without the complication of multipath.

    http://wiki.openwrt.org/doc/ho...

  11. Re:Is it in systemd yet? by Przemo-c · · Score: 1, Insightful

    Oh just please SHUT THE HELL UP. I'm no fan of systemd and i wont use it for foreseeable future (i will actively avoid it for now) But stop injecting this trollish behavior. It adds nothing and convinces noone.

  12. Not to lose my karma by Yew2 · · Score: 3, Insightful

    but as a network engineer, this snippet is painful to read. Inaccurate to say the very least.

    --
    will work for dragon quest localization
  13. Many other tools for multipath by klapaucjusz · · Score: 5, Informative

    There's a lot of multipath-related work being done right now, at the IETF, within OpenWRT, and independently.

    We've been working on providing multiple routes automatically (disclaimer -- I'm a co-author). As to actually making use of the multiple routes, the solution that currently works best is MP-TCP, a set of kernel patches that allows TCP to use multiple routes simultaneously, with no modification to applications. Other solutions are SHIM6, which works below the transport layer, and Multipath Mosh, which works at the application layer.

    I'm pretty confident we'll be able to have most of this stuff enabled by default in mainstream Linux distributions by the end of the year.

    1. Re:Many other tools for multipath by Anonymous Coward · · Score: 0

      I really, really love source specific babels - as many gateways as I want, falling over automatically.

      and the multipath mosh stuff works great on the rare cases when fq_codel has a hash collision and on any other reason you are having odd rtts on one path or another. I would like way more apps to gain this multipathing ability!

    2. Re:Many other tools for multipath by Zarhan · · Score: 1

      Well, this example is just a simple load balancing with outbound NAT and nothing as fancy as the stuff you cited which aims for e.g. multipathing single sessions like MPTCP does.

      Been working on that same area.

    3. Re:Many other tools for multipath by Anonymous Coward · · Score: 0

      For thoses interested in fast portable multipath vpn checkout MLVPN.

  14. Re:Is it in systemd yet? by Anonymous Coward · · Score: 0

    Yes, I think systemd will consume this functionality by end of 2015.

  15. Have you heard of routing protocols? by Anonymous Coward · · Score: 0

    Have you heard of Quagga? It's a fork of GNU Zebra. It's a suite of Linux software that manages routing with protocols like RIP1&2, EIGRP, OSPF, BGP...

    But, I'll warn you now, you're much better off running a dedicated routing box with a ditribution like VyOS a fork of Vyatta, rather than using a desktop system for a router.

    What I'm trying to say is ha this article is a waste of time, even to n00bs advertisement peddlers. Also, Soulskill doesn't have a clue what routing is.

    1. Re:Have you heard of routing protocols? by chuckinator · · Score: 1

      Indeed, I have many times run a linux router by doing nothing but installing quagga, Net-SNMP, ipt_netflow, and I've got a lower end equivalent to some of the highest end commercial networking equipment.

      Both OSPF and BGP provide the tools you need for policy based routing to various degrees, and quagga gives you that. The rest is just icing on the cake.

  16. gwping by ManiaX+Killerian · · Score: 3, Interesting

    There is a small shell script called gwping, which can be used to do the exactly same thing, easier and simpler. It's ~150 lines (with comments and everything) and takes 10-20 minutes to setup with the policy routing and everything, we don't need an overbloated runtime to do something so simple.

  17. Uhm, hello. LSM. by Anonymous Coward · · Score: 0

    http://lsm.foobar.fi/

  18. Re:Is it in systemd yet? by aardvarkjoe · · Score: 1

    I recommend you follow your own advice. Stop feeding the trolls.

    --

    How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
  19. Re:Is it in systemd yet? by JustOK · · Score: 1

    i think emacs has a macro for that

    --
    rewriting history since 2109
  20. streaming schmeaming by Anonymous Coward · · Score: 0

    I am unclear if this would work for a "single stream" like a video webcast source in a remote location using multiple cellular links like jetpacks.

    You don't do that. But consider how it might work with Popcorn Time.

    And once you think about Popcorn Time, if you just think of torrents as a way of downloading rather than "streaming" then you've got a really popular use case covered. And then that use case will remind you of others, which also typically involve making multiple connections (e.g. sabnzbdplus).

    So while the technique might not combine well with "webcast" it's nevertheless very useful for video in general.

  21. BGP? by Anonymous Coward · · Score: 0

    Isn't this what BGP is for? Why bother with this?

    1. Re:BGP? by Anonymous Coward · · Score: 0

      Your wireless provider gives you BGP access to your random IP address?

  22. Not needed by Anonymous Coward · · Score: 0

    I have a very short script in /sbin/ifup-local that simply adjusts the default route(s) according to which interfaces are up.

    Pity people dreaming up so of these elaborate "solutions" appear to be unaware of what can already be done without an elaborate solution.

  23. There is a script for that... by v3xt0r · · Score: 1

    dualgate_multinet.sh supports dual-gateways and multiple subnet vlans.

    https://github.com/sodonnell/b...

    This script can easily be extended to support more than (2) gateways, and can support various VLAN/subnet configurations and isolation.

    --
    the only permanence in existence, is the impermanence of existence.
  24. Done in 2009 already ? by Anonymous Coward · · Score: 0

    http://rodent.za.net/files/ratroute/

    Still works on most debian systems without having to install Ruby ;)

  25. Shorewall and LSM have offered this for years by WuphonsReach · · Score: 1

    We've had a functionally equivalent capability with Shorewall + LSM (Link Status Monitor) for years now. Setup (2) ISP connections, route a proportion of your traffic to each one and when the link goes down, all traffic goes to the other link.

    The hard part of the equation is your public DNS records that need to change to whichever IP address is "active" (or round-robin between the two). But most DNS service providers have a solution for that as well.

    Naturally, it's not as seamless as multi-path, but it works without needing a BGP entry or your own IP address block in the public routing tables.

    --
    Wolde you bothe eate your cake, and have your cake?
  26. A solution to a down interface by laing · · Score: 1
    High Availability (HA) is not an option for many Linux users (or to me for my home systems) so I use iproute2 (which is built into all common Linux distributions). With a few simple rules, one can make outbound traffic go out on the interface that it is associated with. For example: I could have multiple DNS A records for a host (using either single or multiple network interfaces) and have that host respond to client requests via the same interface on which they arrive.

    Iproute2 has worked out very well for me for a quite a long time and I have no need to run any additional routing daemons.

  27. Re:Ruby?? and Unicorn by Anonymous Coward · · Score: 0

    Dunno if you were employing hyperbole or being serious, but Unicorn is a real name of a server used when running Ruby apps...

  28. Network monitoring by Anonymous Coward · · Score: 0

    I'm looking for something basically along these lines, something that monitors which hosts are up and which are down. I don't need to know much about routes, but I can't find a damned thing about them on the net. What makes it harder to find is that a search for "network host status monitor" comes up blank.

    I've seen something along those lines on the walls in a few large IT places. Anybody know the name of the things I'm after? Just want one for a small network.

  29. Ruby... by Bert64 · · Score: 1

    Pity it's written in ruby, that's a bit too heavy for most small routers...

    --
    http://spamdecoy.net - free throwaway anonymous email - avoid spam!
  30. Peplink anyone? by Anonymous Coward · · Score: 0

    They been doing this on hardware for a long time.