Slashdot Mirror

← Back to Stories (view on slashdot.org)

A 50 Gbps Connection With Multipath TCP

Posted by timothy on from the meanwhile-my-att-dsl-crawls-along-poorly dept.

First time accepted submitter Olivier Bonaventure writes "The TCP protocol is closely coupled with the underlying IP protocol. Once a TCP connection has been established through one IP address, the other packets of the connection must be sent from this address. This makes mobility and load balancing difficult. Multipath TCP is a new extension that solves these old problems by decoupling TCP from the underlying IP. A Multipath TCP connection can send packets over several interfaces/addresses simultaneously while remaining backward compatible with existing TCP applications. Multipath TCP has several use cases, including smartphones that can use both WiFi and 3G, or servers that can pool multiple high-speed interfaces. Christoph Paasch, Gregory Detal and their colleagues who develop the implementation of Multipath TCP in the Linux kernel have achieved 50 Gbps for a single TCP connection [note: link has source code and technical details] by pooling together six 10 Gbps interfaces."

104 of 150 comments (clear)

  1. what's happening with SCTP? by Monkius · · Score: 1, Insightful

    Doesn't SCTP provide for these scenarios (and many more)?

    --
    Matt
    1. Re:what's happening with SCTP? by swillden · · Score: 5, Informative

      Doesn't SCTP provide for these scenarios (and many more)?

      No.

      SCTP supports multiple paths between endpoints, but doesn't use them simultaneously. Rather, it picks a primary path to use for data transfers and has the ability to fail over to an alternate path in the event the primary fails.

      A quick glance at the MTCP RFC shows that it is essentially multiplexing packets over n separate TCP streams (called subflows). It's the responsibility of the TCP/IP stack (in the OS, generally) to make this multiplexing transparent to the application, so the application only sees one stream.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    2. Re:what's happening with SCTP? by c0lo · · Score: 4, Informative
      In my understanding (I might be wrong):

      1. SCTP - identified by a protocol number (132) - acts at the network layer. If a router along the route refuses SCTP, you are screwed; Advantage: is capable of UDP as well).

      2. MPTCP - relies on pure TCP for all the connection (acts at the transport layer and fixes the protocol to TCP) and set in place conventions between client-server to discuss over multiple paths. Advantage: no sane public network will try to block it (pretty much like using http on port 80). Disadvantage: TCP only.

      --
      Questions raise, answers kill. Raise questions to stay alive.
    3. Re:what's happening with SCTP? by c0lo · · Score: 1

      Errata: not acts on but relies on/uses.

      --
      Questions raise, answers kill. Raise questions to stay alive.
    4. Re:what's happening with SCTP? by butlerm · · Score: 4, Informative

      On the contrary, SCTP is a transport protocol just like TCP, except with a large number of added features. The main problem with SCTP has nothing to do with SCTP at all. It is that NAT devices do not support any transport protocol that they haven't been programmed for in advance. This makes SCTP next to impossible to deploy on a broad scale - NAT, that wart upon router-kind, is ubiquitous.

      TCP would have exactly the same problem if it were a new protocol. A NAT device requires relatively deep knowledge of TCP to support it at all. It play games with both ports and addresses, keeps track of connection state, and so on. Ordinary routers do no such thing. A NAT device is a transport layer proxy by another name.

    5. Re:what's happening with SCTP? by butlerm · · Score: 5, Informative

      Work is underway for concurrent multipath transfer for SCTP as well. Also known as CMT-SCTP. There are significant challenges in doing this sort of thing though. SCTP wasn't designed for CMT, and probably needs much more radical changes than the current architects are proposing to do it well.

      Changes like subflows with independent sequence numbers and congestion windows, to start with. SCTP is much further ahead in the connection handling and security department, but MPTCP has the odd advantage of resorting to independent subflows to begin with, and if it can handle path failure properly, it might well be ahead in the CMT game, if byte stream semantics are all you need.

    6. Re:what's happening with SCTP? by olivier.bonaventure · · Score: 4, Informative

      SCTP is cleaner than Multipath TCP, but it suffers from two drawbacks that hinder its deployment in today's Internet : - many middleboxes only support IP, ICMP and TCP and discard SCTP packets (or do not perform NAT correctly) - applications need to be modified to support SCTP Multipath TCP is an evolution to TCP that works with unmodified applications and unmodified middleboxes.

    7. Re:what's happening with SCTP? by FireFury03 · · Score: 1

      1. SCTP - identified by a protocol number (132) - acts at the network layer. If a router along the route refuses SCTP, you are screwed

      If your ISP is refusing to route protocols other than TCP/UDP/ICMP its time to change ISP - this will equally break GRE, AH, ESP, 6in4 and a whole host of other commonly used protocols.

      The other thing SCTP has going for it is that its packet based instead of stream based, which makes it more suitable for almost everything we currently use tcp for...

      --
      http://blog.nexusuk.org
    8. Re:what's happening with SCTP? by fa2k · · Score: 3, Insightful

      Your comment is correct, but NAT is not the core problem. In a world without NAT people would still use stateful firewalls. Those firewalls should be configured to drop anything unknown, because as a principle whitelisting is better than blacklisting.

    9. Re:what's happening with SCTP? by gmack · · Score: 1

      It isn't routers you need to worry about its home gateways and firewalls. From my experiments with it so far my ISP DSL modem doesnt allow it but my office firewall supports it fine (mainly because its Linux) but the CISCO ASA we have server side supports it poorly. With the ASA, it is all or nothing. I can allow all SCTP regardless of port or I can allow no SCTP.

    10. Re:what's happening with SCTP? by swillden · · Score: 1

      Very informative. Thank you.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    11. Re:what's happening with SCTP? by butlerm · · Score: 1

      A stateful firewall doesn't need to block transport layer protocols it doesn't understand in order to provide a meaningful level of security. All it needs to do is block packets from IP addresses that corresponding interior address has not recently communicated with, with a reasonable time out. UDP is handled much the same way today.

      If the developers of stateful IPv6 firewalls do not ship devices with such a reasonable configuration by default, they will block the deployment of new transport protocols indefinitely - at least all those that do not resort to the awkward expedient of running on top of UDP.

      Blocking new transport protocols developers can reasonably handle with a standard policy is bad for efficiency, power consumption, latency, user experience, and so on in the long run - TCP is far from ideal as a transport protocol goes. In a number of ways it is outright backwards. If you want to impede the long term development of the Internet, degrading the end-to-end principle unnecessarily is a good place to start.

    12. Re:what's happening with SCTP? by FireFury03 · · Score: 1

      On the contrary, SCTP is a transport protocol just like TCP, except with a large number of added features. The main problem with SCTP has nothing to do with SCTP at all. It is that NAT devices do not support any transport protocol that they haven't been programmed for in advance. This makes SCTP next to impossible to deploy on a broad scale - NAT, that wart upon router-kind, is ubiquitous.

      On top of that, SCTP also won't take off in the IPv6 world (where NAT isn't an issue), purely because Microsoft are on record saying they're never going to support it.

      Which is a shame, because SCTP is actually a much better fit than TCP for almost all applications that currently use TCP, by virtue of the fact that it is packet based instead of stream based. Also, the multi-streaming would be very beneficial for HTTP, which is something that HTTP/2.0 is instead trying to bodge ontop of TCP.

      --
      http://blog.nexusuk.org
  2. Request For Comments by Nethead · · Score: 4, Informative

    RFC 6182 if anyone is interested.

    --
    -- I have a private email server in my basement.
    1. Re:Request For Comments by dreamchaser · · Score: 3, Insightful

      The first part I read when I heard of this was the security concerns. While there's been a good attempt to address them I am not 100% sold. I guess the proof will be in the pudding as the old saying goes. Anytime you make a new protocol, especially one that is more complex, you run the risk of increased vulnerability.

    2. Re:Request For Comments by swillden · · Score: 5, Informative

      RFC 6182 if anyone is interested.

      I think RFC 6824, linked in the summary, is the more relevant RFC.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    3. Re:Request For Comments by swillden · · Score: 4, Interesting

      What sort of security concerns are you thinking of?

      An attacker who controls one of the paths can obviously modify, replace, delay or delete portions of the stream which are multiplexed onto that path. Such an attacker could probably perform a DoS that would shut down the entire stream (disclaimer: I haven't read the details). But of course ordinary TCP is subject to all the same attacks, if the attacker has control of the path that carries it. In many cases an attacker without control of the path can also execute DoS attacks against TCP (e.g. sending RSTs).

      I'm not saying there aren't any new vulnerabilities exposed, but I'm not seeing where they would lie. TCP is not secure in any useful sense, so it's hard to see how MTCP could be worse.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    4. Re:Request For Comments by dreamchaser · · Score: 1

      Oh I didn't say it would be worse, but the more complex any kind of system gets the more uncertainty is introduced. I certainly wasn't suggesting we not evolve protocols to tackle problems as technology advances. I'm just always cautious about security since IT security is my profession. The RFC I linked to is an interesting read about some of the security concerns we need to keep in mind as this gets implemented.

    5. Re:Request For Comments by Nethead · · Score: 1

      I missed that one. You sir, are correct!

      Also the same authors.

      --
      -- I have a private email server in my basement.
    6. Re:Request For Comments by Nethead · · Score: 1

      I would think that since the data is on multiple and diverse wires it would make it harder to tap. Still security through obscurity but every bit helps.

      --
      -- I have a private email server in my basement.
    7. Re:Request For Comments by LordLimecat · · Score: 1

      Security concerns may or may not be relevant. A lot of places have trivial security on their iSCSI between SAN and server, because the security is applied at other levels (segregated switches / airgap, physical security).

      I can think of a number of uses (SAN-server connections where you need more than gigabit) where security is irrelevant.

    8. Re:Request For Comments by Anonymous Coward · · Score: 1

      What I didn't see addressed in the RFC was the possibility that a connection established over a relatively secure channel (such as a VPN or leased line) might then find and use additional paths over less secure channels, such as the public Internet.

    9. Re:Request For Comments by fleisher · · Score: 3, Informative

      The old saying is, "The proof of the pudding is in the eating," not "The proof is in the pudding."

      --
      Max
    10. Re:Request For Comments by camperdave · · Score: 1

      It's like frequency hopping. The attacker will only have part of the message.

      --
      When our name is on the back of your car, we're behind you all the way!
    11. Re:Request For Comments by camperdave · · Score: 1

      The old saying is, "The proof of the pudding is in the eating," not "The proof is in the pudding."

      Exactly! The OLD saying was "The proof of the pudding is in the eating,"
      The NEW saying is "The proof is in the pudding."

      Language changes - deal.

      --
      When our name is on the back of your car, we're behind you all the way!
    12. Re: Request For Comments by Hal_Porter · · Score: 1

      You could imagine doing RAID like striping of the data so that if some was lost you could recover. And/or encryption actually. Plus you can always stop using a path if it is doing something evil with the data. Its easy to imagine multi path TCP being more secure than single path TCP. Also imagine a case where you have multiple, untrusted Wifi networks to leach off - with multi path TCP you don't have to worry about them dropping or snooping packets.

      --
      echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
    13. Re:Request For Comments by HeronBlademaster · · Score: 1

      "The proof is in the pudding" wins by virtue of the fact that it conjures up more amusing images.

    14. Re:Request For Comments by 0ld_d0g · · Score: 1

      What specific security concerns do you have that don't also apply to existing protocols?

    15. Re:Request For Comments by swillden · · Score: 1

      It's like frequency hopping. The attacker will only have part of the message.

      Right, but that would be a strength, not a vulnerability.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    16. Re:Request For Comments by jones_supa · · Score: 1

      Language changes - deal.

      Ahh, so more junk in the steaming pile of "I could care less" and "queue in the musicians".

    17. Re:Request For Comments by Anonymous Coward · · Score: 1

      What I didn't see addressed in the RFC was the possibility that a connection established over a relatively secure channel (such as a VPN or leased line) might then find and use additional paths over less secure channels, such as the public Internet.

      This is why I stay far away from the IETF and committies. People like yourself thinking they are most clever wasting everyones time on utter nonsense.

  3. cell networks already have issues by Osgeld · · Score: 1

    without every user making 3 connects to view their friends cat picture.

    yes, this could have some good uses, no, dont release it to the drooling masses so 12 year old Tiffany can stream Justin Bieber videos while texting her buddy sitting right next to her "faster".

    1. Re:cell networks already have issues by ebno-10db · · Score: 3

      Sheesh, you wanna put even more people out of work? More cell bandwidth needed? Ok, more base stations, new and improved protocols, new frequency allocations, etc. etc., etc. As someone who once made a living working on cellular (phy layer) stuff, I say 12 year old Tiffany has both a Constitutional and a God given right to stream Justin Bieber videos while texting her buddy sitting right next to her. I'll even write the manifesto!

      More seriously, a lot of what we take for granted started out as frivolous luxuries. I tell my daughter about days before cell phones, or PC's, and having seven channels of broadcast TV (and having to get up to change the channel!) and she's convinced I come from the age of dinosaurs. She's probably right. That was good, because I made a living changing it.

    2. Re:cell networks already have issues by c0lo · · Score: 4, Informative

      without every user making 3 connects to view their friends cat picture.

      Rest assured: there'll be a single connection using a cell tower. A second flow will be made using the connection with nearby WiFI hot-spot, and Tiffany's chatting to her buddy sitting next to her will be really faster (without quotes); even better, the above will happen without Tiffany knowing or the extra requirement for Tiffany to have a geek father that's not lazy and does have spare time (even if one may wonder what to what good being a geek will be in the future).

      --
      Questions raise, answers kill. Raise questions to stay alive.
    3. Re:cell networks already have issues by Opportunist · · Score: 1

      If speed is what you are after, the drooling masses is what you have to bet on. Nobody will invest the money needed to give you faster access to your Linux repository, but the ability to sell Joe Washup some porn or Tiffany Teenygirl some videos of the heartthrob du jour, now we're talking.

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
    4. Re:cell networks already have issues by Osgeld · · Score: 1

      probably one who is offended by the idea of a big dongle enough to get a couple real dorks fired

  4. Don't even! by Impy+the+Impiuos+Imp · · Score: 2

    I remember getting dual-channel ISDN, which was 128k, but it was split into two 56k data channels and a 16k control channel. You could never download from any one site faster than 56k because a connection couldn't straddle more than one data channel.

    Still, I could play EQ and surf at the same time on a different computer, a novel thing you young punks take for granted get off my lawn!

    --
    (-1: Post disagrees with my already-settled worldview) is not a valid mod option.
    1. Re:Don't even! by BitZtream · · Score: 3, Informative

      Wow, sucks to be you. ISDN channel bonding was well known and I personally used it to achieve higher speeds than you could on a single channel even over a single TCP connection. The bonding had nothing to do with the modem/circuit actually and in reality was just a standard feature of the PPP protocol called multilink-PPP. You can still do the exact same thing today with multiple connections and pretty much any PPP client on ANY OS on the planet.

      Of course, ISDN is actually 2 64kbps data channels and a 16kbps control channels, as it was meant to carry 2 voice channels, which by standard, are 64kbps data channels, so I'm guessing you really don't know that much about it in general.

      --
      Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
    2. Re:Don't even! by Opportunist · · Score: 1

      Your phone company could handle something as fancy and high-tech as ISDN with no need to assemble every tech in the house to prop up the line? Get offa MY lawn!

      --
      We used to have a Bill of Rights. Now, with the rights gone, all we have left is the bill.
  5. Bad math? by Anonymous Coward · · Score: 1

    Wouldn't six 10 gig connections add up to one 60 gig connection instead of one 50 gig connection?

    1. Re:Bad math? by Zapotek · · Score: 2

      I assume 10Gbps were eaten by protocol overhead and arbitrary resource restrictions. Perfect distribution/load-balancing is seldom the case in the real world and this does seem like quite an achievement, all things considered. Easy link aggregation at the protocol level, a big thank you to the devs. :)

    2. Re:Bad math? by olivier.bonaventure · · Score: 4, Informative

      The limit here is the CPU and on the sender and the receiver. Both servers used in the test reached 98% CPU load to achieve 52 Gbps. Note that 52 Gbps is the googput at the application and not the bandwidth used on the links (which is higher due to the various overheads)

  6. API support by AveryRegier · · Score: 2

    One of the barriers to this technology will be API support. Many APIs provide the IP address (on both sides) with the connection object. Implementors will have to make a choice about which ip to expose and remain backward compatible.

    1. Re:API support by Anonymous Coward · · Score: 1

      No. MTCP is transparent to the application. You can use a normal socket API.

    2. Re:API support by c0lo · · Score: 3, Informative
      In my understanding, this will still rely on multiple IP addresses (not using a single IP address for all the network connections). The difference: it will ride on top of multiple TCP/IP connections - assuming they are available - to multiplex their different paths into a single socket connection (that is: no API changes).

      Sort of: if both WiFi and cell channels are available (think: wandering in a shopping mall with public hotspots), one's Android mobile will use both of them in the same time to manage one's plot in Farmville (or to download the MP3's using that magnet from the PirateBay, or placing whatever buy/sell orders on stock exchange); if one walks out of WiFi spot coverage, the mobile will use only what's available - the cell connection.

      Why I used android in my example? Well, it's a Linux kernel, the first implementation is already available. Besides, that should be great news for Google: their "goggles" will be able to transmit what you see much faster and reliable. What I understand from the MCTCP guys' presentation makes me believe MPTCP is able to cope with the use and drop from use of multiple dynamically IP addresses (are assigned to the many network devices one's mobile has): thus stepping from one hot-spot to another will not impede Google's capability to receive the data from your (their?) glasses.

      --
      Questions raise, answers kill. Raise questions to stay alive.
    3. Re:API support by camperdave · · Score: 1

      IPv6 should be transparent to the application as well, yet still it remains largely unimplemented.

      --
      When our name is on the back of your car, we're behind you all the way!
    4. Re:API support by CAIMLAS · · Score: 4, Interesting

      Yep. And this is a godsend, in some ways: "multipath NFS" should soon be inexplicably easier to accomplish on a high scale. I will be able to put in a single redundant/HA host with 8 1GBps NICs and not have to worry about setting up multipath on each of the individual VM heads I run. This has the significant advantage of not being stuck with immobile "SAN storage" LUNs or, for that matter, "enterprise" hardware vendors which can't bring the reliability their hardware close to anything near what generic Intel or even bcm network cards can provide.

      All the better if I've got unified storage at the backend with abstracted paths (eg. lustre, unionfs).

      And from the looks of it, it's designed 'forward' - it's going to be MUCH easier to do HA TCP connectivity with this than it is with misc. service level TCP (eg. heartbeat), particularly when you're dealing with (mostly) centrally assigned IPv6 addresses. Awesome.

      Granted, from the looks of it, we may have to wait for switch support first, too... I didn't read that carefully.

      --
      ~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
    5. Re:API support by c0lo · · Score: 1

      Coincidence: I'm waiting for my ProLiant N40L G7 MicroServer to be delivered early next week; ordered with 3 WD NAS grade HDD-es (the micro-server: $200. The HDD-es: $450+ all 3)
      Guess for what I'll use it and how I'll be testing MPTCP soon?

      --
      Questions raise, answers kill. Raise questions to stay alive.
    6. Re:API support by olivier.bonaventure · · Score: 4, Informative

      The current implementation in the Linux kernel only exposes the first address used in the connection to the application. If the addresses change, the application is not informed but the TCP connection remains alive. Exposing addresses to the application is an old mistake of the socket interface. The socket interface does not expose packet losses because TCP deals with them and provides a bytestream abstraction to the application. Multipath TCP does the same, it handles all changes in address transparently to the application.

    7. Re:API support by olivier.bonaventure · · Score: 5, Informative

      Multipath TCP supports transparently IPv4 and IPv6. A Multipath TCP connection can start over IPv4 and then use IPv6 without the application being aware of the utilisation of IPv6. This could help the utilisation of IPv6 paths by IPv6 unaware TCP applications.

    8. Re:API support by funkboy · · Score: 2

      Great idea.

      The fact that the protocol supports this without requiring changes to the applications is pretty impressive.

    9. Re:API support by Floyd-ATC · · Score: 1

      Isn't the whole point with MPTCP that only the endpoint hosts (e.g. VMware and the NFS/iSCSI NAS) need to support MPTCP while the routers and switches do not? Or did I misunderstand something?

      --
      Time flies when you don't know what you're doing
    10. Re:API support by jgrahn · · Score: 1

      IPv6 should be transparent to the application as well, yet still it remains largely unimplemented.

      From my point of view (Linux), it is largely implemented at the endpoints by now. We just don't have many servers to talk to, and no ISPs willing to sell it to us.

  7. Re:Uh, I get this with lacp by c0lo · · Score: 1

    yes, I know etherchannel load balancing ... but maybe that would be easier to "fix" than inventing something that mostly exists. ... 10 years ago.

    Do you also know the nowadays mobile devices? Wouldn't it be nice to use both WiFi and mobile wireless communication in the same time without special equipment from Cisco? Even more: transparent to you when you step from on public WiFi hot-spot coverage into another and be assigned with new IP address?

    --
    Questions raise, answers kill. Raise questions to stay alive.
  8. Use Cases? by Anonymous Coward · · Score: 1

    I'm possibly missing the point here, but I'm struggling to understand how this would be put to use:
    Servers that can pool multiple high-speed interfaces:
    This capability has been around for years - port channeling/interface bonding/whatever vendors call it, do this already at L2. Aside from making it easier for those interfaces to sit on different subnets, why would you want to push this capability to L3/TCP? Seems like it just introduces additional complexity.

    "Smart" devices (or any client really)
    I suppose this makes a bit more sense, but given the order of magnitude latency and (typically) bandwidth differential between your typical 3/4G and a local wireless connection, this doesn't seem like a huge boost - 3/4G in particular is more hampered by latency than net available bandwidth anyway. Does a wider pipe really boost things that much?

    On a side note - I thought TCP was all about guaranteed and ordered delivery - aren't you just stuck waiting for the slowest link in the chain anyway?

    1. Re:Use Cases? by aXis100 · · Score: 3, Informative

      You're missing the point. One of the big reasons to have multiple interfaces is for redundancy - with a company's internet interface, redundancy would be vastly improved by choosing two different providers, and even better with different mediums. The subnets will definitely be different.

      Having both of these links acting simultaneously would be great and I could see a lot of people being excited about it.

    2. Re:Use Cases? by fa2k · · Score: 1

      It would be great if the SSH connections from my laptop used the reliable ethernet connection when it's plugged in, and seamlessly switched to wireless LAN when it was unplugged. Even better would be if I could put the laptop to sleep and move to a different location, and that the connections would stay open (given, of course, that there was no I/O on them while the computer was moving, and that it was only moving for a reasonably short time). It seems like it *may* be possible to open a new "subflow" even when no valid existing subflows exist between the computers, as would be the case when the computer disconnects from one network and connects to another. The performance is a bit of a "meh" for me as an end user. I even have two gigabit NICs on my motherboard, but I only use one because the file transfer protocols I use (or the CPU for encryption) max out at ~60MB/s anyway. Otherwise there is usually a "strict hierarchy" in the speeds of the connections. Ethernet >> 802.11? laptops, wired internet connections >> 802.11? on mobiles >> 3G >> GPRS >> Bluetooth

    3. Re:Use Cases? by c · · Score: 1

      Having both of these links acting simultaneously would be great and I could see a lot of people being excited about it.

      Well, "both" links won't work too well.

      One of the main problems with using your one redundant link as just a regular link is your bandwidth "needs" tend to grow to include some or even all of the bandwidth available from the now non-redundant link. Then not only do you not have a redundant link, but you've doubled your bandwidth requirements and made it that much harder to provision in an emergency.

      Now, getting back to multipath TCP, the picture changes when you've got n links and your day-to-day bandwidth requirements only depend on a total of, say, n-2. Of course, in practice getting n truly independent links into any given facility is a bit of a trick...

      --
      Log in or piss off.
    4. Re:Use Cases? by nine-times · · Score: 1

      One of the main problems with using your one redundant link as just a regular link is your bandwidth "needs" tend to grow to include some or even all of the bandwidth available from the now non-redundant link.

      Really, in most cases, there's a big difference between "bandwidth needs" and "bandwidth wants". Businesses who normally use a 10Mb connection will frequently limp along on a backup T1 when necessary, without serious problems. Being without an Internet connection is a much bigger problem.

      But then the question frequently comes up: We have an 10Mb connection, and then a 1.5Mb backup connection that sits around doing nothing all day. Can we somehow combine them into a 11.5Mb connection?

      And the answer is generally "no". You can do a few kinds of load balancing, but each of them have some non-trivial drawbacks.

    5. Re:Use Cases? by olivier.bonaventure · · Score: 1

      This works. The video used on the front page of the Multipath TCP website runs over ssh with 3G,WiFi and Ethernet.

      http://www.youtube.com/watch?feature=player_embedded&v=VWN0ctPi5cw

  9. Re:Is using all connections good? by Anonymous Coward · · Score: 1

    You disable the ones you do not want to use at any given time.

  10. Re:Uh, I get this with lacp by LordLimecat · · Score: 4, Informative

    No, you dont. If I remember correctly, LACP will give you the maximum bandwidth provided by a single link, per connection. You cant just hook up LACP / LAGG / whatever your vendor calls it, fire up iSCSI, and magically have a 2gbps link to your SAN-- because iSCSI does a single connection per LUN, you will get a 1gbps connection even with LACP.

    LACP gets you higher total capacity, so if you were running two iSCSI connections you could get 1gbps on each with no contention. If the summary be believed, this would give you a truly multi-gbps link off of aggregated gbit connections.

  11. Re: Uh, I get this with lacp by jerquiaga · · Score: 1

    No need for expensive Cisco equipment. I have cheap Net gear switches that are completely LACP capable, and some cheap Realtor adapters support teaming now. It's more a driver issue than anything. If cell manufacturers designed their equipment and built the right drivers, you could easily dynamically team a cell and wifi connection.

  12. API by manu0601 · · Score: 1

    I understand the application sets up its sockets as usual, and the kernel add TCP extensions for MTCP, and use them to negociate with remote endpoint and startup the subflows. This is transparent for the application, but is there any way it can inspect what is going on? I think about some kind of generalized getpeername(), which would allow the caller to get information about all subflows in use.

  13. Re:You're supposed to get an AS number. by pcjunky · · Score: 1

    Does not work with nat. Unless you could bgp anounce a single ip address.

  14. Re:You're supposed to get an AS number. by pcjunky · · Score: 1

    On second thought not even then. Each connection has it's own ip address. You would need to find a way to terminate a single TCP connection on two ips. I assume this require change to the server and client end's software/os/

  15. Re: Uh, I get this with lacp by c0lo · · Score: 2

    If cell manufacturers designed their equipment and built the right drivers

    And if Apple refuses to implement it, you will still be able to grab an Android, compile/install the MPTCP stack and do it (without waiting for Apple to resist the mobile providers pressure in not supporting a feature that would hurt their bottom line. Or, for the matter, wait for the mobile providers to upgrade their towers and hurt their bottom line by themselves).

    --
    Questions raise, answers kill. Raise questions to stay alive.
  16. Re:You're supposed to get an AS number. by Guido+von+Guido+II · · Score: 2

    If you want to use multiple links all at the same time, with the packets spread over them, you're supposed to get an Autonomous System number.

    This is more akin to link aggregation than it is multihomed Internet connections. Any two hosts could use this. They could be in the same autonomous system. They could be on the same subnet. There's no need to get a separate AS number for each host.

    Note that one of the other use cases suggested is for smartphones.

  17. Support available already for most unices by c0lo · · Score: 4, Informative

    For those wanting to try, their install howto. Seems supported on:
    1. Linux - either debian binaries or compiling from source. Both kernel module and UserSpace ways.
    2. Virtualized Linuxes - their example is provided for Amazon EC2
    3. Mac OSX - but, obviously, not on iPhone (I estimate slim chances for this to happen in the near future - it's a technology disruptive for the mobile providers income, as it makes the multi-pathing over cell/WiFi hot-spots transparent to end user)
    4. Android (Opinion: see? This is one of the reasons relying on "walled gardens" is bad: you have to wait for the mercy of the garden lord to benefit from something).

    --
    Questions raise, answers kill. Raise questions to stay alive.
  18. Offload by jamesh · · Score: 1

    Is anyone making network adapters with a built in programmable processor with an open specification these days? This particular protocol may well be compatible with some of the existing checksum offload implementations but GSO and GRO would seem to need something special especially for multiplexing across streams.

  19. Re:You're supposed to get an AS number. by petermgreen · · Score: 1

    When you advertise something over BGP you essentially broadcast it to every core router in the world. Having every core router know about every device is just not going to happen. Having every router know about every ISP and large company is bad enough. Also BGP is built on a mixture of trust and manually applied filters. So unless you want to open it up to every idiot breaking other peoples traffic then it's not going to handle systems that regually move arround very well.

    Also internet routing looks for the path with the least number of AS hops and will generally only use one connection for any particular pair of end systems. What you really want as an end user with multiple connections is to use both paths at once to connect to the same place.

    --
    note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
  20. Re:Uh, I get this with lacp by BitZtream · · Score: 1

    Then you setup your load balancing on your LACP links incorrectly for what you were trying to accomplish. Choose a different implementation for pathing and you'll get total aggregate.

    The problem you're refering to is a problem with the implementation of LACP you're using, not the specification itself.

    --
    Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
  21. What am I missing? by jay508 · · Score: 1

    So they're able to get 50Gbps out of 6 10Gbps circuits/handoffs/etc... well simple math would say why only 50 and not 60? Ok so it's cool if you want to tell me that I can simultaneously send data to a single destination via wifi and 3/4g, simultaneously being the key word. that part is interesting because in most cases the public address is different. But the article also mentions "servers that can pool multiple high speed interfaces". What? Any decent firewall, switch or server has been providing this functionality for years. It's called teaming or bonding. You want to send a shitload of data to a destination but it takes too long? Not a problem, throw a couple quad nics in those bitches and bond them up, problem solved providing your network can support the throughput. What am I missing?

    1. Re:What am I missing? by aXis100 · · Score: 1

      Yes, you are missing a clue.

      If I got 50Gbps out of 6 x 10Gbps links I'd be ecstatic. That's pretty good efficiency considering the sheer throughput processing requited and overheads involved.

      Plus it may be difficult if not impossible to get multiple high speed interfaces via one internet carrier - however what if 3 or 4 fibre providers went past your datacenter? Buy bandwidth from all of them and use them simultaneously and add redundancy.

    2. Re:What am I missing? by Anonymous Coward · · Score: 3, Informative

      You want to send a shitload of data to a destination but it takes too long? Not a problem, throw a couple quad nics in those bitches and bond them up, problem solved providing your network can support the throughput.

      What am I missing?

      This is layer 4 not 2. So long as both endpoints support it, it don't matter where the traffic goes. they can go over entierly different paths. This is doing what you describe, but over the internet. Transparent to the network, and the higher levels of the protocol stack.

  22. They should kickstart an appliance by Gerner · · Score: 1, Insightful

    This would be a dream for small businesses and home users. Have 2-4 DSL/Cable/Wireless WAN ports and one port for your LAN/Router. Plug and play for instant redundant internet at a much higher speed than any one low cost connection could provide.

    1. Re:They should kickstart an appliance by funkboy · · Score: 1

      NOOOooo...

      Why would you need an appliance for something that can be done in software?!? The only "appliance" you need is an ethernet switch between your residential connection bridges and your firewall.

    2. Re:They should kickstart an appliance by Gerner · · Score: 1

      Um, so that you don't have to have software written on your VoIP phone, your smart phone, your iPad, your TV media streamer, your Linux, Mac, Windows computers, your Chrome Pixel, etc. Is that enough, or should I wait for all of these to take advantage of this. Do you have any idea how many businesses would jump at a simple way to upgrade their internet without installing something on every device in the building?

  23. Re:Standard DSL + custom host file = 50gbps connec by Anonymous Coward · · Score: 2, Interesting

    or maybe we could just filter comments based on length or number of links. >1000 words or >20 links

  24. Re:Uh, I get this with lacp by silas_moeckel · · Score: 3, Interesting

    Not unless they changed something recently. Read http://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf LACP requires that any conversation goes over only a single link at a time. Out of order packets can do some rather nasty things to tcp connections and adding buffers to correct that does nasty things to voip / other latency sensitive bits. Sure linux boxes have some non standard modes that might work if you sitting one switch away but that's not conforming to the LACP spec. They also do not scale as they require keeping state of every session running through them. What networking gear are you using?

    --
    No sir I dont like it.
  25. Re: Uh, I get this with lacp by Anonymous Coward · · Score: 1

    Yes, you can do it at layer 2 with various different technology, some vendor specific, some vendor neutral. This new method is doing it at layer 4. If you can't see why this matters then I suggest learning why the layered model exists in the first place.

  26. Re:Uh, I get this with lacp by LordLimecat · · Score: 4, Informative

    According to both the article which silas linked below (which is the original source for what I said), as well as a whole boatload of other documentation, thats not correct; its an 802.1ad issue.

    I did find this on serverfault which indicates that ONLY balance-roundrobin can get you 2gbps on a single tcp connection; and it also notes that some protocols dont like it, which means that its not really a transparant bonding technology. All of the other methods of distributing packets rely on a hash of various values, for instance source mac and destination mac IDs, and regardless of method the hash will ALWAYS be the same on a single TCP connection, which means that the same single link will be used.

    Regardless, the Linux Bonding driver is NOT the same thing as LACP, and its not something you implement on the switch.

  27. Re:fault tolerance by patch11 · · Score: 4, Informative

    MPTCP has separate sequence-number spaces. One for the subflow, inside the regular TCP header. And the data sequence-numbers, included inside the TCP option-space.

    This data sequence numbers include data-acks. So, this is your mentioned "cross-subflow ack machinery".

  28. Tor may actually take a nice advantage here. by Anonymous Coward · · Score: 1

    Has anyone actually thought about how The Onipn Network might take advantage of this? This could potentially, or partially, fix one of Tor's biggest problems, which is the inherent low bandwidth due to one single TCP connection via several nodes in a serial manner. If Tor could leverage multiple nodes in parallel into a single connection, this ould certainly make it faster.... And even more anonymous tha it is today.

  29. This was fun, back in 2004 by EmagGeek · · Score: 1

    I was doing TCP multipath in 2004 using iptables to get more upstream out of my box at home. I had two 1.5/384 connections and could and up with 768k upstream. All it took was a clever iptables script that marked alternating packets - even and odd if you will - and mangled odd packets to go out one interface, and even packets for the other.

    Obvioulsy, an actual TCP extension for this is going to be more elegant since it's more scalable and easy to deal with, but the idea is not new.

  30. mobile phone - YES by Anonymous Coward · · Score: 1

    I find it annoying that my app is disrupted when I leave a Wifi and the (Android) phone needs a few seconds to connect to the #G/4G network. But then it might just to that because it needs to save the power on the network interface.

    But then I'd buy the phone with double or tripple the battery over the slim one anyway. If only they would be available.

    1. Re:mobile phone - YES by olivier.bonaventure · · Score: 1

      Multipath TCP works in this use case and supports seamless handover. See http://inl.info.ucl.ac.be/publications/exploring-mobilewifi-handover-multipath-tcp for a detailed explanation. See https://github.com/mptcp-nexus/android for the MPTCP port on the google nexus

  31. Hope Application by Malenx · · Score: 1

    Waiting to see an apartment full of geeks and gamers rocking 10 cable modems all linked together and splitting the bill.

    1. Re:Hope Application by Malenx · · Score: 1

      *home*

  32. NFS on MTCP: was :API support by Monkius · · Score: 1

    NFS on multipath is my interest, too.

    In NFS (v4) on TCP, the endpoints frame messages on the bytestream, independently in each direction. As FireFury03 states above, we're basically (potentially large) packet based. We'd like help from new transports in framing those messages optimally, avoiding head-of-line blocking for entire messages.

    In addition to solving HOL, it's been proposed that we could design message framing on SCTP so as to deliver messages and data chunks on different streams, and get some advantages of NFS on RDMA.

    I think it's the same story with a lot of protocols, including HTTP. In fact, like most web servers, the ONC RPC stack I work on is in user space, so I have a/the more complex version of these problems.

    So do MTCP developers see solutions for any of these problems on the horizon?

    --
    Matt
  33. Re:NFS on MTCP: was :API support by c0lo · · Score: 1

    So do MTCP developers see solutions for any of these problems on the horizon?

    I'm in no way affiliated or knowledgeable in MPTCP - so I may be wrong - but from what I got from their presentation, MCTCP is not actually designed with the NFS-like usecase in mind (even if it may be used for such), but with the more "common" usecase of a mobile device able to use either/both cellular and WiFI networking.

    The second thing that I saw as peculiar: it is not even supported by a network protocol (like IP is supporting TCP/UDP/SCTP/etc), but is supported by TCP. While it will have to deal somehow with re-assembling back a stream from packets streams over different paths, in itself it will be as prone to HOL as the bunch of underlying individual TCP streams which support those different paths.
    That is, assuming the head-of-line is send on one path and that path involves a HOL-blocking, then it doesn't matter if the other paths have lower latencies, the entire original stream will be HOL-blocked

    --
    Questions raise, answers kill. Raise questions to stay alive.
  34. Re:NFS on MTCP: was :API support by patch11 · · Score: 1

    That is, assuming the head-of-line is send on one path and that path involves a HOL-blocking, then it doesn't matter if the other paths have lower latencies, the entire original stream will be HOL-blocked

    The implementation includes a solution to overcome HOL-blocking by reinjecting the blocking data-segment on the lower-latency path. Have a look at our scientific paper, which explains this mechanism: http://inl.info.ucl.ac.be/publications/how-hard-can-it-be-designing-and-implementing-deployable-multipath-tcp

  35. Re:NFS on MTCP: was :API support by c0lo · · Score: 1

    The implementation includes a solution to overcome HOL-blocking by reinjecting the blocking data-segment on the lower-latency path.

    Have a look at our scientific paper, which explains this mechanism: http://inl.info.ucl.ac.be/publications/how-hard-can-it-be-designing-and-implementing-deployable-multipath-tcp

    Oh, wow! Thanks.

    --
    Questions raise, answers kill. Raise questions to stay alive.
  36. Re:NFS on MTCP: was :API support by Monkius · · Score: 1

    Hi,

    Speed-reading the paper, there appears to be some implicit ammunition for the SCTP approach (if it worked!), at least for applications like HTTP, NFS, etc, which are characterized by multiplexing of large and small messages on the stream. I conclude this from section 5.3, which I think states MPTCP over 2 links was slower than ordinary TCP over one link, when message size was 30K.

    (Apologies if I'm misreading.)

    Thanks,

    Matt

    --
    Matt
  37. Re:NFS on MTCP: was :API support by christoph.paasch · · Score: 1

    I conclude this from section 5.3, which I think states MPTCP over 2 links was slower than ordinary TCP over one link, when message size was 30K.

    For very small flow-sizes (like less than 30KB), MPTCP should not try to create additional subflows. Because, the whole data fits in the initial window of the first subflow. However, at the moment the linux implementation always tries to establish new subflows. In the paper's stress-testing scenario these additional subflows just consumed CPU-cycles and thus the "bad" results for MPTCP with very small flows

    An easy fix would be to delay the establishment of additional subflows until a certain threshold of data has been sent or a certain time has passed.

  38. Re:NFS on MTCP: was :API support by Monkius · · Score: 1

    Thanks, Christoph.

    I think I was mis-reading, section 5.3 is discussing performance with short-lived connections, such as happens with HTTP .9 or 1.0. The question I would ask next is, how does MPTCP perform when HTTP 1.1 or similar channel multiplexing is used?

    Matt

    --
    Matt
  39. Re:NFS on MTCP: was :API support by christoph.paasch · · Score: 1

    We have not yet done extensive experiments with HTTP1.1.

    But, MPTCP is a benefit for bandwidth-intensive (thus increasing throughput) and long-lived connections (increased resilience against link-failures).

    So, I would say that HTTP 1.1 would benefit from MPTCP.

    Cheers,
    Christoph

  40. New and Improved! by AndrewX · · Score: 1

    TCP X2 now with Shotgun technology!

  41. Re:Uh, I get this with lacp by LordLimecat · · Score: 1

    LACP uses various methods to choose which link to send frames over-- for example sourceport id, source mac, etc. Regardless of what you choose, a single TCP connection will end up using the same link even when LACP is implemented on the switch.

    You might try reading the linked articles in my and silas' responses before arguing; particularly as one of them is a link to the IETF spec.

  42. Re:Uh, I get this with lacp by LordLimecat · · Score: 1

    Yes, you have that total amount of bandwidth. If you were to have 4 iSCSI connections, each of them would get a full gigabit; if you had 8 connections each would get 500mbps.

    However, a single connection from a single TCP port coming out of a single MAC address / IP address is going to get a single gigabit /sec of traffic; theres not really a good workaround for this.

    If youve found a way to get 4gbps on a single iSCSI connection using LACP, please do share, as a LOT of people would be interested to get that running.

  43. Re:Uh, I get this with lacp by silas_moeckel · · Score: 1

    It's worse than that depending on how things are hashing you can have 4 connections and 2 of them use the same link (or all four) It's really dependent on the networking kit your using LACP does not specify how you do this just that you do something to insure packets for a given "conversation" only go down one path at a time. Simpler networking kit might only look at 1 mac address smarter bits go all they way up to protocol ports.

    For iSCSI the "easy" fix is to run multipath it assuming your san supports it. That makes LACP work much better as it has more mac's IP and ports to hash against.

    --
    No sir I dont like it.
  44. Re:Uh, I get this with lacp by silas_moeckel · · Score: 1

    LACP does not need to care about L3 or L4 it's free to do so. Really cheap networking kit (and some broken really expensive bits) only use mac's ever smarter kit can do whatever it wants higher up in the protocol stack. How it distributes packets is not something that's negotiated it's just fixed or a setting on each end. If you plug the dumbest of the dumb lacp switch into the smartest switch you will get good load balance in one direction and poor the other.

    --
    No sir I dont like it.
  45. Re:You're supposed to get an AS number. by olivier.bonaventure · · Score: 1

    Multipath TCP traverses NAT and other types of middleboxes without problems.

  46. Re:You're supposed to get an AS number. by olivier.bonaventure · · Score: 1

    The main benefit of Multipath TCP when used in multihoming scenarios compared to BGP-based multihoming is that Multipath TCP capable hosts can use different paths simultaneously while BGP-based multihoming would provide one path for each client server pair.

    With Multipath TCP, multihoming must be exposed to the server. For example, consider a small enterprise network connected to two different providers, A and B. With BGP-based multihoming, you assign address C to your server and advertise it through the two providers via BGP. BGP decides which path will be used and the ASPath metric used by BGP is far from being the most accurate metric to evaluate the quality of a path.

    With Multipath TCP, you should ask a block of addresses from both A and B and assign one address from each provider you your server, say A.1 and B.1. Both addresses are advertised in the DNS. Address A.1 is always reachable via provider A and B.1 always reachable via provider B. When a TCP connection reaches the server, say over A.1, it will also advertise address B.1 using Multipath TCP and a second subflow will be established. Multipath TCP will then regulate the usage of the two paths in function of the amount of congestion on each path. If one path fails, Multipath TCP will perform failover automatically.