Slashdot Mirror
A 50 Gbps Connection With Multipath TCP
First time accepted submitter Olivier Bonaventure writes "The TCP protocol is closely coupled with the underlying IP protocol. Once a TCP connection has been established through one IP address, the other packets of the connection must be sent from this address. This makes mobility and load balancing difficult. Multipath TCP is a new extension that solves these old problems by decoupling TCP from the underlying IP. A Multipath TCP connection can send packets over several interfaces/addresses simultaneously while remaining backward compatible with existing TCP applications. Multipath TCP has several use cases, including smartphones that can use both WiFi and 3G, or servers that can pool multiple high-speed interfaces. Christoph Paasch, Gregory Detal and their colleagues who develop the implementation of Multipath TCP in the Linux kernel have achieved 50 Gbps for a single TCP connection [note: link has source code and technical details] by pooling together six 10 Gbps interfaces."
RFC 6182 if anyone is interested.
-- I have a private email server in my basement.
Doesn't SCTP provide for these scenarios (and many more)?
No.
SCTP supports multiple paths between endpoints, but doesn't use them simultaneously. Rather, it picks a primary path to use for data transfers and has the ability to fail over to an alternate path in the event the primary fails.
A quick glance at the MTCP RFC shows that it is essentially multiplexing packets over n separate TCP streams (called subflows). It's the responsibility of the TCP/IP stack (in the OS, generally) to make this multiplexing transparent to the application, so the application only sees one stream.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
1. SCTP - identified by a protocol number (132) - acts at the network layer. If a router along the route refuses SCTP, you are screwed; Advantage: is capable of UDP as well).
2. MPTCP - relies on pure TCP for all the connection (acts at the transport layer and fixes the protocol to TCP) and set in place conventions between client-server to discuss over multiple paths. Advantage: no sane public network will try to block it (pretty much like using http on port 80). Disadvantage: TCP only.
Questions raise, answers kill. Raise questions to stay alive.
without every user making 3 connects to view their friends cat picture.
Rest assured: there'll be a single connection using a cell tower. A second flow will be made using the connection with nearby WiFI hot-spot, and Tiffany's chatting to her buddy sitting next to her will be really faster (without quotes); even better, the above will happen without Tiffany knowing or the extra requirement for Tiffany to have a geek father that's not lazy and does have spare time (even if one may wonder what to what good being a geek will be in the future).
Questions raise, answers kill. Raise questions to stay alive.
No, you dont. If I remember correctly, LACP will give you the maximum bandwidth provided by a single link, per connection. You cant just hook up LACP / LAGG / whatever your vendor calls it, fire up iSCSI, and magically have a 2gbps link to your SAN-- because iSCSI does a single connection per LUN, you will get a 1gbps connection even with LACP.
LACP gets you higher total capacity, so if you were running two iSCSI connections you could get 1gbps on each with no contention. If the summary be believed, this would give you a truly multi-gbps link off of aggregated gbit connections.
For those wanting to try, their install howto. Seems supported on:
1. Linux - either debian binaries or compiling from source. Both kernel module and UserSpace ways.
2. Virtualized Linuxes - their example is provided for Amazon EC2
3. Mac OSX - but, obviously, not on iPhone (I estimate slim chances for this to happen in the near future - it's a technology disruptive for the mobile providers income, as it makes the multi-pathing over cell/WiFi hot-spots transparent to end user)
4. Android (Opinion: see? This is one of the reasons relying on "walled gardens" is bad: you have to wait for the mercy of the garden lord to benefit from something).
Questions raise, answers kill. Raise questions to stay alive.
According to both the article which silas linked below (which is the original source for what I said), as well as a whole boatload of other documentation, thats not correct; its an 802.1ad issue.
I did find this on serverfault which indicates that ONLY balance-roundrobin can get you 2gbps on a single tcp connection; and it also notes that some protocols dont like it, which means that its not really a transparant bonding technology. All of the other methods of distributing packets rely on a hash of various values, for instance source mac and destination mac IDs, and regardless of method the hash will ALWAYS be the same on a single TCP connection, which means that the same single link will be used.
Regardless, the Linux Bonding driver is NOT the same thing as LACP, and its not something you implement on the switch.
On the contrary, SCTP is a transport protocol just like TCP, except with a large number of added features. The main problem with SCTP has nothing to do with SCTP at all. It is that NAT devices do not support any transport protocol that they haven't been programmed for in advance. This makes SCTP next to impossible to deploy on a broad scale - NAT, that wart upon router-kind, is ubiquitous.
TCP would have exactly the same problem if it were a new protocol. A NAT device requires relatively deep knowledge of TCP to support it at all. It play games with both ports and addresses, keeps track of connection state, and so on. Ordinary routers do no such thing. A NAT device is a transport layer proxy by another name.
Yep. And this is a godsend, in some ways: "multipath NFS" should soon be inexplicably easier to accomplish on a high scale. I will be able to put in a single redundant/HA host with 8 1GBps NICs and not have to worry about setting up multipath on each of the individual VM heads I run. This has the significant advantage of not being stuck with immobile "SAN storage" LUNs or, for that matter, "enterprise" hardware vendors which can't bring the reliability their hardware close to anything near what generic Intel or even bcm network cards can provide.
All the better if I've got unified storage at the backend with abstracted paths (eg. lustre, unionfs).
And from the looks of it, it's designed 'forward' - it's going to be MUCH easier to do HA TCP connectivity with this than it is with misc. service level TCP (eg. heartbeat), particularly when you're dealing with (mostly) centrally assigned IPv6 addresses. Awesome.
Granted, from the looks of it, we may have to wait for switch support first, too... I didn't read that carefully.
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
Work is underway for concurrent multipath transfer for SCTP as well. Also known as CMT-SCTP. There are significant challenges in doing this sort of thing though. SCTP wasn't designed for CMT, and probably needs much more radical changes than the current architects are proposing to do it well.
Changes like subflows with independent sequence numbers and congestion windows, to start with. SCTP is much further ahead in the connection handling and security department, but MPTCP has the odd advantage of resorting to independent subflows to begin with, and if it can handle path failure properly, it might well be ahead in the CMT game, if byte stream semantics are all you need.
SCTP is cleaner than Multipath TCP, but it suffers from two drawbacks that hinder its deployment in today's Internet : - many middleboxes only support IP, ICMP and TCP and discard SCTP packets (or do not perform NAT correctly) - applications need to be modified to support SCTP Multipath TCP is an evolution to TCP that works with unmodified applications and unmodified middleboxes.
The limit here is the CPU and on the sender and the receiver. Both servers used in the test reached 98% CPU load to achieve 52 Gbps. Note that 52 Gbps is the googput at the application and not the bandwidth used on the links (which is higher due to the various overheads)
The current implementation in the Linux kernel only exposes the first address used in the connection to the application. If the addresses change, the application is not informed but the TCP connection remains alive. Exposing addresses to the application is an old mistake of the socket interface. The socket interface does not expose packet losses because TCP deals with them and provides a bytestream abstraction to the application. Multipath TCP does the same, it handles all changes in address transparently to the application.
Multipath TCP supports transparently IPv4 and IPv6. A Multipath TCP connection can start over IPv4 and then use IPv6 without the application being aware of the utilisation of IPv6. This could help the utilisation of IPv6 paths by IPv6 unaware TCP applications.
MPTCP has separate sequence-number spaces. One for the subflow, inside the regular TCP header. And the data sequence-numbers, included inside the TCP option-space.
This data sequence numbers include data-acks. So, this is your mentioned "cross-subflow ack machinery".