There is a simple solution to the software integration problems described in your paper: stream sockets - sockets that handle one stream (pair) of an SCTP association at a time, more like a traditional TCP socket.
One a web browser like Firefox, whenever a new connection to the same server is going to be established, just peel off a new stream socket. Then there is no problem with rendering threads processing one stream at a time.
On the web server side, accept() can be made to deliver new stream sockets when the first request arrives on a stream, so that each worker thread handles one stream at a time.
That would allow Firefox, Apache, etc. to support SCTP multi-streaming with minimal changes.
But on the initiating side, the SYN ACK does complete the connection, does it not? The target side has to wait for the ACK, which may or may not be bundled with data from the initiator.
I imagine a typical implementation never bundles data with the ACK because of context switch latency, between the time the SYN ACK arrives, and the time for the first write(2) to issue. Someone really ought to make an API for TCP that does a hybrid connect(2)/write(2) to fix this problem. I imagine sendto(2) with an IP address might be suitable, just as in SCTP. Surely T/TCP does something similar.
Not quite - the decision that you are referring to said that RBOCs did not have to share the *same* copper pair with DSL providers. CLECs can still get their own lines.
The really scary part is the recent FCC decision to classify DSL as an "information service" that does not have to support independent ISPs at all, a decision that gives the Bell operating companies free a complete exemption from common carrier rules that were written to prevent Ma Bell from engaging in precisely the type of behavior that the FCC decided to give them free reign to engage in. Things like blocking or degrading anything they feel like for example. The FCCs current discussion about Internet discrimination is mostly a bunch of hot air, because they exempted the RBOCs from the very laws designed to prevent stuff like that.
(PR)SCTP's timed reliability service is on a per message basis. You just set the sinfo_timetolive for a message in milliseconds, and the message auto-expires if it hasen't been completely transmitted before that amount of time has elapsed. See RFC3758 for details.
Association timeout (SCTP_AUTOCLOSE option) is something else entirely, normally used to make UDP style applications transparent to association setup and teardown.
A typical SCTP socket implementation has four logical layers:
1. Socket 2. Stream 3. Association 4. Path
SCTP sockets come in two varieties, one-to-one style and one-to-many style. The one-to-one style handles up to 64K streams in each direction between two endpoints.
The one-to-many style aggregates several associations on the same endpoint, presenting a socket interface that resembles that provided by UDP, except reliable and ordered. The primary difference with RDS is that SCTP is generally implemented completely in software. An SCTP stream pair is logically equivalent to an Infiniband QP, but if someone wanted to provide a high performance SCTP style interface over IB they would likely run out of hardware QPs with that kind of mapping. SCTP associations with thousands of streams are common in the telco world.
RDS works fine with association level QPs because packet loss and high RTTs are not much of an issue with IB. I believe it has strong ordering guarantees making independently ordered streams moot. Mapping SCTP streams to QPs would be ideal for MPI / RDMA style applications, however.
If you do it yourself and implement a protocol over UDP, complete with congestion control, retransmission timers, duplicate detection, etc. then no problem. If you do it with TCP you still have the head of line blocking which SCTP eliminates.
Actually, most iWARP/RDMA stuff doesn't have a software interface to TCP at all - the hardware handles not only TCP, but three or more layers on top of it (at least MPA, DDP, and RDMA, plus iSCSI in some cases). That type of interface is not a problem. What is controversial is using TOE for conventional TCP applications using kernel space dispatch.
This is a bit of an end run around the Linux kernel bridging, routing, and filtering layers, which is the primary reason why support for it won't get merged in the kernel socket layer until RNICs can at a minimum do IPtables like IP address filtering and proper dispatch so that some packets can be routed through the kernel layers on an Ethertype and IP protocol / address / port specific basis.
It means that the backbone routers of the world cannot handle the routing entries required so that end user IP address blocks can be re-routed through alternate providers when one of them (or the link to one of them) goes down.
The way BGP works is that IP address prefix reachability information is advertised, router hop by router hop, from the destination router backwards until it either flood fills the entire global router backbone, gets merged with a larger provider prefix, or gets filtered for being too small.
Provider independent address space is hard to get, so what normally happens is your advertised IP address prefix gets merged with the larger one for the provider address block it was suballocated from. But then if the link to your provider goes down, you become unreachable as well, because all your traffic is routed through your provider.
The only way to handle that problem is for the providers you have chosen to peer and exchange detailed BGP information on a local (i.e. metro area by metro area) basis, which is relatively uncommon. Most major providers peer with each in only a handful of places in the U.S., and it is common for local providers not to do peer at all. Generally, unless your providers peer with each other at a local traffic exchange point, like PAIX, small prefix BGP re-routing is unlikely to work.
So the alternative to all this network layer re-routing is for higher level protocols to handle multiple IP addresses, each generally from a different provider. That is what SCTP and shim6 do at the transport or transport adaptation layer.
I should have said "half-closed" connections. In any case, some FTP implementations apparently rely on half closed connections in an incompatible manner.
See http://www.cis.udel.edu/~amer/PEL/poc/pdf/Bickhart MSthesis.pdf. Note that Bickhart has implemented a kernel space shim, which requires shim rules because it cannot intercept the name resolution process. A purely user space shim has a different set of problems, so perhaps a hybrid approach would be best.
I agree - the socket interface is the number one reason for transport protocols to be in user space. The time is ripe for someone to find a efficient way to allow user space libraries to implement socket descriptors that are interoperable with normal socket and file descriptors.
Even the socket equivalent of FUSE (where the protocol is run in a different process) would be useful for deploying new transport protocols where there is no kernel support.
Most modern NICs can offload TCP checksum generation and checking. Many can offload send side TCP segmentation (aka TSO / LSO / Large Send support).
However full TCP offload engines are expensive and rare outside of hardware iSCSI and RDMA implementations. They may be worthwhile at 10 Gbit/sec, but modern software TCP stacks handle 1 Gbit/sec with neglible overhead on decent hardware.
Data can be sent on the third leg of SCTP's four-way handshake, bundled with the COOKIE ECHO, making establishment time just as fast as TCP's three way handshake. The fourth leg (COOKIE ACK) is just to tell the client that the server received and accepted the COOKIE ECHO.
So instead of:
1. SYN 2. SYN ACK 3. REQUEST DATA 4. RESPONSE DATA
you get:
1. INIT 2. INIT ACK 3. COOKIE ECHO + REQUEST DATA 4. COOKIE ACK + RESPONSE DATA
This concept has potential but there are some problems. First, SCTP does not support half open connections like TCP, so some applications will not work without modifications.
Second, trying SCTP first and then falling back to TCP later causes considerable delay. To fix that problem the shim would need to insert itself in the name resolution process (getnameinfo) so that it could intelligently decide which protocol to try first. Of course name servers would have to carry SRV extension records to indicate that SCTP was supported on a port / ULP basis.
In fact the Linux TCP stack does most of its processing in process context by default (using VJ style prequeuing), although there is an option to change this for latency sensitive workloads.
Congestion control is an area where SCTP is much like TCP. SCTP uses AIMD on a per destination address basis. However any of the the alternative congestion algorithms for TCP would behave similarly with SCTP.
Of course given the additional message boundary information available in SCTP, further improvements could be made.
Layer 3 multihoming doesn't scale due to the combinatorial explosion of routing table entries. This problem is sufficiently serious that there is an IETF working group dedicated to providing a shim layer between layer 3 and layer 4 for IPv6 so that the problem with the size of global IPv4 routing tables does not repeat with IPv6.
Even now, it is practically impossible for a small to medium size business to be multihomed at the network layer, because they can't get an IP address block large enough so that BGP re-routing works across multiple providers (without being filtered away). And of course managing BGP sessions with multiple providers is time consuming, if you can get them to do it at all.
Both the proposed shim6 layer and SCTP provide solutions to the multihoming problem that have a much lower failover time, and much lower overhead than using BGP to do it. The providers neither need to know nor care.
And of course the advantage of SCTP / shim6 over using DNS to do it is that they both fail over active connections, not just retry new ones with different addresses.
SCTP and Infiniband focus on different areas. IB is largely a high performance HPC / cluster network architecture for LAN applications, where SCTP is a transport protocol designed to operate efficiently under WAN conditions (significant packet loss, high RTTs).
SCTP is a more efficient RDMA/iWARP transport than TCP, but the differential advantage of SCTP over TCP is much lower in a LAN environment due to the low RTTs, so RDDP/TCP dominates so far despite the bizarre marker insertion scheme (MPA). Same goes for iSCSI.
The interrupt issue has largely been solved - on Linux NAPI dynamically switches between interrupt and polled mode to reduce this overhead to negligible levels. Message signalled interrupts also help considerably.
What would be much more helpful (and economical) for iSCSI, SCTP, and RDDP is NIC CRC32C checksum generation. CRC generation is quite expensive in software but trivial in hardware.
SCTP wasn't originally designed for load balancing a single association via simultaneous multi-path transfer (SMT). It can be done, but it requires some loss detection algorithm changes. Someone still needs to develop a option to coordinate this at association establishment time.
One advantage of SCTP over TCP is that on a per stream basis, SCTP connection establishment overhead is much lower than TCP - basically O(1) instead of O(N) in the number of streams.
SCTP excels at low latency delivery of small messages. TCP's head of line blocking is a serious problem in many applications. SSH tunneling is a good example.
The main advantage of using SCTP over multiple TCP connections is connection establishment time as well as server overhead. You can create an association with hundreds of streams in the roughly the same time as a single TCP connection, with little or no overhead for unused streams. Then when you want to initiate a new non-blocking transaction you can send a message on a new stream without the three-way handshake of an extra TCP connection.
In addition, a single SCTP socket can handle reliably delivered messages on thousands of streams from hundreds of associations. No need to use select()/poll() on a long list of file descriptors.
SCTP does have an option for using name resolution to do multihoming, however for practical reasons it is almost universally unimplemented. SCTP multihoming works just fine without it. IP address lists for multihoming are exchanged during the standard connection (association) establishment process.
State cookies are not stored on the server at all, but rather are echoed from the client back to the server as a effective means of SYN flood style DoS attack prevention.
SCTP (properly implemented) is radically superior to TCP for a large class of applications, basically anything that needs low latency reliable message exchange. The lack of message boundary information in TCP causes considerable pain for implementers of upper layer protocols - notably RDMA/RDDP and iSCSI. The running solution for efficient hardware implementation of RDMA and iSCSI over TCP involves *inserting* markers every 512 bytes or so in the middle of a data stream so that the receiver can re-synchronize it efficiently.
The primary SCTP RFC is RFC 2960 for those who are wondering.
Unfortunately the current mainline Linux kernel SCTP implementation (LKSCTP) has some serious performance problems. On platforms with mature SCTP implementations (FreeBSD, OpenSolaris), SCTP performs *much* better.
This is a common claim, but it is an apples to oranges comparison. No one (including the compiler) dynamically allocates objects in C/C++ when they can place them on the stack instead. Garbage collected languages like Java, on the other hand, require practically everything to be managed on the heap.
In addition, an array of objects on the heap requires only a single memory allocation in C or C++, where Java has to allocate and track each separately. As one luminary once said, "C++ is better because there is less garbage to collect."
That might be acceptable, but the worst part is random application pauses of arbitrary duration for garbage collection. Unless that problem can be resolved, garbage collected languages will be always be a poor match for latency sensitive applications, even where the net throughput is otherwise adequate.
The notification procedures in the DMCA are there to protect the communications provider from liability for copyright infringement.
Since there has been no copying, creation of derived works, or public performance of the items in question, there is no copyright infringement, hence the DMCA doesn't apply.
The real question is does the DMCA protect the ISP from held liable by MonsterPatterns for breach of contract for pulling their service in response to a highly dubious copyright claim.
I agree. The main reason to prohibit dumpster diving is maintaining the privacy of the people generating the trash.
Perhaps a more effective way of helping the desperately hungry would be to pass a law absolving the restaurants, grocers, etc, from liability for giving their leftover food away. It would certainly be in better condition than when dug out of a dumpster hours later.
Slashdot Mirror
There is a simple solution to the software integration problems described in your paper: stream sockets - sockets that handle one stream (pair) of an SCTP association at a time, more like a traditional TCP socket.
One a web browser like Firefox, whenever a new connection to the same server is going to be established, just peel off a new stream socket. Then there is no problem with rendering threads processing one stream at a time.
On the web server side, accept() can be made to deliver new stream sockets when the first request arrives on a stream, so that each worker thread handles one stream at a time.
That would allow Firefox, Apache, etc. to support SCTP multi-streaming with minimal changes.
But on the initiating side, the SYN ACK does complete the connection, does it not? The target side has to wait for the ACK, which may or may not be bundled with data from the initiator.
I imagine a typical implementation never bundles data with the ACK because of context switch latency, between the time the SYN ACK arrives, and the time for the first write(2) to issue. Someone really ought to make an API for TCP that does a hybrid connect(2)/write(2) to fix this problem. I imagine sendto(2) with an IP address might be suitable, just as in SCTP. Surely T/TCP does something similar.
Not quite - the decision that you are referring to said that RBOCs did not have to share the *same* copper pair with DSL providers. CLECs can still get their own lines.
The really scary part is the recent FCC decision to classify DSL as an "information service" that does not have to support independent ISPs at all, a decision that gives the Bell operating companies free a complete exemption from common carrier rules that were written to prevent Ma Bell from engaging in precisely the type of behavior that the FCC decided to give them free reign to engage in. Things like blocking or degrading anything they feel like for example. The FCCs current discussion about Internet discrimination is mostly a bunch of hot air, because they exempted the RBOCs from the very laws designed to prevent stuff like that.
(PR)SCTP's timed reliability service is on a per message basis. You just set the sinfo_timetolive for a message in milliseconds, and the message auto-expires if it hasen't been completely transmitted before that amount of time has elapsed. See RFC3758 for details.
Association timeout (SCTP_AUTOCLOSE option) is something else entirely, normally used to make UDP style applications transparent to association setup and teardown.
A typical SCTP socket implementation has four logical layers:
1. Socket
2. Stream
3. Association
4. Path
SCTP sockets come in two varieties, one-to-one style and one-to-many style. The one-to-one style handles up to 64K streams in each direction between two endpoints.
The one-to-many style aggregates several associations on the same endpoint, presenting a socket interface that resembles that provided by UDP, except reliable and ordered. The primary difference with RDS is that SCTP is generally implemented completely in software. An SCTP stream pair is logically equivalent to an Infiniband QP, but if someone wanted to provide a high performance SCTP style interface over IB they would likely run out of hardware QPs with that kind of mapping. SCTP associations with thousands of streams are common in the telco world.
RDS works fine with association level QPs because packet loss and high RTTs are not much of an issue with IB. I believe it has strong ordering guarantees making independently ordered streams moot. Mapping SCTP streams to QPs would be ideal for MPI / RDMA style applications, however.
If you do it yourself and implement a protocol over UDP, complete with congestion control, retransmission timers, duplicate detection, etc. then no problem. If you do it with TCP you still have the head of line blocking which SCTP eliminates.
Actually, most iWARP/RDMA stuff doesn't have a software interface to TCP at all - the hardware handles not only TCP, but three or more layers on top of it (at least MPA, DDP, and RDMA, plus iSCSI in some cases). That type of interface is not a problem. What is controversial is using TOE for conventional TCP applications using kernel space dispatch.
This is a bit of an end run around the Linux kernel bridging, routing, and filtering layers, which is the primary reason why support for it won't get merged in the kernel socket layer until RNICs can at a minimum do IPtables like IP address filtering and proper dispatch so that some packets can be routed through the kernel layers on an Ethertype and IP protocol / address / port specific basis.
Ever tried to get portable address space lately? Or waited for a new BGP route to propagate across the global internet?
It means that the backbone routers of the world cannot handle the routing entries required so that end user IP address blocks can be re-routed through alternate providers when one of them (or the link to one of them) goes down.
The way BGP works is that IP address prefix reachability information is advertised, router hop by router hop, from the destination router backwards until it either flood fills the entire global router backbone, gets merged with a larger provider prefix, or gets filtered for being too small.
Provider independent address space is hard to get, so what normally happens is your advertised IP address prefix gets merged with the larger one for the provider address block it was suballocated from. But then if the link to your provider goes down, you become unreachable as well, because all your traffic is routed through your provider.
The only way to handle that problem is for the providers you have chosen to peer and exchange detailed BGP information on a local (i.e. metro area by metro area) basis, which is relatively uncommon. Most major providers peer with each in only a handful of places in the U.S., and it is common for local providers not to do peer at all. Generally, unless your providers peer with each other at a local traffic exchange point, like PAIX, small prefix BGP re-routing is unlikely to work.
So the alternative to all this network layer re-routing is for higher level protocols to handle multiple IP addresses, each generally from a different provider. That is what SCTP and shim6 do at the transport or transport adaptation layer.
I should have said "half-closed" connections. In any case, some FTP implementations apparently rely on half closed connections in an incompatible manner.
t MSthesis.pdf. Note that Bickhart has implemented a kernel space shim, which requires shim rules because it cannot intercept the name resolution process. A purely user space shim has a different set of problems, so perhaps a hybrid approach would be best.
See http://www.cis.udel.edu/~amer/PEL/poc/pdf/Bickhar
I agree - the socket interface is the number one reason for transport protocols to be in user space. The time is ripe for someone to find a efficient way to allow user space libraries to implement socket descriptors that are interoperable with normal socket and file descriptors.
Even the socket equivalent of FUSE (where the protocol is run in a different process) would be useful for deploying new transport protocols where there is no kernel support.
Most modern NICs can offload TCP checksum generation and checking. Many can offload send side TCP segmentation (aka TSO / LSO / Large Send support).
However full TCP offload engines are expensive and rare outside of hardware iSCSI and RDMA implementations. They may be worthwhile at 10 Gbit/sec, but modern software TCP stacks handle 1 Gbit/sec with neglible overhead on decent hardware.
Data can be sent on the third leg of SCTP's four-way handshake, bundled with the COOKIE ECHO, making establishment time just as fast as TCP's three way handshake. The fourth leg (COOKIE ACK) is just to tell the client that the server received and accepted the COOKIE ECHO.
So instead of:
1. SYN
2. SYN ACK
3. REQUEST DATA
4. RESPONSE DATA
you get:
1. INIT
2. INIT ACK
3. COOKIE ECHO + REQUEST DATA
4. COOKIE ACK + RESPONSE DATA
Same overall latency, better security.
This concept has potential but there are some problems. First, SCTP does not support half open connections like TCP, so some applications will not work without modifications.
Second, trying SCTP first and then falling back to TCP later causes considerable delay. To fix that problem the shim would need to insert itself in the name resolution process (getnameinfo) so that it could intelligently decide which protocol to try first. Of course name servers would have to carry SRV extension records to indicate that SCTP was supported on a port / ULP basis.
Somehow I think Van Jacobsen would disagree with you...
See http://lwn.net/Articles/169961/
In fact the Linux TCP stack does most of its processing in process context by default (using VJ style prequeuing), although there is an option to change this for latency sensitive workloads.
Congestion control is an area where SCTP is much like TCP. SCTP uses AIMD on a per destination address basis. However any of the the alternative congestion algorithms for TCP would behave similarly with SCTP.
Of course given the additional message boundary information available in SCTP, further improvements could be made.
Layer 3 multihoming doesn't scale due to the combinatorial explosion of routing table entries. This problem is sufficiently serious that there is an IETF working group dedicated to providing a shim layer between layer 3 and layer 4 for IPv6 so that the problem with the size of global IPv4 routing tables does not repeat with IPv6.
Even now, it is practically impossible for a small to medium size business to be multihomed at the network layer, because they can't get an IP address block large enough so that BGP re-routing works across multiple providers (without being filtered away). And of course managing BGP sessions with multiple providers is time consuming, if you can get them to do it at all.
Both the proposed shim6 layer and SCTP provide solutions to the multihoming problem that have a much lower failover time, and much lower overhead than using BGP to do it. The providers neither need to know nor care.
And of course the advantage of SCTP / shim6 over using DNS to do it is that they both fail over active connections, not just retry new ones with different addresses.
SCTP and Infiniband focus on different areas. IB is largely a high performance HPC / cluster network architecture for LAN applications, where SCTP is a transport protocol designed to operate efficiently under WAN conditions (significant packet loss, high RTTs).
SCTP is a more efficient RDMA/iWARP transport than TCP, but the differential advantage of SCTP over TCP is much lower in a LAN environment due to the low RTTs, so RDDP/TCP dominates so far despite the bizarre marker insertion scheme (MPA). Same goes for iSCSI.
The interrupt issue has largely been solved - on Linux NAPI dynamically switches between interrupt and polled mode to reduce this overhead to negligible levels. Message signalled interrupts also help considerably.
What would be much more helpful (and economical) for iSCSI, SCTP, and RDDP is NIC CRC32C checksum generation. CRC generation is quite expensive in software but trivial in hardware.
SCTP wasn't originally designed for load balancing a single association via simultaneous multi-path transfer (SMT). It can be done, but it requires some loss detection algorithm changes. Someone still needs to develop a option to coordinate this at association establishment time.
One advantage of SCTP over TCP is that on a per stream basis, SCTP connection establishment overhead is much lower than TCP - basically O(1) instead of O(N) in the number of streams.
SCTP excels at low latency delivery of small messages. TCP's head of line blocking is a serious problem in many applications. SSH tunneling is a good example.
The main advantage of using SCTP over multiple TCP connections is connection establishment time as well as server overhead. You can create an association with hundreds of streams in the roughly the same time as a single TCP connection, with little or no overhead for unused streams. Then when you want to initiate a new non-blocking transaction you can send a message on a new stream without the three-way handshake of an extra TCP connection.
In addition, a single SCTP socket can handle reliably delivered messages on thousands of streams from hundreds of associations. No need to use select()/poll() on a long list of file descriptors.
SCTP does have an option for using name resolution to do multihoming, however for practical reasons it is almost universally unimplemented. SCTP multihoming works just fine without it. IP address lists for multihoming are exchanged during the standard connection (association) establishment process.
State cookies are not stored on the server at all, but rather are echoed from the client back to the server as a effective means of SYN flood style DoS attack prevention.
SCTP (properly implemented) is radically superior to TCP for a large class of applications, basically anything that needs low latency reliable message exchange. The lack of message boundary information in TCP causes considerable pain for implementers of upper layer protocols - notably RDMA/RDDP and iSCSI. The running solution for efficient hardware implementation of RDMA and iSCSI over TCP involves *inserting* markers every 512 bytes or so in the middle of a data stream so that the receiver can re-synchronize it efficiently.
The primary SCTP RFC is RFC 2960 for those who are wondering.
Unfortunately the current mainline Linux kernel SCTP implementation (LKSCTP) has some serious performance problems. On platforms with mature SCTP implementations (FreeBSD, OpenSolaris), SCTP performs *much* better.
See http://sctp.fh-muenster.de/Performance/index.html
He is no doubt getting properly formatted messages from buggy senders with code like this:
/* missing NULL check */
char *user = NULL;
user = lookup_vtext_user_handle(....);
sprintf(email_address, "%s@vtext.com", user);
printf implementations often output "null" or "(null)" when passed a NULL pointer for a %s argument.
This is a common claim, but it is an apples to oranges comparison. No one (including the compiler) dynamically allocates objects in C/C++ when they can place them on the stack instead. Garbage collected languages like Java, on the other hand, require practically everything to be managed on the heap.
In addition, an array of objects on the heap requires only a single memory allocation in C or C++, where Java has to allocate and track each separately. As one luminary once said, "C++ is better because there is less garbage to collect."
That might be acceptable, but the worst part is random application pauses of arbitrary duration for garbage collection. Unless that problem can be resolved, garbage collected languages will be always be a poor match for latency sensitive applications, even where the net throughput is otherwise adequate.
The notification procedures in the DMCA are there to protect the communications provider from liability for copyright infringement.
Since there has been no copying, creation of derived works, or public performance of the items in question, there is no copyright infringement, hence the DMCA doesn't apply.
The real question is does the DMCA protect the ISP from held liable by MonsterPatterns for breach of contract for pulling their service in response to a highly dubious copyright claim.
I agree. The main reason to prohibit dumpster diving is maintaining the privacy of the people generating the trash.
Perhaps a more effective way of helping the desperately hungry would be to pass a law absolving the restaurants, grocers, etc, from liability for giving their leftover food away. It would certainly be in better condition than when dug out of a dumpster hours later.