Slashdot Mirror
Better Networking with SCTP
5-0 writes to tell us that IBM DeveloperWorks has an interesting look at the key features of SCTP in the Linux 2.6 kernel and the ability to deliver multi-streaming. "SCTP is a reliable, general-purpose transport layer protocol for use on IP networks. While the protocol was originally designed for telephony signaling, SCTP provided an added bonus -- it solved some of the limitations of TCP while borrowing beneficial features of UDP. SCTP provides features for high availability, increased reliability, and improved security for socket initiation."
How long do you think it will be before this is adopted into the mainstream?
"I'm going to f***ing bury that guy, I have done it before, and I will do it again. I'm going to f***ing kill Google"
The article makes SCTP sound like it's the greatest thing since sliced bread. It's as fast as UDP, reliable as TCP, and is not susceptible to SYN floods like TCP. It's amazing! It's the fastest, it's the quickest, it's the best!
Really?
Multi-homing with a builtin heartbeat? Youd need a routing table the size of the planet if you wanted to do that over the backbone infrastructure- not to mention gigabytes of wasted bandwidth. Did I miss something?
Second City Transport Protocol!? John Candy would be proud!
Wrong. A connection with a forged source address won't take any more resources than a single incoming packet, a single outgoing packet and the CPU cost of computing a cookie. That's all.
Flooding using the flooder's true address will still work, but it is trivial to block. Sure, having 100000 zombies flood a single destination will put quite a burden and will force the floodee to maintain a huge list of banned addresses, but, a single hash table on the router will alleviate anything except for bandwidth wasted.
This is same as a full TCP connect() flood.
There is a TCP hack named "syn cookies", but this doesn't work very well as TCP wasn't designed to be resistant to SYN floods.
The creatures outside looked from Alt-Right to Antifa; but already it was impossible to say which was which.
Read about SYN Cookies. This is a method of avoiding SYN DOS attacks that has already been implemented in Linux (and probably elsewhere) for a while. I think SCTP just integrates the same concept into the official protocol specification. Once the SCTP server sends the INIT-ACK, it doesn't have to keep track of that association until the client sends a COOKIE-ECHO.
lets change the quote scope a little:
SCTP protects against this type of attack through a four-way handshake and the introduction of a cookie. In SCTP, a client initiates a connection with an INIT packet. The server responds with an INIT-ACK, which includes the cookie (a unique context identifying this proposed connection). The client then responds with a COOKIE-ECHO, which contains the cookie sent by the server. At this point, the server allocates the resource for the connection and acknowledges this by sending a COOKIE-ACK to the client.
Funny how things suddenly makes sense when you read the entire paragraph.
That's total bullshit. MS has nothing to do with this. In fact, no-support of SCTP in MS operating systems will be thee biggets hurdle for the introduction. It's *Linux* that is driving ther adoption !
Disclaimer: I'm using SCTP in my job for several years now.
Utterly incorrect.
If you had only taken ten seconds to check wikipedia's sctp page you would have found it was developed by the Internet Engineering Taskforce's SIGNTRAN working group.
The IETF is an open, all-volunteer standards organization and couldn't be further in spirit from the monopolies you mention.
Give credit where it is due indeed.
(oh and this protocol was defined in 2000 - far later then the telephone signalling I suspect you're thinking of)
My pics.
This is what I always wanted, but never had the time or resources to develop...
What for? Could you give a specific example of an application where such a protocol would be needed? (That is where using TCP or UDP would require a severe overhead in the application layer.)
If you have always wanted such a protocol, you certainly must have had a specific use in mind.
FYI,
p hp
http://www.altoriopreto.com.br/mestrado/index_en.
Seems like there is an implementation for FreeBSD and Darwin (underlying OS used by MacOS X) too, according to this page.
Jumpstart the tartan drive.
A while ago I read the RFC. It is very scary. Multihoming as proposed moves things like name resolution into the kernel.
I will grant SCTP does some neet stuff, the best is that it allows independent non-mutually-blocking streams over one connection. It also has state cookies, yum.
SCTP tries to be all things to all people in one protocol. It reads as though they just decided the whole layered protocol thing was overrated and shoved every new feature into this one layer.
The Linux network stack is having tons of new things lately, SCTP is one, but how it compares with DCCP, which has also been implemented and merged in Linux?
The wikipedia assumes they share some properties, but it's SCTP a better DCCP, or what?
There's also Scalable-TCP, High-Speed TCP, FAST-TCP, BIC-TCP, H-TCP. Each with their own advantages. Check out the site. These guys are doing interesting evaluations. H-TCP is specifically what they work on:
TCP Evaluation Discussion
Interesting plots too
The end result is that TCP is not particularly suited to high-speed networks.
In fact, that's pretty close to how it's done according to SCTP for beginners
I may run Linux myself, but in almost everything I do on my desktop (that isn't itself Linux-related) I am interacting with non-Linux machines. I'm forever "losing out" because I can't receive MSN special features. Sure I could do webcam with what was gnomemeeting (it looks awesome) but does anyone run it? Thankfully now I have friends riding Firefox and one using Jabber (googletalk).
But yes all my friends use windows!
So will such features help Desktop Linux?
Short answer: It might "help Desktop Linux" in general, but it will fix zero interoperability issues and it will do nothing to the problems you listed.
Long answer: You need to learn a few things about network protocols, my friend. Even if SCTP, TCP or UDP had anything to do with your problems, SCTP is not implemented on Windows. Most if not all of the programs you're using use TCP or UDP, and the issues of compatibility you're experiencing have nothing to do with these protocols. The programs you mention have their own protocols that run over TCP and UDP. Seriously, go and learn how to program BSD sockets and you'll understand where TCP and UDP are in the network protocol heirarchy. Once you've done that, maybe you could help out projects like Kopete and Gaim to fix your problems.
I have discovered a truly remarkable proof of this theorem that this sig is too small to contain.
Can someone explain to me how the 4-way handshake is better than the 3-way handshake. I mean, sure the resource allocation has been moved down the process, but a client bent on DOS could still flood the server with INT packets and then just follow up with COOKIE-ACKs, all the while actually not allocating any resources on its side, and you would have the same end result.
Now, if the COOKIE-ACKs required some signficiant processing (like encryption with a public key) then I could understand how this would reduce DOS potential.
...En að Besta Sem Guð Hefur Skapað Er Nýr Dagur
For all its problems TCP/IP is everywhere. This fact has made it the networking technology to use even when it doesn't make technical sense. For example, folks use it in high performance computing and in storage (iSCSI) where there are much better methods available technically. Its commonality (along with ethernet's popularity) often make TCP/IP over ethernet the cheapest solution to many problems (while not the best).
I used to work on InfiniBand where the reliability/congestion detection protocol (Reliable Connected and Link Level flow control in IB terms) are in hardware. This scales to 20 Gbit connections between hosts quite well. Other examples of hardware protocols include myrinet (invented by myricom) and qsnet (from quadrics) and scalable coherent interface current pushed by Dolphin Interconnect. All of these folks struggle to compete with good old TCP/IP over ethernet. Except for the parts of the HPC world, TCP/IP over ethernet wins. In the storage landscape, Fibre Channel, SAS, and SATA seem to be holding out but iSCSI sure is trying.
The performance issue is real though and very few systems can saturate a 10 Gbit TCP/IP etnernet link without massive host CPU overhead. One solution floating around is that instead of trying to make new protocols to replace TCP, we should imitate the competition and put hard work in hardware. TCP/IP offload NICs (TOE) are becoming increasingly more popular. With RDMA technology layered on top of it you get iWARP. For storage you get iSER (ironically from an IB company!). This technology is being adopted by both the MS and Linux camps so it seems to have a good shot. In fact, many of the interfaces used by IB work about as well over iWARP cards. Things like Message Passing Interface, Direct Access Provider Library, Sockets Direct Protocol (SDP), and iSER do not know the difference between iWARP and IB or anyone else.
Software can just post a full size message and it gets sent out the wire without copying, segmentation, timers, resends, or other CPU hogs. This kind of stuff really helps with large messages. With SDP, apps can be made to take advantage of it without changes to the application. MS is also providing a standard way for just TOE NICs without RDMA abilities to work with the OS. Linux doesn't seem to have a standardized way for TCP/IP to be offloaded entirely but is supporting RDMA and SDP.
The things SCTP seems to offer is more explicit understanding of the difference between failure and congestion and multi-home support. This could make load balancing over multiple paths between hosts pretty interesting. The problem I see is that is that it is competing with the established TCP that now has many of its warts fixed with hardware offload. SCTP will still have the issue of a CPU handling segmentation/reassembly, massive amounts of interrupts, timer/retry overhead, etc. It also seems to have a higher overhead for connection establishment (although that is mitigated by being able to send data during the end phases). Is this a solution looking for a real problem? Pehaps not. Does this really have a chance of being taken up? I am not too confident.
-Ack
-- soldack
The first attempt in the IP world to add a protocol of this type was Reliable Data Protocol, in 1984. (See RFC 908). But that never went anywhere. Since then, nobody has really addressed this. There was ISO TP4, but that didn't go anywhere either, althoug it was fully supported in Windows NT.
SCTP has reasonable congestion behavior, like TCP, so it's an improvement over UDP-based protocols in that regard. Moving some UDP-based protocols to SCTP could be a step up. That's where it could be most useful. It might make sense to put remote procedure call type protocols that now use UDP onto SCTP. If a protocol has to do retransmission, it's better to do it at the transport layer than at the application layer.
The "multihoming" thing seems badly placed, because that's not properly a transport layer function. But I haven't really looked at that.
John Nagle
Unfortunately the current mainline Linux kernel SCTP implementation (LKSCTP) has some serious performance problems. On platforms with mature SCTP implementations (FreeBSD, OpenSolaris), SCTP performs *much* better.
See http://sctp.fh-muenster.de/Performance/index.html
That's great, as long as you're the only one that uses your network.
;)
What sucks is when some pompous asshat says "no, I can't provide you with a mechanism to accept incoming connections" or "no, you can't open an outgoing ssh connection".
I feel your pain, however I've found that it makes sense use a simple formula when evaluating an end user request to allow XYZ traffic to traverse a firewall(s) on my organizations network, 1. Is there a business need for it or is it just a user saying "oh this would be nice to have" 2. Is there a more secure and/or functional way to accomplish what the user needs to do? 3. Am I going to open up my network to significant security risks if I do this? do those risks outweigh the business need?
Characterizing any admin that says to you "no you can't traverse the company firewalls with XYZ traffic because doing so represents a significant security risk to the company network" as a "pompous asshat" is a bit unfair don't you think? perhaps you should attempt to look at it from the "pompous asshats" point of view and ponder what your response would be if the positions were reversed, if your answer is "well I'd let any traffic that any user asks for to traverse my firewalls" then there's a very good reason you're not the one making the determination what crosses company firewalls.
SCTP does have an option for using name resolution to do multihoming, however for practical reasons it is almost universally unimplemented. SCTP multihoming works just fine without it. IP address lists for multihoming are exchanged during the standard connection (association) establishment process.
State cookies are not stored on the server at all, but rather are echoed from the client back to the server as a effective means of SYN flood style DoS attack prevention.
SCTP (properly implemented) is radically superior to TCP for a large class of applications, basically anything that needs low latency reliable message exchange. The lack of message boundary information in TCP causes considerable pain for implementers of upper layer protocols - notably RDMA/RDDP and iSCSI. The running solution for efficient hardware implementation of RDMA and iSCSI over TCP involves *inserting* markers every 512 bytes or so in the middle of a data stream so that the receiver can re-synchronize it efficiently.
The primary SCTP RFC is RFC 2960 for those who are wondering.
This sounds somewhat similar to TIPC which we're using in some projects where I work. Like UDP it is message based, but it provides a reliable message transport. It also runs in the kernel as a protocol stack. It does have some differences, though. It is not based on a source and destination, but rather a publish/subscribe mechanism which sounds similar to the SCTP multi-homing support. With the publish/subscribe, one or more clients can indicate that they're interested in a certain service. When that service becomes available or disappears on the node, cluster, or network (depending on the scope of configuration) the client stack will automatically notify it.
It also has the concept of priority in it, so that messages may be prioritized.
Unlike SCTP, however, it does not run on top of IP but is its own protocol that runs directly over the wire, which means that it cannot be routed across an IP network. It is great as an internal embedded messaging protocol, but not as useful when a network is involved.
TIPC is also not connection oriented. There is no connection setup required to send messages much like UDP.
-Aaron
This post is encrypted twice with ROT-13. Documenting or attempting to crack this encryption is illegal.
That's not necessarily fair to MS.
Linux is a very popular platform for researchers to try out new kernel ideas in a real-world system, like networking protocol ideas. This is for a number of reasons (it's well-written, it's easy to hack on, it's open source and doesn't have any weird restrictions, it's free, it's already popular among CS folk, etc.
I'm not familiar with the particulars of SCTP, but just the fact that this is available under Linux (and perhaps that some (all?)) distros make it available by default does not mean that MS is trying to squash the standard. They just have different interests.
Microsoft is a business, and a very conservative one (for the technology industry, at any rate). Unless they have demands from their large corporate customers for a feature, or they think it's going to expand their market, they have little reason to include said feature. It's irrelevant to them whether their own acceptance of something is essential to it catching on -- if you can't make a business case for it, they aren't going to put something into the kernel. That doesn't happen because Microsoft is being particularly antisocial. They're just a business, and they function with certain limitations, as such.
Linux (well, Linus's tree -- I'm sure Novell has their own kernel tree) is built by a bunch of engineers who, as a whole, have no business constraints to worry about. They're interested in making the best technical system possible. Sure, maybe IBM's not going to fund engineers to add Coda support to Linux, but if Linus says "yes, this is technically good", it can go into Linux if someone else wants to write it.
Thus, you have a suit at the highest echelon in one system, and an engineer at the highest echelon (in one tree -- another way in which Linux ensures competition) in the other system.
The case that a libertarian or other hard-core free-markets-no-matter-what advocate would probably make would have is that nobody can have an influential monopoly on the market (not true in this case) and so if someone does something sub-optimal for the customer, they will quickly lose business to the competition.
Now, I don't know whether SCTP is a good idea or not. I'm just pointing out that there are times that having *any* business control the bulk of the market is going to lead to suboptimal handling of consumer interests -- that business need not be Microsoft.
Any program relying on (nontrivial) preemptive multithreading will be buggy.
In a corporate situation, no you wouldn't be a pompous asshat for doing that, you'd just be doing your job.
However if you were an ISP, tasked assumedly with providing connectivity to your customers, and did something similar, then yes, you would be, in my opinion. And there are a fair number of really crummy ISPs who, for one specious reason or another, block various ports and protocols.
And perhaps most unfortunately of all, it's quite common for these ISPs to have regional broadband monopolies, so that a customer doesn't really have the option of just dropping them and switching to a provider that doesn't suck so badly.
"Ladies and gentlemen, my killbot features Lotus Notes and a machine gun. It is the finest available."
SCTP excels at low latency delivery of small messages. TCP's head of line blocking is a serious problem in many applications. SSH tunneling is a good example.
The main advantage of using SCTP over multiple TCP connections is connection establishment time as well as server overhead. You can create an association with hundreds of streams in the roughly the same time as a single TCP connection, with little or no overhead for unused streams. Then when you want to initiate a new non-blocking transaction you can send a message on a new stream without the three-way handshake of an extra TCP connection.
In addition, a single SCTP socket can handle reliably delivered messages on thousands of streams from hundreds of associations. No need to use select()/poll() on a long list of file descriptors.
I spent last semester taking a class about the nuts and bolts "Upper layer protocols" class with one of the leading SCTP researchers, so I've heard a good bit about this protocol. It's quite good, better than TCP in almost every respect. The one problem is (as you probably guessed) overcoming the fact that TCP is ubiquitous and has a gigantic code base.
So I asked - why not have an API for translating TCP calls into SCTP? He told me that this is called a "shim" and that one already exists. He also said the primary area of interest regarding the shim was getting the shim working on windows and deployed by default with windows. That would significantly reduce the gap.
To make laws that man cannot, and will not obey, serves to bring all law into contempt.
--E.C. Stanton
SCTP and Infiniband focus on different areas. IB is largely a high performance HPC / cluster network architecture for LAN applications, where SCTP is a transport protocol designed to operate efficiently under WAN conditions (significant packet loss, high RTTs).
SCTP is a more efficient RDMA/iWARP transport than TCP, but the differential advantage of SCTP over TCP is much lower in a LAN environment due to the low RTTs, so RDDP/TCP dominates so far despite the bizarre marker insertion scheme (MPA). Same goes for iSCSI.
The interrupt issue has largely been solved - on Linux NAPI dynamically switches between interrupt and polled mode to reduce this overhead to negligible levels. Message signalled interrupts also help considerably.
What would be much more helpful (and economical) for iSCSI, SCTP, and RDDP is NIC CRC32C checksum generation. CRC generation is quite expensive in software but trivial in hardware.
SCTP wasn't originally designed for load balancing a single association via simultaneous multi-path transfer (SMT). It can be done, but it requires some loss detection algorithm changes. Someone still needs to develop a option to coordinate this at association establishment time.
One advantage of SCTP over TCP is that on a per stream basis, SCTP connection establishment overhead is much lower than TCP - basically O(1) instead of O(N) in the number of streams.
Congestion control is an area where SCTP is much like TCP. SCTP uses AIMD on a per destination address basis. However any of the the alternative congestion algorithms for TCP would behave similarly with SCTP.
Of course given the additional message boundary information available in SCTP, further improvements could be made.
This concept has potential but there are some problems. First, SCTP does not support half open connections like TCP, so some applications will not work without modifications.
Second, trying SCTP first and then falling back to TCP later causes considerable delay. To fix that problem the shim would need to insert itself in the name resolution process (getnameinfo) so that it could intelligently decide which protocol to try first. Of course name servers would have to carry SRV extension records to indicate that SCTP was supported on a port / ULP basis.
Some of the protocols that could benefit from SCTP include:
now we need to go OSS in diesel cars
SCTP sounds like a kitchen sink solution; it has some nice features and some useless features.
For example, manually opening multiple connections through different interfaces and then having the SCTP implementation figure out which one to send through is nonsense; if the system has multiple routes to the Internet, then that can be taken care of at the IP level.
Similarly, preservation of write boundaries is a useless gimmick that is rarely needed, and when it is needed, can be easily implemented in user code.
The four-way handshake during setup is possibly useful, but you can trivially get the same with TCP in a backwards compatible fashion if you configure your kernel to protect against SYN spoofing.
Altogether, I'm not quite sure what problem SCTP is supposed to solve. SCTP has made its way into some other standards, so it will probably be unavoidable, but it's not a well-designed protocol in my opinion.
Actually, most iWARP/RDMA stuff doesn't have a software interface to TCP at all - the hardware handles not only TCP, but three or more layers on top of it (at least MPA, DDP, and RDMA, plus iSCSI in some cases). That type of interface is not a problem. What is controversial is using TOE for conventional TCP applications using kernel space dispatch.
This is a bit of an end run around the Linux kernel bridging, routing, and filtering layers, which is the primary reason why support for it won't get merged in the kernel socket layer until RNICs can at a minimum do IPtables like IP address filtering and proper dispatch so that some packets can be routed through the kernel layers on an Ethertype and IP protocol / address / port specific basis.
Multithreading should be treated rather like rabid dogs -- something to be avoided if at all possible.
A wonderful post. I usually say it a little more concisely: "can you draw me the complete state machine? no? then you don't know what your code's doing."
What would Lemmy do?