Slashdot Mirror
New Router Manages Flows, Not Packets
An anonymous reader writes "A new router, designed by one of the creators of ARPANET, manages flows of packets instead of only managing individual packets. The router recognizes packets that are following the first and sends them along faster than if it had to route them as individuals. When overloaded, the router can make better choices of which packets to drop. 'Indeed, during most of my career as a network engineer, I never guessed that the queuing and discarding of packets in routers would create serious problems. More recently, though, as my Anagran colleagues and I scrutinized routers during peak workloads, we spotted two serious problems. First, routers discard packets somewhat randomly, causing some transmissions to stall. Second, the packets that are queued because of momentary overloads experience substantial and nonuniform delays, significantly reducing throughput (TCP throughput is inversely proportional to delay). These two effects hinder traffic for all applications, and some transmissions can take 10 times as long as others to complete.'"
Damn right, they manage flows. It keeps the tubes from clogging.
Duuuurrrrrr.
Sent from your iPad.
So we have a router that does stateful packet inspection and prioritizes traffic based on internal rules. Aren't we supposed to be against this? Because it sounds a lot to me like encrypted packets, UDP, and peer-to-peer, three things that certain well-funded groups have been trying to kill or restrict for awhile, would seem to be the worst-affected here.
#fuckbeta #iamslashdot #dicemustdie
It just makes the packet switching faster. But really, we're talking about the same idea here: datagram networks. Congestion avoidance has been known to be a difficult problem in datagram networks for a long time.
TCP's congestion control algorithm, which causes congestion and then backs off is the real culprit here, and this router does nothing to fix that. The way to fix that is to dump TCP's congestion control and replace it with real flow control in the network layer. That requires lots of memory on intermediaries, because you need all the hosts along the data path to cooperate with each other to communicate about flow control, and that means keeping state. At which point, we're not talking about datagram networks anymore. And that means dumping the other desirable thing about datagram networks: fault tolerance. Packets are path-independent.
Anyway: getting back to TCP's congestion control: his article even says that "During congestion, it adjusts each flow rate at its input instead." Wait, what? "If an incoming flow has a rate deemed too high, the equipment discards a single packet to signal the transmission to slow down." That's how it works right now! The only difference that I can see is that he's being a little smarter about which packets to discard, unlike RED, which is what he's comparing this to. If so, that's an improvement, but it doesn't solve the problem. It will still take awhile for TCP to notice the problem, because the host has to wait for a missed ACK. TCP can only "see" the other host-- it does not know (or care) about flow control along the path. Solving the problem requires flow control along that path, i.e., in the network layer, but IP lacks such a mechanism.
Is it just me, or does the article read like an Anagran ad for the FR-1000?
Cut out the bullshit, and you get a router that prioritizes packets for already-established connections, amirite? Are stateful routers actually a new thing, or can I start mocking the word "flow" now?
a router tampon?
Ah yes, Larry Roberts. He seems to poke his head up every once in a while. From Caspian Networks, and now Anagran. He certainly likes to push flow routing, although it's been shown not to scale in practice.
Why can't we just put a pretty girl on top of it and make the packets go faster.
Seems to work with car advertising and on animals.
Finally! It's about time! I mean, jeez, could it have been more obvious? I've been saying this for years!
But in all seriousness, it would be pretty sweet if it helps streamed media; be it audio, video, games, or some super google chrome plot to launch skynet.
It manages flow of traffic, recognizing when one packet belongs with the others. This sounds wonderful, at least for people trying to inject packets.
I hope these things recognize the evil bit.
Check out my sysadmin blog!
The main players in the routing industry have been working on flow-aware routing for years.
(I'm in the hardware side of our company so I'm not sure where how many and which of the features built on the flow-based architecture are already in the field. But I'm willing to bet a significant chunk of change that that the full bore will be deployed on more than one name-brand company's product line and be the dominant paradigm in routing long before these guys can convince the telecoms and ISPs to adopt their product. No matter how many big names they have on staff - or how good their box is. Breaking into networking is HARD.)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
But seriously, flow management/queuing may be useful at the very edge of the network, like a BRAS. But most provider edge products (Juniper, Ericsson/Redback, ...) already have similar capabilities. Flow management past the edge of a network is pointless, especially for TCP/IP traffic.
I read the article and can't exactly distinguish this from IntServ. What's the difference?
If you're an ISP, you route IP packets. TCP is a payload like any other.
First, routers discard packets somewhat randomly, causing some transmissions to stall.
While it is true that whether or not a particular packet will be discarded is the result of a probabilistic process, it is unfair to call it "random". Based on a model of the queue within the router and estimation of the input parameters the probability of a packet being discarded can be calculated. In fact, that's how they design routers. You pick a bunch of different situations and decide how often you can afford to drop packets, then design a queueing system to meet those requirements. Queueing theory is a well-established field (the de-facto standard textbook was written in 1970!) and networking is one of the biggest applications.
Second, the packets that are queued because of momentary overloads experience substantial and nonuniform delays
You wouldn't expect uniform delays. A queueing system with a uniform distribution on expected number of customers in the queue is a very strange system indeed. Those sorts of systems are usually related to renewal processes and don't often show up in networking applications. That's actually a good thing, because systems with uniform distributions on just about anything are much more difficult to solve or approximate than most other systems.
"Substantial" is the key word here. Effectively the concept of managing "flows" just means that the router is caching destinations based on fields like source port, source IP address, etc. By using the cache rather than recomputing the destination the latencies can be reduced, thus reducing the number of times you need to use the queue. In queueing theory terms you are decreasing mean service time to increase total service rate. Note however that this can backfire: if you increase the variance in the service time distribution too much (some delays will be much higher when you eventually do need to use the queue) you will actually decrease performance. Of course assumedly they've done all of this work. In essence "flow management" seems to be the replacement of a FIFO queue with a priority queue in a queueing system, with priority based on caching.
Personally, I'm not sure how much of a benefit this can provide. Does it work with NAT? How often do you drop packets based on incorrect routing as compared to those you would have dropped if you had put them in the queue? If this was a truly novel queueing theory application I would have expected to see it in a IEEE journal, not Spectrum.
And of course, any time someone opens with "The Internet is broken" you have to be a little skeptical. Routing is a well-studied and complex subject; saying that you've replaced "packets" with "flows" ain't gunna cut it in my book.
this sounds fancy, but the only real improvement is hash-table lookup, everything is already implemented with current generation routers.
and it starts at $30000 a model, ROFLMAO. Thanks, umm , but NO thanks!
Yippee.
Cisco (and probably several others) have done this by default for many many moons now. By way of practical demonstration, notice that equal weight routes load balance per flow, not per packet. What it allows is subsequent routing decisions to be offloaded from a route processor down to the asics on the card level. And don't try to turn CEF off on a layer 3 switch - even a lightly loaded one - unless you want your throughput to resemble 56k.
Among the innovations:
no ram for buffering flows to cope with any temporary overcommitments. Instead it does this:
"Even more significant, the FR-1000 does away entirely with the queuing chips. During congestion, it adjusts each flow rate at its input instead. If an incoming flow has a rate deemed too high, the equipment discards a single packet to signal the transmission to slow down."
Um, discarding a random packet in the middle of my session will indeed slow the flow down, much in the same way as if you shoot me in the knee it will slow me down.
Revolutionary? The concept isn't new. On software based router, we cache route information after the first lookup from the routing table for a certain period of time based on parameters like destination ip address, nexthop and interface. So instead of looking up the route table again, we just look up the cached route. It's called IP Flows and it's way old.
Someone put the spades back. flow routing is SUCH old news... How the heck did this make slashdot?
Routing/Switching based on flows is highly flawed. The article claims that the benefit is due to reduced table lookup based on individual packet content. Instead if the 5 tuple is hashed to a flowid. then the presence of flowid indicates that the flow is already active and will be treated preferentially during a congestion. First of all, if the number of flowids are large then there is no way to store all the different flowids in a scalable and cost effective manner. Which means you associate an eviction clause which can hurt you more with all these complexities. Secondly, there is concept of hardware caching which works better than hashing flowids. Finally, all the classes of flow which are really important, can be protected with class based queuing.
This is called x25 protocol and was the European equivalent of TCP/IP. Since the EU never had an IT industry, US had no problem enforcing it. Interestingly, when the standard war of ATM took place, US-JP made a compromise between two round numbers, midway. 53.
Why?
It would be bad.
I'm fuzzy on the whole good/bad thing. What do you mean, "bad"?
Try to imagine all the packets on your network stopping instantaneously and every router on the Internet exploding at the speed of light.
Total TCP reversal!!
Right, that's bad. Important safety tip. Thanks, Egon.
If routers started doing this, wouldn't torrent clients just start randomizing their port numbers? According to him, different port numbers will get counted as a different "flow". I'd think, if they wanted to do this, they'd at least have to look at IPs, port numbers are easy to change.
Billy Brown rides on. Yolanda Green bypasses Gary White.
"TCP throughput is inversely proportional to delay"
Absolutely wrong, 2Mb/s at 1ms delay gives the same throughput as 2Mb/s at 10ms delay
As long as the window is large enough
He has re-invented the layer 3 switch... now with less jitter and latency because:
The FR-1000 does away entirely with the queuing chips. During congestion, it adjusts each flow rate at its input instead. If an incoming flow has a rate deemed too high, the equipment discards a single packet to signal the transmission to slow down. And rather than just delaying or dropping packets as in regular routers, in the FR-1000 the output provides feedback to the input. If thereâ(TM)s bandwidth available, the equipment increases the flow rates or accepts more flows at the input; if bandwidth is scarce, the router reduces flow rates or discards packets.
So we are going to implement WRED on a per flow basis, get rid of the queuing, and force the tcp stream to scale back it's window size when we run out of bandwidth by dropping a packet out of that conversation...
I mis-spoke, this is a layer 2 and a half switch!
it can do the exact reverse and stop flows it recognizes
and sounds like a type a DRM they will slip by you in the name of "think of the children"
P.S. interesting news is that the pirate party of Canada now states in chat and on the ite that they are not supporting non commercial p2p / fair use / file sharing. Wonder how that's gonna work fer em?
All older cisco equipment worked this way. This was nice, and worked very well for the first router(s) closest to the end customer. However for routers meant to route for large numbers of users this turned out to be a disaster.
Just to give you an idea, this was EOS (end of support) before I turned 10 (look for "netflow routing")
There are a number of very problematic properties :
-> trivial to ddos (just generate too many flows to fit in memory, or generally increase the per-packet lookup time)
-> not p2p compatible (p2p will cause flow based routers to perform at a snail's pace, because they open so much connections)
-> possible triple penalty for every new flow (first a failed flow lookup, followed by a failed route lookup, going to default route)
-> very hard to have a good qos policy this way. A pipe has a fixed bandwidth, and you almost always oversubscribe. Therefore useful policies are very hard to formulate per-flow.
-> if you divide bandwidth per-flow over tcp then a large overload will "synchronize" everything. So let's explain what happens if 3 users are happily surfing about and another user starts bittorrent. Bandwidth gets divided over all the flows, and *every* connection closes, due to timeouts.
There are a number of advantages
-> easy, very extensive QOS is trivial to implement
-> stateful firewalling is almost laughably easy to implement, and very advanced firewalling can be done (e.g. easy to block ssh but not https, just filter on the string "openssh" anywhere in the connection. Added bonus : hilarity ensues if you email someone the text "openssh", and his pop3 connection keeps getting closed)
Here's the deal : a router has to lookup in a table of about 300.000 entries in per-packet switching (excepting MPLS P routers). My PC is, at this moment, opening 331 flows to various destinations, each sending an average of 5 packets (probably a lot of DNS requests are dragging this number down), but you have to keep in mind that a flow-based router has to look up first in the "flow table" AND in the route table (which still has 300.000 entries).
As soon as a flow-based router services more than 1000 machines (in either direction, ie. 100 clients communicating with 900 internet hosts = 1000 machines serviced), it's performance will fail to keep up with a packet-based router. That's not a lot. If a single client torrents or p2p's you will hit this limit easily, resulting in slower performance. 2000 machines and packet-based switching is double as efficient.
So : flow-based routing ... for your wireless access point ... perhaps. For anything more serious than that ? No way in hell.
He's doing it *without* custom ASICs and without TCAM. TCAM is very expensive. I'm not sure this is faster than CEF or the like, but it may very well be cheaper.
"Where quality is like a dead stinking rat - you just can't miss it."
Yeah, I dont see what's radical either... this tech has been out for some time now.
Comment removed based on user account deletion
Fun watching people say this "doesn't work" -- back when I was at Caspian, the real world runs were working quite well at gigabit speed and if memory serves, they had a 10 gigabit line card (this was 2006). The cost was they had to design asics to do this and they were trying to get the same performance out of commodity hardware. It looks like this is the case - which means it's dropped the cost of the equipment significantly.
Where it does improvement over current routing and qos is that it does it on the fly and at wire speeds with an administrator putting in parameters that state the type of performance he wants for "detected flows". In addition, a lot of profiling was done on various traffic to figure out what type of traffic produces what. For instance, the stuff Caspian was selling could identify a VOIP connection on the fly without doing deep packet inspection even if the traffic was encrypted. It did the same with torrent traffic, video traffic, web surfing traffic, im traffic, irc traffic, etc. So, instead of having a deep packet inspection, a router and a switch, you'd get the flow traffic, identify it based on the traffic profile, establish a qos on the flow and then maintain it. It would help against DDOS situations - maintain the current connections that are coming through while establishing new ones as needed. And this is all in the same box.
I don't know what the Anagram folks have managed to do but if they're working off the same model (and probably have a bunch of the same people working on things) the stuff I mentioned should be definitely part of the same equipment.
From the article, it sounds like it's trying to sell the idea of managing traffic (specifically P2P) fairly, which is admirable. I like the idea of a long-lived high-throughput P2P connection being treated the same as a long-lived high-throughput HTTP connection. That seems incredibly fair to me.
However, if they start treating long-lived low-throughput connections differently (more likely in a P2P setting, I'd imagine), then that seems a little unfair.
The very quick workaround would be for P2P clients to set a limit on the duration of a connection, and once that connection expires, to drop the connection and re-establish it. From what the article says, that would get around their "fairness" system. Of course, this assumes the port is part of the hash table key (which is how the article makes it sound.) If it is not (and the "fairness" is based on one IP to another, then that's a different story.
Isn't this something that you can accomplish with OpenBSD packet filter?
"managing flows" unburied a memory of a tampon commercial.
During your period, your flow level can change from one day to the next. That's why Tampax developed the Compak Multipax. You get 3 tampon absorbencies to meet your changing needs, in one convenient package.
the united states is a nation of laws; badly written and randomly enforced -- frank zappa
Will it manage how often she comes to visit? That would be a sure-fire hit with the married men, at least.
Cool, a transfer protocol that adapts what's sent when according to traffic flow. It needs a catchy name.
I suggest Zmodem.
"I may be synthetic, but I'm not stupid." -- Bishop 341-B
And it appears most of the responders didn't. Understandable given it's length. One choice quote I'd like to point out though is:
We designed the equipment to operate at the edge of networks, the point where an Internet service provider aggregates traffic from its broadband subscribers or where a corporate network connects to the outside world. Virtually all network overload occurs at the edge.
This isn't to replace routers, this is supposed to sit between end-users and the rest of the infrastructure so things get throttled before they get into the main router/backbone/wherever it's going.
That's absolutely correct - routers can benefit from understanding what the end-to-end service is doing, but ultimately what they're doing is routing, and it's the endpoints that are doing the TCP or UDP.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
IPv6 is a new version of IP, not a new version of TCP/UDP (though it forces those protocols to change, because the IP addresses are longer.) Yes, there are priority bits, but there are also priority bits in IPv4, and some ISPs support them for traffic within that ISP, but very few support them between ISPs. The important change in IPv6 is of course longer addresses, plus a lot of boundless optimism about "if we're changing IP anyway, we can fix all the problems it has", some of which is warranted but most of it wasn't. The job of a network layer protocol is to figure out where the other end of the connection is and get packets there - some of the things we hoped IPv6 would fix were to make it easier to do aggregation so you don't need exponentially-large routing tables to get there.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Yeah, CEF has done that for a while, but it's pretty dumb as well as fast. This article sounds like what he's doing also provides somewhat fancy queue management to take advantage of the flow information during periods of congestion. It's possible that he's also doing a fancier version of CEF, which can get into trouble if there are too many traffic flows so the router runs out of TCAM, but there's not really enough detail in the article to tell.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Netcraft confirms - OpenBSD is... oh, sorry, wrong thread...
Both of them are identifying flows, but the similarity pretty much stops there.
OpenBSD packet filters are a firewall function, typically used to protect your endpoint from evil traffic. This is a router, deciding which interface to send a packet out when it arrives on a different interface at an ISP backbone location. Packet filters are running at the speed of your computer's interfaces and application processing, typically well under a gigabit/sec - this is trying to run at network-backbone speeds of many gigabits/second.
One thing this router is trying to do is have a $30k device using mostly-standard parts that performs as well as a $300k Cisco box using expensive fancy parts.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks