Slashdot Mirror
MIT May Have Just Solved All Your Data Center Network Lag Issues
alphadogg (971356) writes A group of MIT researchers say they've invented a new technology that should all but eliminate queue length in data center networking. The technology will be fully described in a paper presented at the annual conference of the ACM Special Interest Group on Data Communication. According to MIT, the paper will detail a system — dubbed Fastpass — that uses a centralized arbiter to analyze network traffic holistically and make routing decisions based on that analysis, in contrast to the more decentralized protocols common today. Experimentation done in Facebook data centers shows that a Fastpass arbiter with just eight cores can be used to manage a network transmitting 2.2 terabits of data per second, according to the researchers.
Derp. No lag. "solved all your data center network lag issues". "invented a new technology that should all but eliminate queue length in data center networking".
Happy Friday from the Golden Girls.
Good idea, however, its main problem is that it only scales up to a couple of racks and to scale to anything larger it will probably have to sacrifice the zero-queue design principle that it argues for...
Good grief: they appear to have invented a scheduler of some sort. I read the rather thin Network World article and that reveals little.
Nothing to see here - move on!
Now I can see pictures of other's people's food and children so much more quickly...can't wait..>.>
All those moments will be lost in time, like tears in rain.
Nearly any network tech should be faster than Ethernet in certain circumstances. Ethernet is generally good though and appears to be quite good a scaling.
I remember the good old days and the joys of beaconing 8)
If you're going to be a smug windbag, at least think things through!
Instead of using just one arbiter, use a small number of them, connect them, and let them interact with one another to make the routing decisions. Now it's "distributed".
But what you need to realize is that there really is no such thing as "distributed networking". There is just networking. The entire system is the network. Or as my chums back at Sun in the good old days liked to yell out of car windows while driving through San Fran, "The Network Is The Computer".
Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works.
Case in point: ATM To the Desktop.
In a modern datacenter "2.2 terabits" is not impressive. 300 10-gigabit ports (Or about 50 servers) is 3 terbits. And there is no reason to believe you can just add more cores and continue to scale the bitrate linearly. Furthermore... how will Fastpass perform during attempted DoS attacks or other stormy conditions where there are small packets, which are particularly stressful for any centralized controller?
Furthermore.... "zero queuing" does not solve any real problems facing datacenter networks. If limited bandwidth is a problem, the solution is to add more bandwidth -- shorter queues does not eliminate bandwidth bottlenecks in the network; you can't schedule your way into using more capacity than a link supports.
Central planning works rather poorly for humans. Maybe, it will be better for computers, but I remain skeptical.
Oh, and the term "holistically" does not help either.
In Soviet Washington the swamp drains you.
Is your comment some sort of thinly-veiled slur against the BSD and MIT licenses? Is it some sort of pro-GPL huzzah?
Why are you trying to start a software licensing flame war here, when we're talking about networking technologies?
Ok, but the most important question is: did they implement it in Javascript, Go, Rust, Ruby or some other hipster, flavor-of-the-month-language?
This is a really bad idea. No need to elaborate further.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
How different is this to http://www.opendaylight.org/?
Call me when its faster than MPLS. Routing decisions aren't slowing down the world...
Your 300 x 10GB ports on 50 Servers is ... not efficient. Additionally, you're not likely saturating your 60GB off a single server, and you're running those six 10GB connections per server to try to eliminate other issues you have, without understanding them. You're speed issues are elsewhere (likely SAN or Database .. or both), and not in the 50 servers. In fact, you might be exasperating the problem.
BTW, our data center core is running twin 40GB connections for 80 GB total network load, but were not really seeing anything using 10GB off a single node yet, except the SAN. Our Metro Area Network links are is being upgraded to 10GB as we speak. The "network is slow" is not really an option.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
I thought Nginx was created by Igor Sysoev?
This hook doesn't even have any bait.
Where in the comment did you read anything that suggested it would be about licensing? Or were you unaware that Berkeley and MIT are actual, real-world institutions, and it's possible to use those names without necessarily referring to the corresponding open source license.
File under 'M' for 'Manic ranting'
Maybe because that is what Token Ring did! Just sayin'!
"My immediate reaction is "WTF? What kind of moron doesn't make things 64-bit safe to begin with?" Linus
In fact, you might be exasperating the problem.
I hate it when my problems get angry, it usually just exacerbates things.
Nearly any network tech should be faster than Ethernet in certain circumstances. Ethernet is generally good though and appears to be quite good a scaling.
The key word, there, is scaling.
It looks like this is meant to make the network more efficient within a data center that handles a high volume of traffic, including high traffic spikes, by receiving a network time slot request from the end point (i.e. software running on a UNIX server) and sending a response that schedules packets to arrive just-in-time along a specific path to avoid queuing.
However, there is a less complicated way of achieving the same goal: Scalability - Increase your switch and server up-link bandwidth to eliminate congestion and queuing.
Yes, it costs money to add network capacity. But the big question is which would cost more? Adding capacity? or installing a pair of servers, rolling out software clients to all of your endpoints (servers), and supporting the system? Personally, I'd rather add network capacity and be done...
Fastpass is their trademark (since 1999), after all.
http://en.wikipedia.org/wiki/Disney%27s_Fastpass
I for one welcome all but our new Fastpass &,dash; static scheduling overlords.
CLI paste? paste.pr0.tips!
... my Candy Crush Saga.
It little behooves the best of us to comment on the rest of us.
And big network service provider will implement it to the detriment of their revenue (think Comcast and Netflix). Riiiiiight.
ELOI, ELOI, LAMA SABACHTHANI!?
...are they trying to say that "Arbiter macht frei"?
No he's saying that developers from Berkeley and MIT have used a lot of drugs. A whole awful lot. Like the drugs you should be on.
This is about zero in-plane queuing, not zero queuing. There is still a queue on each host, the advantage of this approach is obvious to anyone with knowledge of network theory (ie. not you). Once a packet enters an ethernet forwarding domain, there is very little you can do to re-order or cancel it. If you instead only send from a host when there is an uncongested path through the forwarding domain, you can reorder packets before they are sent, which allows, for example, to insert high-priority packets into the front of the queue, and bucket low priority traffic until there is a lull in the network.
Bandwidth is always limited at the highend. Technology and cost always limits the peak throughput of a fully cross-connected forwarding domain. That's why the entire internet isn't a 2 Billion way crossbar switch.
Furthermore, you can't install 6x 10-gigabit ports in a typical server, they just don't have that much PCIe bandwidth. You might also want to look at how much a 300 port 10GigE non-blocking switch really costs, multiply that up by 1000x to see how much it would cost Facebook to have a 300k node DC with those, and start to appreciate why they are looking at software approaches to optimise the bandwidth and latency of their networks with resources that are cost-effective, considering their network loads like everyone else's network loads never look like the theoretical worst-case of every node transmitting continuously to random other nodes.
Real network loads have shapes, and if you are able to understand those shapes, you can make considerable cost savings. It's called engineering, specifically traffic engineering.
-puddingpimp
Your 300 x 10GB ports on 50 Servers is ... not efficient. Additionally, you're not likely saturating your 60GB off a single server, and you're running those six 10GB connections per server to try to eliminate other issues you have, without understanding them.
You haven't worked with large scale virtualization much, have you?
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
Nah. They put MPLS logic-- deterministic routing by knowing the domain into an algorithm that optimizes time slots, too.
All the hosts are know, their time costs, and how much crap they jam into wires. It's pretty simple to typify what's going on, and where the packet parking lots are. If you have sufficient paths and bandwidth in and among the hosts, you resolve the bottlenecks.
This only works, however, if and when the domain of hosts has sufficient aggregate resources in terms of path availability among the hosts. Otherwise, it's the classic crossbar problem looking for a spot marked ooops, my algorithm falls apart when all paths are occupied.
Certainly it's nice to optimize and there's plenty of room for algorithms that know how to sieve the traffic. But traffic is random, and pathways limited. Defying the laws of physics will be difficult unless you control congestion in aggregate from applications where you can make the application become predictable. Only then, or you have a crossbar matrix, will there be no congestion. For any questions on this, look to the Van Jacobsen algorithms and what the telcos had to figure out, eons ago.
---- Teach Peace. It's Cheaper Than War.
And it's an expensive single point of failure, which no one knows how to program and has never been tested under badly configured local networks.
This is about as stunning as the MIT built network router that thoroughly optimized BGP maps, unless there was a router loop, in which case it would hard crash the router and require a power cycle. See, adding the checks to verify no router loops actually made it considerably slower than the existing technologies.
Your 300 x 10GB ports on 50 Servers is ... not efficient. Additionally, you're not likely saturating your 60GB off a single server,
It's not so hard to get 50 gigabits off a heavily consolidated server under normal conditions; throw some storage intensive workloads at it, perhaps some MongoDB instances and a whole variety of highly-demanded odds and ends, .....
If you ever saturate any of the links on the server then it's kind of an error: in critical application network design, a core link within your network being saturated for 15 seconds due to some internal demand burst that was not appropriately designed for is potentially a "you get fired or placed on the s***** list immediately after the post-mortem" kind of mistake. Leaf and spine fabrics which are unsaturatable, except at the edge ports: are definitely a great strategy to approach sizing of core infrastructure --- from there most internal bandwidth risk can be alleviated by shifting workloads around.
Latency performance seriously suffers instability at ~60% or higher utilization, so for latency-sensitive applications especially: it would be a major mistake to provision only enough capacity to avoid saturation, when micro "bursts" in bandwidth usage are the reality for real-world workloads.
An internal link with peak usage of 40% or higher should be considered in need of being relieved, and a link utilized 50% or higher should be considered seriously congested.
no,
not many people have.
Meanwhile some traffic lights in my town are green a whopping 12 seconds on a backed up to hell road before they turn color for a minute to let 5 cars meander past on the perpendicular rout.
You can get consumer hardware with 40 PCIe 3.0 lanes that run right into the CPU, wouldn't that be enough PCIe bandwith?
Exactly. Toss in the added bonus of ignoring prior work because the problem is "new," compare it only to a sub-par baseline and voila, revolutionary paper.
Papers like this are exactly why I usually shake my head at SIGCOMM.
barely any link layer is shared today though, hubs are dead and gone (or at least hidden in cupboards by people who think they need them for packet sniffing). MIT's solution seems eerily like a Chris Christie plan to eliminate congestion on the NJ turnpike.
Nullius in verba
This paper shows no tangible benefit other than a slight decrease in TCP retransmits, something that the authors never test if it shows any real benefit.
Crucially, this system is not "zero queue". They simply move queuing to the edge of the network and in to the arbiter. Notice that there is no evaluation of the total round trip delay in the system. The dirty secret is because it's no better, especially as the load increases, since the amount of work that the arbiter must do grows exponentially with both the size and utilisation of the network this is guaranteed to have scalability issues. Finally, the evaluation at "Facebook" used only one rack and never tested the path selection, so in reality this paper has shown nothing and demonstrated no benefit.
42 terabit/s on one core
https://translate.google.com/translate?hl=da&sl=da&tl=en&u=http%3A%2F%2Fing.dk%2Fartikel%2Fdtu-slaar-verdensrekord-datatransmission-169478
two years ago petabit /s one 12 cores
https://translate.google.com/translate?hl=da&sl=da&tl=en&u=http%3A%2F%2Fing.dk%2Fartikel%2Fdtu-professor-slar-rekord-sender-en-petabit-i-sekundet-gennem-en-optisk-fiber-132652
2.2TBit/sec is just under 40 ports which is just over 2 switches...
It will only take one extra management processor (8 cores) to manage two switches... Get back to me when you can drive 100TBit/sec with one core
PS - is there extra compute needed on the management plane of the edge switches here? I don't think so but it is hard to tell
I have mod points and I am not afraid to use them
I hate it when my problems get angry, it usually just exacerbates things.
I hear most problems can be kept reasonably happy by properly acknowledging their existence and discussing potential resolutions.
Problems tend to be more likely to get frustrated when you ignore them, and anger comes mostly when you attribute their accomplishments to other problems.
Nearly any network tech should be faster than Ethernet in certain circumstances.
We have 10GbE in the data center. We also have a 376lb, 38 year old network tech named Bob in the data center. Maybe if I had a dozen hot Krispy Kreme's on the far side of the room, Bob might just beat the packets. But I doubt it.
You haven't worked with large scale virtualization much, have you?
In all fairness.. I am not at full scale virtualization yet either, and my experience is with pods of 15 production servers with 64 CPU Cores + ~500 Gb of RAM each and 4 10-gig ports per physical server, half for redundancy, and bandwidth utilization is controlled to remain less than 50%. I would consider the need for more 10-gig ports or a move to 40-gig ports, if density were increased by a factor of 3: which is probable in a few years, as servers will be shipping with 2 to 4 Terabytes of RAM and run 200 large VMs per host before too long.
It is thus unreasonable to pretend that large scale virtualization doesn't exist or that organizations are going to be able in the long run to justify not having large scale virtualization, OR moving to a cloud solution which is ultimately hosted on large scale virtualization.
The efficiencies that can be gained from a SDD strategy versus sparse deployment on physical servers are simply too large for management/shareholders to ignore.
However: the network must be capable of delivering 100%.
Perfectly content to overallocate CPU, Memory, Storage, and even Network port Bandwidth at the server edge. However the network at a fundamental layer has to be able to deliver 100% of what is there --- just like the SAN needs to be able to deliver within a degree of magnitude the Latency/IOPS and Volume space size that the vendor showed as the capacity of the SAN --- we will intentionally choose to assign more storage than we actually have, BUT that is an informed choice, the risks simply become unacceptable if the lower level core resources can't make some absolute promises about what exists and the controller architecture forces us to make an uniformed choice, OR guess about what our own network will be able to handle affected by the loads created by completely unrelated networks or VLANs outside our control, E.g. perhaps another tenant of the datacenter.
This is why a central control system for the network is suddenly problematic. The central controller has suddenly removed a fundamental capability of the network to be heavily subscribed, fault-isolated within a physical infrastructure (through Layer 2 separation), and tolerate and minimize the impact of failures, if designed appropriately.
Which is getting harder with decentralized network fabrics. Gotta snoop those conversations.
In addition to eliminating the queue latency, it could enforce fairness among customers.
Which would make the Netflix/Comcast who should pay for what debate moot.
Good for customers and Net Neutrality.
Bad for double charging strategies.
In the current 'competitive' environment, I won't hold my breath.
it is
No, I don't think so. RMS worked at MIT for over a decade.
Even if it is just a data center technology, a key placement might be SAN switching. Currently, much of "the cloud" or rather server cluster based computing suffers heavily from latency and you can never have enough storage bandwidth issues.
if that component goes down in flames you are screwed!
While it is possible to fill your Data pathways up. Aggregate data is not the same as Edge Server data. In the case described above, s/he is running 300 x 10GB on 50 Servers. Okay, lets assume those are 50 Blades, maxed out on RAM and whatnot. The Only way to fill that bandwidth is to do RAM to RAM copying, and then you'll start running into issues along the pipelines in the actual Physical Server.
To be honest, I've see this, but only when migrating VMs off host for host Maintenance, or a boot Storm on our VDI.
Agent K: A *person* is smart. People are dumb, stupid, panicky animals, and you know it.
To be honest, I've see this, but only when migrating VMs off host for host Maintenance, or a boot Storm on our VDI.
Maintenance mode migrations are pretty common; especially when rolling out security updates. Ever place two hosts in maintenance mode simultaneously and have a few backup jobs kick off during the process?