Slashdot Mirror
How To Build a 100,000-Port Ethernet Switch
BobB-nw writes "University of California at San Diego researchers Tuesday are presenting a paper (PDF) describing software that they say could make data center networks massively scalable. The researchers say their PortLand software will enable Layer 2 data center network fabrics scalable to 100,000 ports and beyond; they have a prototype running at the school's Department of Computer Science and Engineering's Jacobs School of Engineering. 'With PortLand, we came up with a set of algorithms and protocols that combine the best of layer 2 and layer 3 network fabrics,' said Amin Vahdat, a computer science professor at UC San Diego. 'Today, the largest data centers contain over 100,000 servers. Ideally, we would like to have the flexibility to run any application on any server while minimizing the amount of required network configuration and state... We are working toward a network that administrators can think of as one massive 100,000-port switch seamlessly serving over one million virtual endpoints.'"
I hope they have invented something better than ordinary Ethernet cables to wire that ting with.
http://www.intellipool.se/ - Intellipool Network Monitor
I have nightmarish pictures popping into my head of a waterfall of ethernet cables spewing from this with user's ports un-numbered with no network diagrams. People bashing on the server room door in a zombie like state muttering "MRRRHH FACEBOOK!" "TWWIIIITEEEuggggghh" with me inside screeching "NO! NO! I DONT KNOW WHAT PORT YOUR DESK IS! NO! I CAN'T MAKE THINGS GO FASTER!" before curling up in a ball listening to the hum of servers and the lamentations of the users outside the door desperately scratching to get in.
Be you Admins? nay, we are but lusers!
I would seriously hate to be the guy that tripped over that power cable.
On the plus side it would be interesting to time how long it took for the DC's phone lines to melt.
-Matt
(redundant, redundant power. I know, I know)
--- Need web hosting?
I've long been of the opinion that putting more than a few hundred hosts on a single layer 2 network is almost always a bad idea.
What do you do about broadcast storms? How do you prevent some clown from anywhere in that 100,000 machine cloud from poaching another machine's IP address (either maliciously or by an accidental typo)?
Subnets and routers were invented for a reason. Just because you can bridge the whole world together into one massive virtual Ethernet segment doesn't mean you should.
The paper is about adding a layer of addressing so that IP and Ethernet addresses can be moved from one machine to another as instances of virtual machines are migrated around. It's not about the problems of physically building a very large switch. The switch components are mostly stock items.
Have fun replacing it when it fails. In my head I imagine something like this.
Lets see... That's 100,000 ports with 2 LEDs each (link, action/fdx/speed/poe) for a total of 200,000 LEDs. Lets say they use some of the cheapest SMD LEDs on the market. Well use digikey part number 160-1183-1-ND which is a cheap 0603 foot print green LED. At quantity 200,000 that comes out to $12,000 in cut-tape packaging or $9,450 if you buy 210,000 of them in 3,000-qty reels.
Lets say that all of the link LEDs are on 100% of the time and the the activity LED is on 50% of the time. That gives us 150,000 LEDs on at any given point in time. Our example LEDs use 20ma at 2.1V. So 150,000 LEDs at 20ma uses 3Ka. In total, 6.1Kw is burned by the green LEDs.
All that blinking... Damn. I want one NOW!!! More than a girl friend!
I can't just go out and buy 33,334 d-links and turn off DHCP on all but one of them?
For justice, we must go to Don Corleone
Without getting too far into it, their brilliant plan to to insinuate a layer 2 and a half using "pseudo MAC addresses," using a directory service rather than broadcasts. They're hoping they can use this mess to paper over horrific network design.
Yeah, I'll grant you you might be able to cobble this mess together in an academic setting, and sure, you'll even be able to rig some demos that show miraculous increases in speed.
I can guarantee they'll find funding with their promise you'll even able to hire even LESS skilled network admins, meaning Zaboomafoo the Typing Lemur now has a shot at his CCIE.
But, damn, you ignorant twits. Most corporate networks are already mashed together by the most cut-rate cable monkeys they can find. The last thing we need is some half-assed "protocol" that will guarantee even more network designs that are guaranteed to trip and break their necks over the first packet.
He put his boots up on the table and made a face. "The sig," he smirked. "You can waste your life in search of the sig."
This seems to be a solution to a nonexistent problem. A big router, for example a cisco CRS, can be a single node supporting any data center. And it is a router, so there is no need for any exotic solution (L3 inspection on a switch?). It has a max bandwidth of 80Tb/s or 80,000 Gb Ethernet nodes. The beauty is of course that you can configure your entire data center with a single router, which greatly simplifies the network configuration, and makes changes simple.
don't cut it off www.mgmbill.org
I wonder if D-Link has any?
(swoooosh)
And then... let's say 10% of all computers starts up a SMB-share... welcome to broadcast heaven (or hell) :)
They're basically NATting the layer two protocols. Combined with a super spanning tree for the natted addresses they're practically boosting layer two into layer three.
Before I read the paper I was thinking that it would be easier to just run all your services NATted at layer three, even using something like PPPoE (which is how cable networks solve the same basic problem, with something like half a million end-points on the same subnet). I guess it's more efficient to work with the simpler layer two protocols instead.
... they have only needed 1 port! :)
...and when this switch blows the fuses, you have 100.000 servers offline instead of 24... Brilliant!
"Welcome to LEDs Magazine, the leading global information source for the LED community."
Wow, just wow !
Squirrel!
I regularly read Dr. Vahdat's blog. I first got interested in it after reading his paper on Epidemic Routing which can be found in his list of publications here.
If you read his blog post you will see that he accomplishes his goal by creating a hierarchical tree of MAC addresses instead of a simple table. He also states that a large part of the proliferation of MAC addresses in these systems is due to virtual machines. Therefore everyone's nightmares of cabling hell are relatively moot.
Though I haven't contacted him yet, it seems that this solution would require reassigning new MAC addresses such that they can be organized hierarchically as we are accustomed to doing with IP addresses. If this is the case then it seems one would have two choices:
Now, I am not an expert in the details of switches, routing, or NAT so I may have gotten some of the details wrong. But you get the idea.
Wizards, scripts, GUIs and "automagic" are awesome tools. I love my OSPF. I love my Spanning Tree. I love my VTP. I love my Auto speed and duplex settings. I love every tool that helps me take care of tedium and drudgery.
But before you hand these tools to a network designer, they absolutely need to understand HOW and WHY those tools do what they do, lest your network ends up looking like it was built by Mickey the Wizard's Apprentice. Powerful tools require MORE skill on the part of the network admin, not less, because when those tools go wrong, they cause instant damage. Screw up a static route, and one subnet will not ping. Screw up OSPF settings, and multiple subnets may not ping. Screw up VTP settings, and your whole network can go away.
Your argument basically amounts to this. My young son doesn't have the strength yet to cut firewood safely with an ax and saw, so obviously I need to hand him a top-of-the-line Stihl chainsaw.
He put his boots up on the table and made a face. "The sig," he smirked. "You can waste your life in search of the sig."
Yes, I'm going on and one trying to explain the technical side of it to you, but it's starting to feel a little like trying to explain math to a dog.
You're complaining about network complexity when you have no clue about WHY it's complex. Your asking that building networks be "easier," but you have no clue what you even mean by that.
So please, if you're not able to talk to the grownups about the real issues, step away from the keyboard. You're worse than the idiots showing up locked and loaded at the local healthcare discussions.
You're spouting opinions about things you know nothing about.
He put his boots up on the table and made a face. "The sig," he smirked. "You can waste your life in search of the sig."
They're not reducing complexity. They're proposing sandwiching another layer between two and three. It's not going to make things easier to design and troubleshoot. It's going to end up causing more trouble than it's worth. The only people who like this idea are salesguys like you who will have a new buzzword to sell.
But hey, by all means, implement this scheme. You're going to end up needing twice the network engineers you do now. The network explosions it will cause will be epic, the stuff of legend like Mt. St. Helens.
And for the love of Mike, I'm currently working 60-70 hours a week. We're not the Maytag repairmen. Most of us would LOVE to find a better way to do things. I have no doubt that 100 years from now, computer networking will make current schemes look slow and stupid. But those future protocols will still need to connect to the node -- layer one, identify the node -- layer two, and group the nodes together to make them easier to address -- layer three.
Look, I have no doubt you spend your week with your SE wildly gesticulating at you and shouting. I know by the time those frantic shouts get through your ears, it sounds like Charlie Brown's schoolteacher.
Show him some patience. He's trying to wedge some understanding between your ears.
He's not having much luck, apparently.
He put his boots up on the table and made a face. "The sig," he smirked. "You can waste your life in search of the sig."