Slashdot Mirror
IEEE Ethernet Specs Could Soothe Data Center Ills
alphadogg writes "Cisco, HP and others are waging an epic battle to gain more control of the data center, but at the same time they are joining forces to push through new Ethernet standards that could greatly ease management of those increasingly virtualized IT nerve centers. The IEEE 802.1Qbg and 802.1Qbh specifications are designed to address serious management issues raised by the explosion of virtual machines in data centers that traditionally have been the purview of physical servers and switches. In a nutshell, the emerging standards would offload significant amounts of policy, security and management processing from virtual switches on network interface cards (NIC) and blade servers and put it back onto physical Ethernet switches connecting storage and compute resources. 'There needed to be a way to communicate between the hypervisor and the network,' says Jon Oltsik, an analyst at Enterprise Systems Group. 'When you start thinking about the complexities associated with running dozens of VMs on a physical server the sophistication of data center switching has to be there.'"
This is a huge deal for cloud hosts. We aren't a cloud provider, but we do offer similar services on our corporate network. We're using Xen to run over 5000 FreeBSD instances on a singe high-end server. When you're dealing with this many instances, all under constant use, the networking overhead becomes huge.
At first we were using Linux, but it just couldn't offer the throughput that we need. We aren't in a position to acquire more hardware (which is, of course, why we are using virtualization so extensively), so we had to find a better software solution. We found that FreeBSD was compatible with our applications, but had a much more efficient network stack.
Sounds like Cisco wants to sell you more expensive equipment.
Who knows? It might be worth the six-figure price tag. :)
The World Wide Web is dying. Soon, we shall have only the Internet.
Cisco / VMware has done some work in this space, abeit it is a Cisco / VMware solution.... The Nexus 1000V basically provides an overlay to the virtual networking stack from VMware and places it into an appliance with a Cisco CLI. It can then be hooked into the usual Cisco management suspects. The solution makes sense because it also gives back control of the network aspects back to netops, instead of the server ops/virtual ops... http://www.vmware.com/products/cisco-nexus-1000V/
I'll hold out for IEEE 802.1WtfBBQ
When using virtual machines you loose some control and visibility compared to the tradition pizza box server. A physical server is easy to pinpoint, easy to implement ACLs (ethernet/ip), Quality of Service, traffic monitoring or just to shut down a network port. :) Both VEPA and VN-link are technologies that allow you to better seperate different virtual machines on the same physical box.
For VMware, Cisco developed a virtual switch ( YES, a downloadable switch! :) that integrates with VMware ESX 4 that offers all this network security, monitoring goodness. This virtual switch is called the Nexus 1000v and can be downloaded at http://www.cisco.com/en/US/products/ps9902/index.html ( 60-day trial ).
About a year ago the ethernet specifications for data centers already got an extension called FCoE or Fibre Channel over Ethernet ( http://www.t11.org/fcoe ). Basically this allow you to use one ethernet network for both your lan and your storage san. And thus not needing to build out a seperate Fibre Channel SAN.
I'm still waiting for the Data Center IIs.
In my experience this is down to
1. belief that nothing of significant importance is being transmitted via HTTP anyway.
2. complication/expense of setting up secure certificates. It's much cheaper than it used to be, but it's still quite complicated to install on your average server. Wildcard certificates are still a lot more expensive than they need to be (can anyone explain why these are so much more expensive - other than "because they can get away with charging that")
3. inability for HTTPS to work with shared IP addresses. This is probably the major factor, many websites run as vhosts on the same IP, which is great for reducing our 'IP4 footprint' but not very good for HTTPS.
4. Performance is obviously lower for HTTPS than HTTP, so for popular websites the hosting cost differences can be significant. Probably why google has put off shifting gmail to HTTPS for so long...
None of these are justifications for NOT using HTTPS, just the usual problems I come up with when trying to persuade clients to switch to HTTPS.
Jolyon
Please read my Canon EOS tech blog at http://www.everyothershot.com
For a large part, existing standards could still work, if the hypervisors would more fully embrace their role as 'edge switches'. Most problems already are addressed for edge management when the edge is a physical switch via various standards. The issue is that VMWare particularly doesn't bother to implement those, and as a consequence the networking industry has been applying various higher level hacks to gloss over it or work around it without actually having VMWare budge on their implementation.
For example, VLAN membership. This is simple, we have GVRP/MVRP for standards based solution to the problem. There needs to be nothing special to virtualization here
XML is like violence. If it doesn't solve the problem, use more.
Honestly, this entire thing is giving the wrong answer to the wrong question.
Creating huge layer 2 networks and relying on elaborate management systems to try to keep your cloud system running is insane.
I'm currently admining a system with several hundred servers, and a few thousand clients. Each of the servers is on it's own layer 3 network. There is some up front overhead, but ongoing operations of the entire thing, from a network point of view, is a breeze.
DR is built in. It's the ultimate in flexibility. Feel like outsourcing an application? Move the network and VM to the outsourcer, and change the routing, done. Nothing changes from the app or users standpoint. The network becomes virtual with the servers and the applications. I have some servers that have multiple networks assigned to them (run multiple apps).
Layer 2 is evil. STP is evil. VTP is the devil. Don't do evil. Virtualize the network with your servers. Do layer 3.
moo
Your post didn't even mention AOE. ATA-over-ethernet is where the bang for the buck is right now - fast, reliable, but there aren't knucklehead-tolerant toolsets yet so you have to actually know what you are doing if you want to do storage clustering, I/O fencing on shared storage, etc.
AOE is clearly the path forward, though, from my experience. I ripped out all our FC except on the biggest HP-UX hosts, and all of our iSCSI. Been very happy for several years now!
The end game for this is clearly not running all inter-VM traffic out to an external switch and then making a hairpin turn and sending it all back to the host. That is a workaround at best - it is a waste of hardware, a waste of energy, and a waste of bandwidth.
One way or another this is going to be done on the host - either with appropriately enhanced switching support on the NIC or other advances in the CPU and the networking stack. Server CPUs should have inter-VM networking capability built into hardware, not an inter-VM Ethernet switch, but rather something more like an inter-VM RDMA engine. There is no point in using "Ethernet" to communicate between VMs on the same host other than portability.
Breaking inter-VM communications down into little itty bitty packets, running them one by one through a virtual bridge table without the benefit of content addressable memory, and then back up through a virtual ethernet interface is not a particularly efficient use of resources.
Cisco are scared of becoming irrelevant, and VMWare have a third rate product to start with.
Go play with xVM, use this:
http://www.opensolaris.com/learn/features/networking/networkcrossbow/index.html
There is nothing new about software switches. Linux has had one for years. The nice part about the Cisco software switch is the addition of all the extra management and filtering features.
The problem with software switches (or bridges) is that they aren't all that fast - anything in the data path that can't be offloaded to some sort of hardware is going to be relatively slow compared to a hardware switch. In fact, some of the latest Intel server NICs have built in hardware switches for inter-VM communications. I have no idea whether the Cisco VMware switch is able to leverage that. If not, it is going to be a *lot* slower. One might well wonder why Cisco doesn't get into the virtual Ethernet switch on a card market itself.