A Peek At Google's Software-Defined Network

CowboyRobot writes "At the recent 2013 Open Networking Summit, Google Distinguished Engineer Amin Vahdat presented 'SDN@Google: Why and How', in which he described Google's 'B4' SDN network, one of the few actual implementations of software-defined networking. Google has deployed sets of Network Controller Servers (NCSs) alongside the switches, which run an OpenFlow agent with a 'thin level of control with all of the real smarts running on a set of controllers on an external server but still co-located.' By using SDN, Google hopes to increase efficiency and reduce cost. Unlike computation and storage, which benefit from an economy of scale, Google's network is getting much more expensive each year."

Comments

centralized = fault-tolerant?

[Anonymous Coward]

"it provides logically centralized control that will be more deterministic, more efficient and more fault-tolerant."

I'll agree with deterministic and efficient, and perhaps even less likely to fault, but more fault-tolerant seems like a stretch. SDN might get you better fault-tolerance, but that is not because the control is centralized. I suspect the control has more information about non-local requirements and loads, and that can get you better responses to faults. That happens because the controllers can communicate more complex information easier, since that is pure software, not because its centralized. You can have these fault tolerance gains via non-centralized SDN too.

Re:centralized = fault-tolerant?

[bbn]

Compare it to the alternative such as the good old spanning tree protocol. You have a number of independent agents who together have to decide how to react to a fault. This is complex and requires clever algorithms that can deal with timing issues and what not.

With a centralised controller the problem is much easier. One program running on one CPU decides how to reconfigure the network. This can be faster and possibly find a better solution.

Of course you need redundant controllers and redundant paths to the controllers. Apparently Google decided you need a controller per location.

Re:centralized = fault-tolerant?

[bill_mcgonigle]

With a centralised controller the problem is much easier. One program running on one CPU decides how to reconfigure the network. This can be faster and possibly find a better solution.

I can see how centralizing the control can be easier. But if the history of Internet networking has taught us anything, we should expect somebody to come up with a more clever distributed algorithm (perhaps building on OpenFlow) that will make SDN's a footnote in history while the problem gets distributed out to the network nodes again, making it more resilient.

That's not to say that trading off resiliency for performance today isn't worthwhile in some applications.

Re:centralized = fault-tolerant?

[Anonymous Coward]

When I was asking around about what's the main thing about SDN, I got back because it's programmable. If there is a bug 10 years from now, it can be fixed. With a regular router, you're stuck hoping for support from the manufacturer.

Re:centralized = fault-tolerant?

[Alomex]

ut if the history of Internet networking has taught us anything, we should expect somebody to come up with a more clever distributed algorithm

The internet has moved from centralized to decentralized to centralized again. It is not the case that it has moved one-directionally towards a distributed system. Currently big parts of the internet are centrally managed (e.g. SuperDNS/GoogleDNS, IBGP, MPLS routing, most of network provisioning).

Current view is that centralizing BGP would be a "good thing" (TM).

Re:centralized = fault-tolerant?

[AK Marc]

Networks connected to the Internet being centrally managed was universal. You are right for non-Internet things (NNTP), but DNS is just as distributed as always, and MPLS doesn't cross network boundaries, and BGP *is* somewhat centralized, as it always was. You can't just make up your own AS to use (well, you can, but only from the private range).

There may be a move to concentrate traffic in fewer large networks, but that's not the same as the Internet getting more central management.

Re:centralized = fault-tolerant?

[Alomex]

but DNS is just as distributed as always

Google DNS is centralized.

BGP *is* somewhat centralized, as it always was

The change is that now many organizations drop centrally computed routing tables on the routers as opposed to the OSPF+manual tweaks that used to dominate before.

Re:centralized = fault-tolerant?

[AK Marc]

Google DNS is centralized.

Well, yes. Every network has "centralized DNS" it's how DNS operates. That this is a sudden and startling discovery to you indicates nobody should listen to you.

The change is that now many organizations drop centrally computed routing tables on the routers

That's always been relatively common. Especially if you have only one or two peers, dynamically learning the entire Internet routing table was a massive waste of resources. Many holders of a single class-C run BGP to advertise their route, not to learn routes. They default out, and advertise, so that their block is reachable if a link goes down, without concern of link optimization for both-links-up, and symmetrical routing more important than load balancing or optimum performance.

Re:centralized = fault-tolerant?

[Alomex]

It is clear you do not know what Google DNS is. It is not the DNS that serves the "google network" but a global provider of DNS services for all and people are encouraged to use it instead of their local DNS. This makes your comment

That this is a sudden and startling discovery to you indicates nobody should listen to you

rather ironic.

That's always been relatively common. Especially if you have only one or two peers, dynamically learning the entire Internet routing table was a massive waste of resources.

I'm talking AS level organizations including internal routers as well as border routers.

Re:centralized = fault-tolerant?

[AK Marc]

It is clear you do not know what Google DNS is. It is not the DNS that serves the "google network" but a global provider of DNS services for all and people are encouraged to use it instead of their local DNS.

Ah yes, the traditional "you must not have all the information, or you'd agree with me" argument. It's proof your logic is flawed, not proof of my ignorance. You do realize that "back in the day" there were people encouraging others to use things like 198.6.1.3, the DNS server for the largest (by volume, not reach) and fastest growing (by $ per day spent on infrastructure) ISP on the planet, rather than local ones because local ones were much more prone to failure than link failure to 198.6.1.3, right? You speak as if cross-provider DNS is something you discovered for the first time yesterday. Others of us have used it for 20+ years. Your problem is that you just don't get it. Everything you talk about being "new" has been done before.

It's like virtualization. We had that in the '60s. It was called a "mainframe" then. Then we had PCs. Then we had "terminal servers" which was more virtualization. Then PCs/tablets again. Nothing is new. It's cyclic, and if you are dumb, you might think the next big thing is new, but if you aren't dumb, you recognize it as a re-marketing of something that's been done multiple times in different ways in the past 50 years.

Re:centralized = fault-tolerant?

[Alomex]

You are funny trying to play the I'm older and wiser card. You are likely to lose that one too.

And all you prove with your 198.6.1.3 example is what I said in my original posting: there have been waves or centralization (such as that one) and waves of decentralization and back again (e.g. Google DNS).

Re:centralized = fault-tolerant?

[AK Marc]

I never played the "I'm older and wiser" card. I played the "you're dumb" card.

How can you have a software defined network?

[Viol8]

A network is physical infrastructure - software isn't going to be rerouting cables or installing new wifi nodes anytime soon.

If all they mean is routing tables are dynamically updated then how is this anything new?

This isn't a troll, I genuinely don't see where the breakthrough is.

Re:How can you have a software defined network?

[lobiusmoop]

You're missing the point. The summary describes it as a 'Software Defined Network Network', a true innovation.

Re:How can you have a software defined network?

[DarkOx]

Its not what they are doing here exactly but there is not reason you can't have a logical topology over top of a physical one. Actually its very useful, especially when combined with a virtual machine infrastructure. Perhaps you want to have two machines in separate data-centers to participate in software NLB they need network adjacency, for example, yet I doubt you want a continuous layer two link stretched across the country. Sure if its just two DCs maybe a leased line between them will work, what if you have sites all over the place and potentially want to migrate the hosts to any of them at any time? That would allow for maintenance at a facility, or perhaps you power on facilities during off peak local electrical use, and migrate your compute there?

People are doing these things today but once you get beyond a singe VM host cluster it gets pretty manual. With admins doing lots of work to make sure all the networks are available where they need to be hard coded GRE tunnels, persistent ethernet over IP bridges, etc. They all tend to be static, minimal overhead when not in use sure, but overhead and larger attack surface non the less. A really good soup to nuts SDN might make the idea of LAN and WAN as separate entities an anarchism. Being able to have layer two topology automatic wherever needed would be very cool.

Re:How can you have a software defined network?

[bbn]

There is no routing as such. For each new "flow" the switch needs to ask a computer (controller) what to do. The controller will then program the switch with instructions for the new flow.

You claim that the flow table is just a glorified routing table. Maybe it is but much more fine grained, you can match on any fields in the IP packets, including layer 2 and 3 such as MAC, IP, port numbers, IP TCP packet types (syn packets) etc. Also you can mangle the packets, for example modify the MAC or IP address before forwarding the packet.

With this you can build some amazing things. The switch can be really dumb and yet it can do full BGP routing: RouteFlow: https://sites.google.com/site/routeflow/

The other canonical use case is virtualisation. No it will not be rerouting physical cables. But it can pretend to do so. Combine it with VMs you can have a virtual network that can change at any time. If you migrate a VM to another location, the network will automatically adapt. And still the switches are dumb. All the magic is in the controllers.

Before OpenFlow you would need to make a vlan (or MPLS). When moving the VM to a new location, you would need to reconfigure a number of switches to pass around this vlan and there is no standard protocol to do so.

OpenVSwitch supports OpenFlow so you can pretend your virtual network with virtual switches includes the VM host itself: http://openvswitch.org/

Re:How can you have a software defined network?

[swb]

Sometimes it seems that SDN is just a new dress on an old pig, sometimes it starts to make sense.

When I'm feeling enlightened or charitable about the concept I envision it as an encapsulation system for layer 2 on layer 3, allowing layer 2 networks to be created independent of the physical constraints of actual layer 1/2 topologies.

I imagine the goal is to define a layer 2 switching domain (ports, VLANs, etc) and connect systems to it regardless of how the systems are physically connected or even located. This all seems fine and dandy -- draw a network diagram, connect systems, voila!, you have a SDN.

But when you start to actually think about it seems kind of problematic...

It seems hard to separate SDN implementation from virtualization, though. If I have a SDN, how do I connect VMs to it if the SDN isn't part of the virtualization environment? Do you install a virtual network adapter in your OS to configure SDN network membership?

Or is it a switch-level system? I feel somewhat less enthusiastic about this as a concept as it just seems like more configuration for the same basic product (VLAN or VLAN trunk membership), with benefits only to really the largest and most complex networks with maximum bandwidth trying to re-solve problems sort of already solved other ways (like LAN bridging over WAN links).

Since encapsulation appears to me to be an inherent part of it, I also worry about performance but I suppose everyone in the SDN world are go-fast, low drag operators on fully meshed, aggregated 10 gig ethernet end-to-end and doesn't care about encapsulation penalties.

And then there's my inherent skepticism about the value payoff relative to the level of complexity added, as well as asking isn't that why we have layer 3 protocols? To define networks above and beyond their layer 2 memberships?

Re:How can you have a software defined network?

[jader3rd]

And then there's my inherent skepticism about the value payoff relative to the level of complexity added, as well as asking isn't that why we have layer 3 protocols? To define networks above and beyond their layer 2 memberships?

What once was old is now new again.

Re:How can you have a software defined network?

[Anonymous Coward]

what is the payoff? no Cisco support contracts on gazillions of switches and interconnects (which support really is purchased for firmware updates--support/replacement is or should be quite rare) this will have a very fast payoff, despite the initial complexity curve.

a lab should be quite cheap for POC testing of production changes.

Re:How can you have a software defined network?

[bbn]

"it's all stateless" - no not exactly. First OpenFlow has counters and flow rules can apply to those counters. You can use this to rate limit flows or you can use it to sample packets (copy every 500th packet etc). Or to load balance.

But most important, the whole point of OpenFlow is that you do not upload the whole set of rules to the switch. Indeed the actual rules might be too complex for the switch to hold or to compute.

Take the BGP implemented by RouteFlow as an example. The global BGP table has about half a million routes. Your cheap OpenFlow enabled switch might not be able to hold half a million OpenFlow rules. Is all lost? No, because you need to upload all the routes, in fact you will upload no routes. Instead the first time the switch has a IP packet with a new destination, it will ask the controller. The controller will consult the BGP tables and program the switch with a new rule. Now the switch knows how to deliver for this destination. In the process the switch might need to flush an older rule to make space for the new rule in memory. This is made possible by the before mentioned counters - there are also timers. So we can remove the least used rule and avoid removing any recently active rules.

OpenFlow turns cheap switches into advanced devices that can solve many tasks that before required expensive equipment. The cheap switch can become your BGP border router. It can be your HTTP load balancer. It can be your carrier NAT device. It can support the full range of protocols even if the maker did bother to implement them in firmware.

Re:How can you have a software defined network?

[bbn]

Small typo: "No, because you need to upload all the routes, in fact you will upload no routes" should be "No, because you need NOT to upload all the routes, in fact you will upload no routes".

Re:How can you have a software defined network?

[swillden]

Translation: Google Big.

Yep. And there comes a point when you're scaling up that quantitative differences become qualitative differences that demand completely different solutions to the old problems.

Translation: Firmware Is Magic.

No, firmware is static, and the code it contains must fit in limited capacity storage devices and run on low-end CPUs, unless you want to pay big money for your switches. Much better to make the switch firmware simple and the switches cheap, and put your logic in a few much more powerful machines with visibility into the bigger picture.

Re:How can you have a software defined network?

[bbn]

Can you point to any cheap switch that can hold 500.000 BGP routes in the dataplane? I didn't think so.

You are also missing the point: Do you really want to pay extra for software features? Software that has been done way better in open source controllers?

A Juniper router with 6x 10 Gbit/s is $50,000. An OpenFlow enabled switch with four times as many 10 gig ports is only one tenth of that. I do not know where you work, but in my shop that is some savings that we will take.

Re:How can you have a software defined network?

[evilviper]

A network is physical infrastructure

No it isn't. Sure, there's one ethernet cable connected from a server to the rack switch, but even there, the packets coming in could have hundreds of different VLAN tags on them.

Everywhere else, you have multiple redundant links from everything to everything else, and deciding which one to use for each packet is the complex part.

Re:How can you have a software defined network?

[bbn]

I will give you that the OpenFlow system is stupid in some ways. For example I can push a MPLS label on a packet, but I can not push a LISP header. Why not? Because they made separate instructions such as "push VLAN label" and "push MPLS label" - instead of a generic "push N bytes".

OpenFlow is two things. It is a language for the data plane. Not much different from what you are asking for. It is not turing complete, probably by design. So you can not make the data plane do just anything, but on the other hand you can guarantee that it will not do an infinite loop or use up all memory. It is possible to have OpenFlow in the data plane and still be able to guarantee that your data plane will switch at line speed. That would be impossible with a stronger turing complete data plane language. Yet they could have made it more generic, like having a generic push and pop.

OpenFlow is also a protocol. Currently we think the controller speaking the OpenFlow protocol must be external from the switch. But nothing prevents a switch manufacturer from granting access to the build in control plane computer in the switch. If it is just a Linux computer, as you say, I could just login and upload my controller software there. My software would still speak OpenFlow with the data plane, because that is the standard for how to program data planes. Also it would allow my program to be the same regardless if it is being used on a switch that allows uploading controller software or if it is run on an external server.

One thing is for sure thou - the big players like Cisco and Juniper do not want to go in this direction. You say that 50k juniper router providers lower latency than the cheap OpenFlow-switch - but that is just BS. They will switch at line speed and if the open source world gains access to program the things, there will be nothing to sell the expensive hardware on. We will be down to the pure specs of the hardware. Right now I see a lot of line rate 10G switches coming out at a very attractive price point - some of those made by these same brands but artificially limited in the software.

Re:How can you have a software defined network?

[AK Marc]

Do you know what VLANs are? It's a logical imposition on a physical network. SDN is an extension of that idea. No reason you couldn't put every computer in its own VLAN, with ARP and DHCP forwarded to the correct server, or configure a full mesh of connections and disable all but the best route spanning tree style, with your explicit rules, no 3rd party decisions required.

Re:How can you have a software defined network?

[AK Marc]

You say that 50k juniper router providers lower latency than the cheap OpenFlow-switch - but that is just BS.

That's proof you don't know what you are talking about. The expensive router/switches have dedicated ASICs for ports. RAM and processing at the port to get the packet in, processed, and back out as soon as possible. Pulling something in a "linux" router (often a PC with more networking cards) and you have to pull it through the cards, through the bus/MB to the CPU, process it, and send it back to the card for exit.

Re:How can you have a software defined network?

[bbn]

How the hell did you manage to conclude anyone here was talking about PCs with networking cards? The "cheap" switches I am talking about are products such as Juniper E4550 that got 32x 10G and 960 Gbps bandwidth for $19k. Compare that with Juniper M320 which is twice as expensive with only half as many 10G ports and 320 Gbps bandwidth.

Sure the M320 can do more in the data plane, but people are using it for stuff that the E4550 would do just fine, if the software would allow it.

Or you could go for a HP 5820X-24XG-SFP+ switch with 24x 10G and 488 Gbps bandwidth for just $5k.

If you believe the HP 5820X is a "linux router just a PC with more networking cards", then you are truly an idiot.

Re:How can you have a software defined network?

[AK Marc]

Next clue you are clueless is that you are comparing L3 switches and routers.

Re:How can you have a software defined network?

[bbn]

No that is the point of OpenFlow. The switches becomes routers.

Re:How can you have a software defined network?

[AK Marc]

I thought it was to do "new" and "innovative" things, not the same old thing we've been doing for almost 50 years, but, this time, at a lower cost!

Re:How can you have a software defined network?

[bbn]

Please elaborate on what you mean by stateless. I already told you how it is not stateless.

Re:How can you have a software defined network?

[bbn]

An OpenFlow switch will:

Update counters and timers. Make decisions based on those counters and timers. Support multiple queues with different limits of delay etc. QoS. Rewrite source and destination IP address and UDP/TCP port numbers allowing the switch to do NAT without querying any external entity on a per packet basis. Add and remove VLAN, MPLS, etc tags, modify the tags, modify the MAC and much more. Automatically drop flow rules by certain events such as the last packet in a TCP flow or by counters, timers. Allow rules that recognise a missing rule and query the controller to add the rule.

It will basically do anything routers can do in the data plane without querying a controller.

I fail to see by what property you can call the above for "stateless". On the contrary it is a little programming language with state updates such as counters, timers and queue lengths and the ability to make decisions based on those.

I recognize your belief that the controller software should run an a CPU in the same chassis as the data plane. This however does not necessary make the controller any faster reacting. Many switches only have limited bandwidth between data plane and control plane. It is assumed that most of the brunt work will be done in the data plane and that any work that needs to go through control will have higher latency and less bandwidth. It is this property that makes it possible to move the control plane out of the chassis.

Is it perfect? No but it is a good start. As to having the controller in the same chassis, why don't you talk your employer into allowing uploading OpenFlow controllers to run on the control CPU? That is actually a good idea and might help sales of your product...

To implement NAT with OpenFlow you would need a rule that recognizes new connections and lets the controller add a new rule for that connection. The controller will not actually route or modify any packets, not even the initial one.

Re:How can you have a software defined network?

[bbn]

OpenFlow will only pass as much of the packet as you need to. For most cases that is just the headers. Say the controller is on a 10G interface and 100 bytes needs to be transferred out and then the reply will be about 100 bytes too. The time to process the packet will be the same or less compared to the switch build in controller (external controllers will generally be more powerful servers than the controller CPU in a switch or router). Time to transfer 200 bytes on a 10G is 200 ns.

Of course there might be multiple hops to reach the controller but that would be the network designers choice. Google apparently put the controllers adjacent to the switches, so they would have a direct connection.

Extra delay of this order, and only for the first packet in a new flow, is negligible. If it is a standard 1500 byte packet, it will be 200 ns to query the controller and then 1.5 microsecond to actually forward it.

By the way, there are multiple commercial available switches with OpenFlow support already. HP is retrofitting their entire product line with OpenFlow support. Juniper has experimental support too. Both companies seem to be doing it without rebuilding any ASIC or other hardware, considering adding OpenFlow is just a firmware update.

Nothing stops you from adding your own proprietary solution. But we need standards if we are to write software that will work on multiple brands and models.

Re:How can you have a software defined network?

[bbn]

It does not matter if it sends one bit per packet -- latency is per packet, not per byte. Packets must be sitting in a queue while the switch is waiting for response -- so the time for response is determined by the time for the queue to overflow, or the packet will have to be dropped. It will never work.

So you are saying my estimate of 200 ns delay is wrong? Give me your own calculations.

Yes the incoming packet is in a queue while the switch waits for response from the controller. That response can be there within 200 ns. In the meantime the switch is not blocked from processing further packets.

A 200 ns delay on the first packet in a flow of packets is so little that is barely measurable. You will be dealing with delays much larger than that simply because you want to send out a packet on a port that is already busy transmitting.

I am not going to comment on the rant about management protocols. OpenFlow is not a management protocol.

Re:How can you have a software defined network?

[bbn]

That is bullshit. Here is a guy that benchmarked the Intel X520 10G NIC that wrote a small piece titled "Packet I/O Performance on a low-end desktop": http://shader.kaist.edu/packetshader/io_engine/benchmark/i3.html

His echo service manages to do between 10 and 18 Gbit/s of traffic even at packet size of 60 bytes. And there is plenty of optimizations he could do to improve on that. The NIC supports CPU core affinity so he could have the load spread on multiple cores. The memory bandwidth issue could have been solved with NUMA. But even without really trying we are hitting the required response time on desktop hardware.

The simple fact is that after the packet has been transferred over the 10G link it will go through a PCI Express (x8) bus and be processed by the Linux OS - the same OS that you earlier claimed to be running on the control plane of the switches designed by your company. The only difference here is that I would probably get a faster system CPU than would be in your hardware.

As to the blocking issue, only packets from the same (new) flow would be queued. Say this was a NAT implementation, all other existing connections would continue with no blocking. Or if it was a BGP implementation, all other already cached destinations would continue to be routed. Also given that it is possible for the controller to reply in less time that it takes to actually receive a full sized 1500 bytes packet, this blocking idea is a bit far fetched.

Also given that protocols like TCP do not just suddenly burst out 10G of packets, the next packet following the initial SYN packet is not likely to arrive before the SYN has been processed by both switch and controller and forwarded long ago. And again packets to other destinations will not be blocked while we wait for the controller and somehow I get the impression that you think they would.

Re:How can you have a software defined network?

[bbn]

That's without data ever being accessed from userspace, no protocol stack, average packet size being half of the maximum, and there is a good possibility that the measurements are wrong, because then it would be easier to implement the whole switch by just stuffing multiple CPU cores into the device, and the whole problem would not exist.

The article was written by the guy that did the driver, I think we can assume he knows his stuff.

No it appears that if you want to switch more than 10-18 Gbit/s the computer would have a memory bandwidth problem. Trying to use multiple cores and NUMA might improve on that, but I do not think you would manage to build a 24 port switch that switches at line speed this way :-).

But if you could somehow get an external switch to do 99% of the work, this might work...

I am not sure how much more we can get out of this discussion. From my side I believe you are going too far in trying to make a problem out of something that actually works quite well for some very large companies (Google and HP!). Packets need to be delayed when the controller needs to be queried and that is true for both OpenFlow and traditional switches. We are just fighting over some nano or possible microseconds here with no one showing that it actually matters. It very likely does not matter for the use case that Google uses for, or they wouldn't be doing it. At my company we are using it too and it works very well for us. We are an ISP by the way.

There might indeed exist a work case where a 10G flow just pops into existing out of nowhere and where even 1 microsecond delay on the forwarding of that stream is not acceptable. I am just having a real hard time imaging that case.

Huh what is IOS and DDWRT then

[mjwalshe]

you know swtitches and routers already run software - having a single controller goes against the design goals of the internet.

Re:Huh what is IOS and DDWRT then

[jon3k]

This is for a network under a single administrative control not the entire Internet. Eg: An ISP, a datacenter, an enterprise campus, etc.

Re:Hype

[squiggleslash]

Whether it is or it isn't, it sure feels like it. It feels, from the description, like stuff we've been doing for decades, except now it has a fancy name and people are doing more of it.

I'm genuinely interested to know whether my impression of SDN is totally off base and whether it is radically new and different.

Re:Hype

[citizenr]

You had to buy specialized routers/expansion cards for decades to do certain things. Now you reconfigure those things on the fly.

Re:Hype

[AK Marc]

I finally had someone tell me something that can't be done with regular networking that can be done with SDN. Programmable STP. Currently, there's STP, and some proprietary replacements, like FSPF. But you can program your own primary and secondary links network wide, without having to rely on some more generic protocol to do it for you.

All the other things I've seen mentioned were SDN within a NIC for CPU offload. But if you are putting a computer in a NIC, you can do other things with it anyway.

Re:Hype

[jon3k]

SDN is separating the control plane functions from your network devices and centralizing. Yes, there is a lot of hype around it.