Google Deploys IPv6 For Internal Network
itwbennett writes "Google is four years into a project to roll out IPv6 to its entire internal employee network. At the Usenix Large Installation System Administration (LISA) conference in Boston last week, Google network engineer Irena Nikolova shared some lessons others can learn from Google's experience. For example: It requires a lot of work with vendors to get them to fix buggy and still-unfinished code. 'We should not expect something to work just because it is declared supported,' the paper accompanying the presentation concluded."
Something no one would need if proper assignment of IP ranges had been done.
What vendors, Cisco? They seem big on advertising and limited on support.
"'We should not expect something to work just because it is declared supported,' the paper accompanying the presentation concluded."
I think that if something is declared "supported", it is perfectly reasonable to expect it to work. If it turns out it doesn't work, I think the problem is more that the vendor hasn't done as good a job as they should have than that your expectations were too high.
Please correct me if I got my facts wrong.
"Each campus or office got a /48 address block, which meant that it was allotted 280 addresses. In turn, each building got a /56 block of those addresses (or about 272 addresses) and each VLAN (Virtual Local Area Network) received a /64 block, or about 264 addresses."
a /48 block is 65536 subnets for each campus. A /64 has 18,446,744,073,709,551,616 IP addresses.
The RFCs on this type of thing are RFC 6177 which replaced 3177 and RFC 5375. For a itworld/usenix article, fact checking is really low.
Simple solution, bump it up a notch.
My octets go to 257. Solved.
For example: It requires a lot of work with vendors to get them to fix buggy and still-unfinished code. 'We should not expect something to work just because it is declared supported,'
In other words, business as usual in all other areas of IT. Glad to see there is nothing "special" about ipv6 deployment.
And while the current versions of most OSes support IPv6, they do not do so by default.
What are those OSes? Its been a long time since I turned on ipv6 at home. As I recall I had to do little other than turn it on. There is a difference between "activate" which is kind of like setting the sound mixer output to a comfortable level no big deal, vs searching on the internet to install 3rd party drivers and/or recompiling kernels.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Even I am kind of curious to see what would happen if we set a week in the future to switch everyone over. I say a week, not a day, because vendors will need at least 72 hours to issue emergency firmware upgrades after sections of the internet disappear, and allowing for different time zones and what not, of course.
Does anyone know if all the major service providers have upgraded their equipment to ipv6 yet? Any laggards?
I am John Hurt.
> It requires a lot of work with vendors to get them to fix buggy and still-unfinished code.
Google should be used to that. They could always lazily stick 'beta' next to the product name, I guess.
There is a lot of stuff that does not have IPV6. Do they have some kind of NAT for the older IPv4 stuff?
Just think how long it would take companies without access to virtually unlimited funds and brain power. It's no wonder everyone is reluctant to make the move.
Given the Google has absolutely no shortage of capital and brain power as noted before, I am surprised Google didn't just build its own routers, wireless access points, etc. Linux and BSD have come along way in their routing capabilities. Heck, Vyatta sells an open source router that probably competes very favorably. If I were Google, I would have opted for the open source methodology and contributed back to the community. You pay a vendor and expect quality, you don't beg them to improve their product. They should be jumping through hoops to help you.
IPv6 is cool, I get it. But how many ISPs are offering it to their consumers? If I want to build a web presence, would I settle for only IPv6 address space? If not, how much would I pay to buy into the IPv4 space so I can reach all my potential customers?
Loading...
I'm lucky enough to use an isp that offers native ipv6.
This coupled with a nifty firefox plugin (IPvFox) enables me to determine with some certainty that somewhere between 95-99% (tongue in cheek) of all ipv6 traffic on the internet is googles.
They are pretty much the only company using it.
(O.K. rss.slashdot.org... kudos to you guys).
needs to be IPV6 so it can be like NAT is just need to make the out side stuff work with IPV6 and the in side can still have the older IPV4 only stuff.
Early large-scale adopters like Google have suffered the leading edge of vendors trying to get ready. In terms of the problems Google ran into, I'd wager a large chunk of them won't be inflicted again by the same company. Once kinks are worked out for even one customer, they are generally worked out for all customers.
That said, while I've seen a large amount of increased IPv6 capability from vendors (showing they have expertise *somewhere*), it's still an arcane art for almost everyone at these companies still yet relative to IPv4.
XML is like violence. If it doesn't solve the problem, use more.
While I anticipate Google to have one of the most complex networks, they also probably have a more reasonable organizational structure populated by more talented individuals on the whole. I say this not because I think Google is magic, but I optimistically *hope* they aren't as bad as some of the companies I have dealt with. Most companies have a technical staff either not talented enough, bound up in an impossibly convoluted organizational structure that paralyses them in any efforts to technically advance the state of things, or some combination of the two.
XML is like violence. If it doesn't solve the problem, use more.
yes
Michael J. Ryan - tracker1.info
Most computers will use Dual-Stack for the foreseeable future for precisely this reason.
Just imagine the amount of funds and brain power that is available to me...
Each home is supposed to get a /48 from the IPv6 ISP. Then the residential subscriber can provision up to 65 thousand subnets. The remaining 64 bits are left for the autoconfigured MAC address.
Because of privacy concerns, the MAC address can be obfuscated. That way, nobody will be able to tell for sure which physical device in your home posted the controversial contribution.
The talk about 2**128 IPv6 addresses are rubbish. The address allocation schemes have carefully been designed to support about as many addresses as there are MAC addresses, that is, in the range of 48 bits. It's much better than IPv4 but only by 5 orders of magnitude.
The Google guys need one of these: http://www.ipv6buddy.com/ :-)
Why does the soon-to-be-defunct USPS need a /8?
http://xkcd.com/865/
Right now I'm running a free IP v6-over-v4 tunnel from my router to Hurricane Electric. I got assigned my own v6 LAN range. Mac OS X works fine, hits the v6 version of a website if it exists, the v4 version otherwise. Doesn't always work, I know. The DNS part is the problem to figure out. The larger infrastructure DNS servers (comcast, at&t, verizon, etc) need to support IPv6. Comcast has just begun rolling it out to end users, so hopefully they've got dnsv6 servers that work now and still return the correct regionally sorted IP addresses for cloud services like akamai.
IPv6 will be very important next year... ... so we are told for 15 years now.
It will just never happen.
Running out of IPv4 addresses internally... give me a break - who believes that?
NAT is the answer delivered for a long time now. And it will remain there forever. Amen.
While the rest of the world can use their instant messenger software to share files or make calls, you are stuck on IPv4 and must use slow 3rd party servers to proxy data between you and your other IPv4 friends because your NAT won't allow them to connect directly. I'm sorry, but the problem is not about running out of IPv4 addresses internally, it is about connectivity with the rest of the world.
'We should not expect something to work just because it is declared supported, . . ."
Why should IPv6 be different than any other feature a vendor documents?
> Each home is supposed to get a /48 from the IPv6 ISP. Then the residential subscriber can provision
> up to 65 thousand subnets. The remaining 64 bits are left for the autoconfigured MAC address.
> Because of privacy concerns, the MAC address can be obfuscated. That way, nobody will be able
> to tell for sure which physical device in your home posted the controversial contribution.
Well... like... whoopee. Marketeers (e.g. Fecesbook) will love it. It'll still let them know that certain web requests are coming from the same home. They'll be able to aggregate all your web browsing, etc, regardless of how much you spoof your mac address, because it's just a matter of seeing which /48 it's in. I prefer dynamic IP addresses thank you. Those l33t h@x0r d00ds out there who want to run your own public webservers are more than welcome to ask your ISP for static addresses.
And WTF are they thinking, handing out that many addresses per account? That's 9.022 * 10^14 addresses for every man+woman+child in China or 1.727 * 10^14 addresses for every man+woman+child on this planet. A /96 should be enough to run an ISP anywhere except China or India.
25 years from now, we're going to run into an unexpected shortage, and we're going to have to scrap a whole bunch of routers, etc which are hardcoded to expect /48's, and replace them with routers, etc that expect slightly smaller blocks. The Ciscos of this planet will love it.
I'm not repeating myself
I'm an X window user; I'm an ex-Windows user
OSI vs TCP/IP - check out OSI Addressing, it would have coped much better but TCP/IP and 4 octets won out. IPv6 is a retrofit to a wider space...but its much more...no more broadcasts. Its cool but the Hex will get clunky as the addresses get eaten up.
Something no one would need if proper assignment of IP ranges had been done.
Or if IPv4 had been defined @ 128 bits the first time around, instead of 32. Only allowance I'd give is that in the 70s, nobody imagined that computers would be as inexpensive and widely available as they are today, or that a whole list of new devices would go digital and start needing IP addresses, not just some huge servers in some 50 big organizations in the US.
The difference with IPv6 is that the address space is big enough, that even if we were incredibly shortsighted and not able to see the future coming and squandered every single /48 except for one with inefficient assignment and we end up needing more than we predicted, we'll still have 2^80 addresses left to work with. Placing them under a new addressing scheme, assigning them more frugally, that's a hell of a lot more than 2^32 where we are today.
A small address space such as 2^32 has shown to be practical to manage, since this is how the Internet works today. It stands to reason that 2^80 would not be impractically small to be managed using similar techniques.
Over time, it's reasonable to assume that more old large assignments can be reclaimed (as we have seen in IPv4 - although in IPv4 returning a single net block won't have nearly the same impact as in IPv6), giving us virtually limitless potential for expansion.
To put it all in perspective, I for one wouldn't really mind if my home got a IPv6 /120 rather than a /48 - say. I doubt many people run a home network that requires more than 256 addresses. That's still a hell of a lot better than the single IPv4 address I get now.
While you may not believe it, there are companies with over 17 million devices that need to be connected and directly addressed. Once you get up to those numbers RFC 1918 no longer is big enough to address all the machines (RFC 1918 covers 17891328 addresses). Add to that the need for rational sub netting and you can start to run out of address with less that a million machines.
No, I don't believe that...
The NAT traversal problem has been solved by a lot of services/applications. And instead of asking the entire world to change their home routers and throw away their embedded devices, NAT solves most issues more or less. IPv6 introduces a lot more problems than it solves. And privacy concerns is one of them.
You may not believe it, but it part of my day job to support such companies.
17 million devices? Nah! Bad subnet planning; yes!
Or more likely, Dual-Stack lite. Dual Stack won't be an option if one is out of routable IPv4 addresses.
The reason for spending billions on their Aerospace industry is that if they were to buy stuff from the superpowers, they'd be at their mercy when it comes to the sale of spare parts, and I'm not talking about the price of those things, but whether the country would be authorized to even buy it in the first place, or if they paid for it, whether it would be delivered. That's why they do it.
As for IT workers, a number of Americans and other foreign IT workers can be seen in Indian tech hubs like Bangalore, Pune, Hyderabad, et al. If an American tech worker is willing to accept what are market salaries in India and move and work there, it's easier for them to get a work visa and job than it is for an Indian to get an H1B, which is capped for workers from India and China, but unlimited from countries like Mexico, Somalia, Saudi Arabia, Iran, Lebanon, et al.
Do you have any idea how expensive it is to continually re-size networks just so you can fit all your machines into a allocation that is in reality too small for your company? The fact that you say:
Bad subnet planning; yes!
means you just don't understand. One really shouldn't have to think about subnet sizes. That is one of the advantages of IPv6. You don't have to think about subnet sizes. You just subnet at the /64 and you have a subnet big enough for any conceivable use. No more juggling, No more getting it "wrong". No more renumbering because you need more space.
I agree that it's a hassle. But the hassle of using IPv6 is a _lot_ bigger. It's plainly stupid to try and fix a problem by creating a bigger problem.
Have you actually been in a company that has deployed IPv6 internally and externally? I have.
Have you run a dual stack network? I have.
Have you dealt with the issues involved in moving from IPv4 only to IPv4 + IPv6? I have.
Have you dealt with the issues of run numbering networks? I have.
I will tell you this. I would much rather deal with the minor issues of bring up IPv6, than to repeatedly have to deal with the issues of renumbering. At least with IPv6, once you fix the problem it stays fixed.
The problems people are seeing with IPv6 are mainly lack of planning issues. Failures to build in IPv6 initially despite it being the only viable solution to address exhaustion. Failures to make IPv6 support a requirement. We are playing catchup at the moment, trying to cram what should have been 10 years of incremental development into 1 or 2 years.
For most applications that deal with IP addresses or sockets they cost to support IPv4 and IPv6 is actually minimal or zero when the application is being developed.
Most machines actually support IPv6. There are a few, memory limited, machines that can't but overall they are in the minority and are also relatively inexpensive machines to replace.
I would actually recommend that every company bring up IPv6 at the network level today and connect to the global IPv6 with a firewall that only allows reply traffic in initially. Don't add AAAA records for your servers initially. Do add them for your workstations. Add corresponding PTR records. You will find that IPv6 isn't as scary as you think it is. It also gives you a environment where you can test your servers, by adding AAAA records to the host file of the machines involved in the test. When the service is working you then add the AAAA records to the DNS and remove them from the host files. Don't forget to open up the firewall to allow external connections to the service if appropriate.
An extra screen in the config box to set a static IPv6 address on an embedded device? Not seen one yet... Why? Because these embedded boxes are typically run in a seperate VLAN in the company.
Corporate requirements for IPv6 are close to nonexistent, so nobody cares, nor will. It's not that I'm against IPv6, but one has to be realistic about what to expect from the rest of the world - and a drastic change without a game-changing urgent need is not one of these things.
And I'm still waiting for an example of any organization _ANY_ organization who needs to have in the order of 16 million directly communicating devices on their private network. Just a million will do as well. Probably Google is the only organization which comes close to that order.
And even for them, there is not really an important reason why their infrastructure could not be split up between the google search cloud as one 10.x.x.x range and the gmail infrastructure as another one, for example, as direct communication between the two is probably unnecessary and managed by separate teams anyway.
One could argue that the 'renumbering' is difficult. Yet the cluster which Google build handles server failover, swap-in and swap-out and data partitioning as one of the major features. Fail to see why they couldn't implement it on the 'private IPv4'-level either...
While I agree that on their scale, such an experiment might be valid, my guess is that it will remain as such;... an experiment with a lot of problems:
1) increased latency because of IPv6 tunneling - and Google is very latency conscious 2) less proven technology leading to exotic problems which show up even more at the Google scale - because nobody uses it
And for what? ;)
To solve the 'we are too lazy to write a stupid IPv4 pool re-numbering/re-partitioning'-problem? While it can be done with a very small shell script(TM)?
Really? Please back up your statements with some arguments. The NAT traversal problem has not been solved by a long shot. Sometimes software can configure the the NAT remotely (UPnP) but that obviously doesn't work when your ISP starts NATting you.
To me it looks like you miss some basic information about how client-server networking actually works and why NAT is a serious hindrance to correct internet functionality.
Care to elaborate on why IPv6 introduces more problems than it solves?
It just will not happen. Many embedded devices build today don't even enable the IPv6 stack while it's only a configurable option away in the Linux kernel. And these devices aren't even released yet. They will after release run for a decade in the infrastructure. Sure we tell people who make them to care. Yet mostly they have more important things to care about, as for example to get them working in the first place.
An extra screen in the config box to set a static IPv6 address on an embedded device? Not seen one yet... Why? Because these embedded boxes are typically run in a seperate VLAN in the company.
For some reason you seem to think turning on IPv6 precludes running IPv4 devices or being able to reach them from IPv6. IPv6 only internal networks are still a long way away.
Corporate requirements for IPv6 are close to nonexistent, so nobody cares, nor will.
It's not that I'm against IPv6, but one has to be realistic about what to expect from the rest of the world - and a drastic change without a game-changing urgent need is not one of these things.
Lots of corporate desktops talk to machines on the outside. These machines will be a mixture of IPv4 only, IPv6 only and dual stack. The need to talk to these machines alone will result in IPv6 being deployed internally. We are starting to see ISP complaining that they can't get enough IPv4 addresses to meet their needs. It won't be long before the first forced IPv6 only sites start appearing. When that happens, offices will start to adapt by enabling IPv6 to allow the desktop machines to reach these sites.
And I'm still waiting for an example of any organization _ANY_ organization who needs to have in the order of 16 million directly communicating devices on their private network. Just a million will do as well.
Probably Google is the only organization which comes close to that order.
Pick any large ISP that manages CPE equipment. These needs to be addressed.
And even for them, there is not really an important reason why their infrastructure could not be split up between the google search cloud as one 10.x.x.x range and the gmail infrastructure as another one, for example, as direct communication between the two is probably unnecessary and managed by separate teams anyway.
Multiple routing realms are additional operational complexity. Given the choice of "deploy IPv6" or "run multiple routing realms" I'm sure many companies will pick run IPv6.
One could argue that the 'renumbering' is difficult. Yet the cluster which Google build handles server failover, swap-in and swap-out and data partitioning as one of the major features. Fail to see why they couldn't implement it on the 'private IPv4'-level either...
While I agree that on their scale, such an experiment might be valid, my guess is that it will remain as such;... an experiment with a lot of problems:
1) increased latency because of IPv6 tunneling - and Google is very latency conscious
2) less proven technology leading to exotic problems which show up even more at the Google scale - because nobody uses it
Tunnels will go away and be replaced by native connections. This is a observable trend today.
The amount of IPv6 traffic world wide is growing rapidly. The bugs in equipment will be worked out.
And for what?
To solve the 'we are too lazy to write a stupid IPv4 pool re-numbering/re-partitioning'-problem? While it can be done with a very small shell script(TM)? ;)
There are lots of reasons to deploy IPv6. We are at a tipping point where staying with IPv4 will start to get more and more expensive. IPv6 will be seen as the cheeper alternative.
Now where did that come from? I do not have any routing or DNS problems and you would have no way to know if I did anyway.
Where does this blind rabid attack dog going for the messenger bullshit come from just because I dared to mention a problem on some machines that happen to have MS Windows 7?
I hear the same mantra since 1995, yet my smartphone isnt even NATed by my ISP and large corps still sit on large IP blocks for nothing.
Yes, some ISPs are now looking at IPv6, just as they are looking at NAT - I agree that there is no clear winner there yet. Though an IPv6-only ISP would still be the first... And using IPv6 on the internal corp network is an entirely different story. And just not happening.
Yes, dual stack is an option, yet it means that some parts of your infrastructure are unable to communicate - basically replacing the broken part with another broken part.
But hey, dont take my word for it. Ive been an IPv6 believer too... 15 years ago. Now Im just an IPv6 cynic ;)
Actually, I'm pretty scared of this announcement.
It basically says Google started offering public IPv6 services without the experience by running IPv6 on their own corporate network.
You need to be running IPv6 on your corporate network first (maybe not "everywhere", but at least "in most places"), so your own developers don't stick to IPv4-only code and learn what actually happens with IPv6 and your networking dept makes IPv6 an important requirement with your upstream or peering ISPs.
Your internal IT helpdesk needs to learn about IPv6 and promptly address it in a user-friendly way ("I do have trouble printing." - "Do you print via IPv4 or IPv6?" is NOT a good example), and so you do need to do this in order to educate any other customer-facing employee who may ever need to talk about IPv4/IPv6 (if you're an ISP, this also includes your customer helpdesk, your sales staff, product developers, marketing and public relations).
Another one is the often-retold stories about IPv6 being broken and being unreliable, and Google is exactly one of those companies doing a lot of publicity and buzz about it.
Probably one of the easier ways to solve this myth is by simply offering router advertisements on a workstation network and make your users actually use IPv4 and IPv6 in parallel (my employer did this a few years ago, with full management backing). So when they do come up with something like "I've heard that 30% of IPv6 connections are completely broken and IPv6 is hard to configure", prepare for seeing stunned faces when you tell them that their workstations have been running IPv4 and IPv6 for five years now.
Tell me which business or government agency has filled up 10.x.x.x. IPv6 doesn't matter internally unless your a communications company. Yet it's the communications companies that are keeping it from their customers because it invites a more distributed internet.
It's trivial and easy to upgrade users, just get a new routers, upgrade the firmware on existing ones, or use simple IPv6 to IPv4 endpoint converters. So long as the internet tunnels are IPv6 there are no deployment problems. Servers want visitors which are predominantly web surfers, so they need need to be IPv6. IPv6 users can connect to IPv4 servers easily, the reverse is not true.
Speculation:
Geeks drive this technology. The reason for IPv6 is the contention for addresses. Yet the new IPv6 hands out /64's to end points like it's water... It's not like it was handing out /16's which might be reasonable. So anyone looking at this can clearly see we are being set up for failure and their will obviously need to be an IPv7 or IPv8 to fix the /64 mess. It looks like new scheme is trying to supplant/abandon port numbers.