Slashdot Mirror
Google Deploys IPv6 For Internal Network
itwbennett writes "Google is four years into a project to roll out IPv6 to its entire internal employee network. At the Usenix Large Installation System Administration (LISA) conference in Boston last week, Google network engineer Irena Nikolova shared some lessons others can learn from Google's experience. For example: It requires a lot of work with vendors to get them to fix buggy and still-unfinished code. 'We should not expect something to work just because it is declared supported,' the paper accompanying the presentation concluded."
"'We should not expect something to work just because it is declared supported,' the paper accompanying the presentation concluded."
I think that if something is declared "supported", it is perfectly reasonable to expect it to work. If it turns out it doesn't work, I think the problem is more that the vendor hasn't done as good a job as they should have than that your expectations were too high.
Please correct me if I got my facts wrong.
assignment of smaller blocks may have extended the life of IPv4 addresses however, there are physically not enough addresses for the devices we currently have. While, there may be enough at the moment, there wont be soon.
What is IPv4; 4.3 billion addresses. There are over 6 billion people on earth and many people in the western world have numerous devices. My household of 2 has 8 devices that are nearly always online. (Computers, Phones, Top-set Boxes, printers, etc....) This number does not take into account either one of our work sites which probably add another 1-2 addresses to that number.
And no, NAT is not a solution.
Something no one would need if proper assignment of IP ranges had been done.
No point asking what you mean, since you evidently speak from ignorance. Even with optimal assignment of IPv4 addresses, it would only delay the inevitable shortfall. Sooner or later, the number of addressable end-points on the internet would exceed 4 billion. NAT is an unfortunate workaround to delay the effects of the shortfall; it should be a freely-chosen option, not an enforced requirement.
Those who can make you believe absurdities can make you commit atrocities. - Voltaire
Right, if decades ago the inventors of the internet had realized that it would scale from 10s of users to billions. I'd say the address space length that they used still makes it outrageously overengineered for the time, and we're lucky they had the vision that they did. To criticize them is preposterous.
"Each campus or office got a /48 address block, which meant that it was allotted 280 addresses. In turn, each building got a /56 block of those addresses (or about 272 addresses) and each VLAN (Virtual Local Area Network) received a /64 block, or about 264 addresses."
a /48 block is 65536 subnets for each campus. A /64 has 18,446,744,073,709,551,616 IP addresses.
The RFCs on this type of thing are RFC 6177 which replaced 3177 and RFC 5375. For a itworld/usenix article, fact checking is really low.
Simple solution, bump it up a notch.
My octets go to 257. Solved.
Every vendor is short on delivery.
The only reason they have some support is because of the U.S. Federal Government mandate that all vendors support basic IPv6 by (i forget the year its somewhere between 2008 and 2012)
Now, that doesnt mean its a comprehensive solution (those cost even more development dollars). They simply did the least amount of work needed to still sell the product to the government.
It wont be until the rest of us demand proper support any vendor will put the time and money into a proper solution
For example: It requires a lot of work with vendors to get them to fix buggy and still-unfinished code. 'We should not expect something to work just because it is declared supported,'
In other words, business as usual in all other areas of IT. Glad to see there is nothing "special" about ipv6 deployment.
And while the current versions of most OSes support IPv6, they do not do so by default.
What are those OSes? Its been a long time since I turned on ipv6 at home. As I recall I had to do little other than turn it on. There is a difference between "activate" which is kind of like setting the sound mixer output to a comfortable level no big deal, vs searching on the internet to install 3rd party drivers and/or recompiling kernels.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
2^32 - 2^24 - 2^16 - 2^20 - 2^16 - 2^28 = 4008574976. That's if you put them all on one giant flat network from hell, and so didn't use any for network or broadcast addresses. Yes, 2^16 in there twice - don't forget APIPA. The 2^28 is reserved for multicast.
I'd say the address space length that they used still makes it outrageously overengineered for the time, and we're lucky they had the vision that they did.
Not really. Don't forget there is a HUGE difference between the old classfull and VLSM/CIDR/classless numbering. That gain is the whole point of spending all that effort implementing netmasks. There really were not that many possible classfull lans compared to the number of minicomputer owning businesses in the world, etc.
For the post-92ish noobs, a really simple one line explanation is the netmask used to be stored inside the address itself, so for example if the first octet was 0 to 127, that meant that LAN had to be a (presumably giant bridged) /8, first octet 128-191 meant the netmask had to be a /16, not defaulted or was a pretty good guess, but operationally "had to be".
The early years of VLSM were pretty entertaining, old timers lecturing us how a LAN addressing scheme like 1.2.3.0/24 was "impossible" and so forth.
Without VLSM we would have to have done the ipv6 conversion years before the dotcom boom, rather than a decade or so after. Not entirely sure if we'd all be better off now, or not.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
Even I am kind of curious to see what would happen if we set a week in the future to switch everyone over. I say a week, not a day, because vendors will need at least 72 hours to issue emergency firmware upgrades after sections of the internet disappear, and allowing for different time zones and what not, of course.
Does anyone know if all the major service providers have upgraded their equipment to ipv6 yet? Any laggards?
I am John Hurt.
NAT has improved protocol design a lot though. Before NAT, there were things like FTP, with inband port signaling. Most modern protocols avoid mentioning port numbers in the payload and can run on any port, through multiple port forwardings if necessary. Notable exception and bad example: SIP. I expect more bad protocol design once people again assume that end-to-end IP addressing is universally available.
Why is that bad in the absense of NAT?
And for the post-1980s noobs, the original idea was that the first octet would be the network part and the last three would be the host part. Since 250 or so networks was 10 times what was expected. Classful addressing is a jonny-come-lately.
And yes, the fact that IP was expandable from 250 subnets to the present day shows that the initial engineering was phenomenal, but we're well past time for the next version of IP. If people spent a quarter of the time they spend complaining about IPv6 just implementing it, we'd be in a much better Internet.
-Kevin
Just think how long it would take companies without access to virtually unlimited funds and brain power. It's no wonder everyone is reluctant to make the move.
Oh man, what I would have given to be there for that conversation.
"How many addresses do you figure we need?"
"Couple billion I guess."
"But what if we need more?"
"Dude, okay, let's just say one per person. 4 and a half billion or so. Now everyone on the world can have one."
"But what if, you know, there ends up being a few more people than that in the future?"
"Jesus Christ man, it's not like 3 billion extra people are gonna pop up out of nowhere in the next 30 years!"
Random Thoughts From A Diseased Mind (Not For Dummies)
And that's the rub, the hosting companies probably won't provide it until they absolutely have to as the ISPs are generally not providing access. And the ISPs won't be providing it until after the customers demand it. The customers mostly think that the internet is Youtube and probably Facebook and probably won't ever request it unless those sites go unavailable.
IPv6 is cool, I get it. But how many ISPs are offering it to their consumers? If I want to build a web presence, would I settle for only IPv6 address space? If not, how much would I pay to buy into the IPv4 space so I can reach all my potential customers?
Loading...
OK, but that's not very clear. I can see why a program that picked a completely random port might be awkward to get to work with a firewall. But restricting the range of ports that it can use, then permitting those, would work wouldn't it?
I'm not sure it's a good idea to restrict protocol flexibility in that way anyway. There's a fundamental issue with NAT or firewalls in that they need to know details of what the users behind them want and don't want to do. This may be true for a business with a central IT department who can configure the device as necessary, but it's not true in general. If my ISP runs a NAT to conserve IP space, am I supposed to contact them to forward whatever ports are necessary? I don't think that'll work well. I just hope IPv6 actually does get rolled out before that becomes necessary.
Because the hardware that can handle large amounts of small packets fast when you install your own software ('firmware'), does not exist AFAIK. Atleast not the type which will also be supported by (multiple) vendors (no1 wants to be stuck on, locked into, one vendor). designing not-massproduced ASICS isn't cheap. It would be like Google designing their own CPU's for their servers.
The closest things are:
- NetFPGA (some people at Google worked on that project I believe) / LibreRouter - which use FPGA's to handle packets, you tell it how to do that.
- projects like Netmap, handle packets in userspace so you don't have to push packets through the kernel on normal PC-hardware, making it faster: http://www.youtube.com/watch?v=SPtoXNW9yEQ
The best chance currently to be useful in 'doing your own thing' is probalby:
- OpenFlow, which basically is an API standard which multiple vendors would support to describe what the hardware in a switch should be doing, a programming language almost. Some demo's:
http://www.youtube.com/user/stanfordopenflow
Which can allow for lots of tricks, like 'software defined networking'
New things are always on the horizon
Remember the mini-computer didn't even exists then.
So a computer was a large machine which took up a room.
And it was just an experiment, the experiment never ended.
If you want to know more about what the original creators thought, you should look up talks by Vint Cerf:
http://www.youtube.com/results?search_query=vint+cerf+ipv4+ipv6+depletion
For example this video:
http://www.youtube.com/watch?v=LcXCieD5YKE
New things are always on the horizon
I think you probably have that number backwards.. the vast majority of addresses are held/assigned to various ISPs and being used for peer devices, home internet, mobile devices etc. Most small-medium businesses are using 1-8 addresses. Most of the unused IPs are in the mid-large businesses that aren't using all they've been assigned, though segmenting an address block may, or may not be possible.
I would suggest that anyone with even a class B should probably be encouraged to break them up and return unused blocks. That will only help for so long. With 4 billion addresses (maybe 3.5 billion usable) and 6 billion people and counting, more and more with multiple devices, it wil only go so far. I really think that mobile companies should be among the first on IPv6 with IPv4 access via NAT & proxy. Just my $.02
Michael J. Ryan - tracker1.info
Nice random hit on H1B's there. Blame ignorance and lack of initiative on the foreigners -- that always works out!
you see, the good thing is not the NAT, but the firewall dropping packets from outside, again. As always, the people say the security comes from NAT, and really mean the requirement of having a firewall which drops packets coming in, because there is no mapping to which internal ip they should be routed.
I'm lucky enough to use an isp that offers native ipv6.
This coupled with a nifty firefox plugin (IPvFox) enables me to determine with some certainty that somewhere between 95-99% (tongue in cheek) of all ipv6 traffic on the internet is googles.
They are pretty much the only company using it.
(O.K. rss.slashdot.org... kudos to you guys).
Decades ago, the engineers did in fact consider 128 bit addresses, but in the end they went with 32 specifically because v4 was not considered a "production" version. There's a link on the wikipedia page for ipv6 to a video with vint cerf explaining exactly that.
"With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea...."
RFC 1925
What happens when both end-points are behind a hide-NAT? ... ...
Many-to-one NAT by nature breaks the bi-directional model of TCP and UDP communications. You can workaround it by using dynamic port mappings ala uPNP, but it's a ugly hack really.
:. Ultimate Control Dedicated/VM Servers
And no, NAT is not a solution.
Well, since IPv6 just will not happen, it's the best (which is not hard, because it's the only one) solution we have.
I suspect that most of the pain was suffered by the vendors in this case. Google will have written the IPv6 requirements into the multimillion dollar purchase orders and is quite capable of phoning a VP of sales and telling him that if this is not fixed NOW you might find yourself no longer qualified as a Google supplier.
BTW I read that the DoD has come up with a unique way to encourage vendors to make sure that their IPv6 implementations actually work. They've been told that whether or not their own Web sites are accessible via IPv6 will be a factor in acquisition decisions. I can't reach Cisco on IPv6, though.
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
Of course sometimes its still necessary, avoiding that just isn't as flexible.
SIP/H323 are a good example as the media has to be sent in a separate RTP connection. If it's not immediately obvious why that's the case RTP has to be sent as UDP to avoid latency/loss making a call unusable which TCP would. SIP can use TCP and H323 always does, so you can't send the media in the same connection.
Plus a lot of telecom environments don't have the same server handling the media as the signalling. One such use case is sometimes you get the phones to bypass the server and talk directly. That means less latency and less bandwidth used at the server, but it is only possible where end-to-end connectivity between the phones is is possible and NAT almost always breaks that.
I knew there was a language issue. Had they only realise that in manager speak "it still have some issues" means "ship it" ...
Google may have the largest networks, but I doubt they have the most complex networks. Otherwise they wouldn't be able to "scale out" as easily and quickly. I suspect most Google data centers are very similar in network topology and technologies used.
Old large organizations are the ones with weird complex networks which are not self-similar and use different network technologies. x.25 over tcp/ip, frame relay, netbios over tcp/ip, SDLC, token ring, FDDI, stuff that's still using Novell 802.3 ethernet frames ( http://support.novell.com/techcenter/articles/ana19930905.html ). If you're unlucky you'd need network equipment that can handle both the old stuff and ipv6 properly. The networks may not be connected to each other, but what if the old expensive equipment handling the "legacy network stuff" are also handling some IPv4 stuff?
Unless forced to I wouldn't bother upgrading an old bank to IPv6. Users inside can't connect directly to the outside world, unless they go through a proxy? That's a feature not a bug ;).
NAT breaks the internet, and it isn't a solution to running out of IP addresses.
The real issue is that in their eagerness to make sure we never run out again, they made it too complicated. It would've been far more sane to add a fifth set of numbers. That way all existing IP's would've been 000.XXX.XXX.XXX --> essentially not requiring ANY renumbering at all. And they still would've been in a format that people could relatively easily memorize or manually enter.
NAT killed one of the basic principles of the internet and you're trying to make it look like a good thing.
You should be aware that due to my attention deficit due to extreme computer usage i cannot read all your post.
Thank the internet-based attacks. I've had the pleasure of plugging in a fresh Windows XP (before SP3/firewall) computer to get security updates and have it infected 30 some odd seconds later.
I thought there was an announcement that the IPv4 address space is now totally exhausted. Or at least there are no new blocks to be assigned. The tunnel broker, Hurricane Electric indicates that IPv4 is exahusted.
The announcement - http://www.nro.net/news/ipv4-free-pool-depleted - was made when IANA, the central authority, ran out of addresses to give to the five regional internet registries. These regional registries will run out at different speeds. Geoff Huston's graph is very useful to see how fast this will happen - http://www.potaroo.net/tools/ipv4/plotend.png
Right now I'm running a free IP v6-over-v4 tunnel from my router to Hurricane Electric. I got assigned my own v6 LAN range. Mac OS X works fine, hits the v6 version of a website if it exists, the v4 version otherwise. Doesn't always work, I know. The DNS part is the problem to figure out. The larger infrastructure DNS servers (comcast, at&t, verizon, etc) need to support IPv6. Comcast has just begun rolling it out to end users, so hopefully they've got dnsv6 servers that work now and still return the correct regionally sorted IP addresses for cloud services like akamai.
As to your fridge example, before you share your fridge's address with every other fridge in the neighbourhood, I'd recommend you study man-in-the-middle attacks a little more carefully.
No man stands between me and my fridge!
When our name is on the back of your car, we're behind you all the way!
While the rest of the world can use their instant messenger software to share files or make calls, you are stuck on IPv4 and must use slow 3rd party servers to proxy data between you and your other IPv4 friends because your NAT won't allow them to connect directly. I'm sorry, but the problem is not about running out of IPv4 addresses internally, it is about connectivity with the rest of the world.
http://download.wsusoffline.net/
WSUS Offline Update. For those who can't/won't run a Microsoft WSUS Server but have enough machines to need one. Can be run on Linux.
'We should not expect something to work just because it is declared supported, . . ."
Why should IPv6 be different than any other feature a vendor documents?
I thought there was an announcement that the IPv4 address space is now totally exhausted.
IP allocation is heirachical. The IANA assigns IPs to the RIRs, the RIRs assign IPs to ISPs and big companies, ISPs assign IPs to their customers and so-on.
Currently the IANA have run out and APNIC have run-out. The other RIRs still have IPS to hand out for now (not for much longer though).
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
try having two IP's on the 'outside' of nat forward the same port to the same server (ie, port 80 on both IP's to your web server).. I have yet to find a single vendor that can do that, since it would not be able to figure out source traffic..
My ISP is a rural wireless ISP that does NAT at their POP. (I don't have much choice in Providers, its them, dial up, or satellite) Their whole wireless infrastructure is a 192.168.168.x network. All client sites sit behind another NAT device (the CPE router) that then translates that to a 10.10.x address.. I can't use any service that needs to address a certain port.. (people in my area get mad they can't host games on their WII's.. things like "whatsMyIP.com" are useless, so is dynamic DNS, since the public IP is a box serving thousands of customers.. This is the future of NAT, as IP's get scarce.
What are we going to do tonight Brain?
You've got to be kidding. Were you just looking for some way to criticize his post?
That is a bloated figure - the actual number is something like 3.7 billion. I had calculated it in a previous thread on IPv6 on /. - take 2^32, subtract all the private addresses of class A, B, C, subtract all the class D & E addresses as well, and also lose all the network and broadcast addresses. The number comes down to 3.7 billion. Even that is somewhat approximate, since it doesn't count all the classless configurations that are there, which would haemorrage even more network and broadcast addresses. I detailed the calculations there - don't feel like re-doing it here.
Bottom line - the total #IP addresses would amount to 1 for every 3 devices today, and going to be even less going forward. Even a 64-bit wouldn't have helped, since in some cases, more than 1 level of NAT is required, so a minimum needed would have been 96 bit. No, structurally, IPv6 is the best laid plan out there. Yeah, I can think of ways in which it could have been better, but I'm impressed w/ what is there.
Oh, and I'm glad that Google is planning to transform its network into IPv6. Once the big hitters - Google, Facebook, and others take the lead, the reasons to migrate to IPv6 will get more compelling.
Little known fact only 15% or so if the IP V4 addresses are actually being used by honest to God websites
It's funny how the network is designed so that multiple clients can access a single server.
Talk about misusing numbers in furtherance of an argument! I'd expect the number of servers to be relatively low in any network -- servers are typically designed to be shared resources, and (in general network topography terms) only really make sense when there are multiple clients to access it.
Little known fact: there are currently enough people on this planet to overwhelm the IPv4 address space if we just gave every person one address. And this doesn't include even having any web servers with independent addresses. Nor any SMTP servers, POP3/IMAP/Exchange servers, FTP servers, NTP servers, DNS servers, nor any other sorts of servers you care to imagine. Nor any routers (they each need an address) or other infrastructure devices.
So even if your number is correct, so what? Would we want to live in a world where 100% of IPv4 addresses are used by websites, with none left for actual clients? Websites are hardly the most voluminous nor the most important part of the Internet. Anyone with half a brain would expect that clients and other systems would make up the most voluminous parts of the network; claiming that only 15% of addresses are used for the web and then trying to intimate that the other 85% are just "wasted" is completely non sequitur.
Yaz
Have you actually been in a company that has deployed IPv6 internally and externally? I have.
Have you run a dual stack network? I have.
Have you dealt with the issues involved in moving from IPv4 only to IPv4 + IPv6? I have.
Have you dealt with the issues of run numbering networks? I have.
I will tell you this. I would much rather deal with the minor issues of bring up IPv6, than to repeatedly have to deal with the issues of renumbering. At least with IPv6, once you fix the problem it stays fixed.
The problems people are seeing with IPv6 are mainly lack of planning issues. Failures to build in IPv6 initially despite it being the only viable solution to address exhaustion. Failures to make IPv6 support a requirement. We are playing catchup at the moment, trying to cram what should have been 10 years of incremental development into 1 or 2 years.
For most applications that deal with IP addresses or sockets they cost to support IPv4 and IPv6 is actually minimal or zero when the application is being developed.
Most machines actually support IPv6. There are a few, memory limited, machines that can't but overall they are in the minority and are also relatively inexpensive machines to replace.
I would actually recommend that every company bring up IPv6 at the network level today and connect to the global IPv6 with a firewall that only allows reply traffic in initially. Don't add AAAA records for your servers initially. Do add them for your workstations. Add corresponding PTR records. You will find that IPv6 isn't as scary as you think it is. It also gives you a environment where you can test your servers, by adding AAAA records to the host file of the machines involved in the test. When the service is working you then add the AAAA records to the DNS and remove them from the host files. Don't forget to open up the firewall to allow external connections to the service if appropriate.