Slashdot Mirror
The IPv4 Internet Hiccups
New submitter pla writes: Due to a new set of routes published yesterday, the internet has effectively undergone a schism. All routers with a TCAM allocation of 512k (or less), in particular Cisco Catalyst 6500 and 7600's, have started randomly forgetting portions of the internet. 'Cisco also warned its customers in May that this BGP problem was coming and that, in particular, a number of routers and networking products would be affected. There are workarounds, and, of course the equipment could have been replaced. But, in all too many cases this was not done. ... Unfortunately, we can expect more hiccups on the Internet as ISPs continue to deal with the BGP problem." Is it time to switch to all IPv6 yet?
Surely 512k ought to be enough for any router?
"Is it time to switch to all IPv6 yet?"
No.
We changed all our systems over time to handle this great IPv6 change, and haven't used IPv6 yet. Our service provider doesn't even offer it. Come on, some of us are more than ready. We will probably have failures, because it hasn't been truly tested, but we are far more ready than we were for Y2K.
Seems useless and not future proof.
I call for IPv128.
Well, if you pay for the cost, otherwise it will be much easier to just patch the problems and keep on going.
That way we will have access to more mature technology when we do make the switch. Also, it is unfeasible to switch it all at once.
Gradual switching when needed is preferable.
There's still plenty of time to postpone that. Not until the last /2 is sold will I start to worry. And can't we start using a few 127.x.x.x? Do we really need 16 million addresses for testing?
/sarcasm
Tic-Tac-Toe, Global Thermonuclear War, and relationships all have the same winning move.
IPv6 will only make this worse, with more routes to be kept in a local routing table and more stuff cached in the TCAM tables.
The solution is the one Cisco gave them: buy a router that can accomodate more entries in that particular table. I'm sure they explored this option vs paying SLA penalties to whomever requests it.
just to avoid problems like this.
We seem to have a bunch of things failing somewhat on the same day... is Cisco effectively saying "We're taking back what you have... please pay more!"?
This isn't really to do with BGP or IPv4 as such, it's an inherent problem in the way "The Internet" regards addresses.
You might be able to get some efficiencies in IPv6 by incorporating formerly-unrelated address allocations under a single prefix. But that doesn't solve the problem of a continuously growing network, increasingly complex (and commercially controversial) peering arrangements, the fact that IPv6 addresses are actually larger and the fact that you're going to have to support IPv4 anyway in parallel with any IPv6 transition (I don't personally believe it will ever happen, but that's a different story).
You could, however, get rather more efficiency in core routing tables if network addresses only had a very transient existence and were related to the source/destination route to be employed (eg: look up a domain name, do some route pre-computation, allocate some addressing tokens that make sense to the routers on the path, recalculate the route periodically or in response to packet loss). That's not IPv6, though. IPv6 has the same order of dependence on every router knowing about every destination network as IPv4 does (give or take the slightly greater prefixing efficiency).
TL;DR - The Internet is getting bigger. Buy more kit.
It uses 128 bit addresses instead of 32 bit addresses. And it does not officially support subnetting or hierarchical routing, although somewhere in the flood of IPv6 RFCs (there are dozens and dozens, most of which are at least partly deprecated) there might be some specification for somethat like that which is being ignored.
googling verizon, comcast, and time warner it seems like their original pledge in 2012 to start rolling out ipv6 has quietly halted. most of their sites simply say "check back" while others imply certain undisclosed service areas may be exposed to both 4 and 6. forums are another story, with most customers and techs confirming the support exists, but either modems arent enabled to receive ipv6 due to bugs, or the support is broken in all-in-one devices in the case of DSL.
speaking from a linux neckbeard standpoint, i dont care. ive had competent functional v6 support for almost a decade and in many cases implemented it for pay. In my experience the problems associated with implementing v6 are related to companies angry about any downtime at all, or vendor specific appliances that just cant for some reason or another. they either lied about their ipv6 support, only partially support routing IPv6, or have egregious bugs in their implementation that cause stability problems in the rest of the network. Hosting providers have done an excellent job of supporting it from what ive seen, and most (with the exception of godaddy) are very generous in their IP offerings (i get 30 with ramnode.)
Good people go to bed earlier.
The fact is, TCP v6 was defective by design, because of what it does not have, and that is a mechanism for a long transition period between ipv4 and ipv6. If we had such transition period, ipv6 would now be widespread. The transition period means that ipv4 and ipv6 networks can communicate with each other. Making Ipv6 talk send packets to an ipv4 network is easy: give the ipv4 address block a subset of the ipv6 address block. The more complex but entirely doable part is ipv4->ipv6. Since ipv6 is larger address space than ipv4, ipv4 cannot directly see a lot of ipv6 addresses. The answer lies in the DNS system. When a user on an ipv4 network askes for the IP address associated with a DNS address which only has an ipv6 address associated with it, somewhere upstream, an upstream router and DNS server will conspire to 1) give the user (ipv4 peer) a fake IPv4 address for a DNS address 2) give the information on the ipv6 to fake ipv4 mapping to the router 3) which the router uses NAT to rewrite the packets headed out from from the fake ipv4 destination address to the real ipv6 destination address. Ipv6 packets headed in would be rewritten to ipv4 replacing the ipv6 source address with the fake ipv4 source address. Each ipv4 peer should be able to re-use the same block of ipv4 fake addresses, the mappings can be done on a per ipv4 peer (user) basis. Using this, its also possible to give ipv4 clients direct access to ipv6, using an .ipv6 DNS TLD, which can be used in the form .ipv6. You could even write an HTTP and other application protocol proxy that would automatically rewrite all ipv6 addresses in HTML with ipv6 TLD addresses. This makes ipv6 a upstream ISP thing rather than something that affects things on the users end, greatly simplifying things.ISPs as a complementary measure could also offer 6over4 gateways as well, and then over time transition to allowing raw ipv6 over their networks, a transition which can be gradual.
Probably why I couldn't reach NeoGAF for most of yesterday, unless I went through tor. Which I did, because I'm a man and I have my needs.
Belief is the currency of delusion.
This is a real question: Do you know what IPv6 does instead of BGP? Because as far as I know, IPv6 is still using BGP, and that is what this is a problem with. In fact I can only see IPv6 making things worse in that regard because tons more address space means that more AS assignments would be easy to do.
So if it really does offer a solution, please enlighten me I'd be very interested. If this is just an example of trying to use a problem to push a favoured agenda, then please knock it off.
To some degree obviously, there is a lack of incentives for ISPs to change - if they still have enough addresses for themselves, then switching to IPv6 is only costs, not benefits.
Maybe some of the larger sites, like youtube, facebook, wikipedia should have a meeting to discuss the switch-over and then start shaping IPv4 traffic - just reduce capacity on IPv4 by 5% every month and see how long it will be, before ISPs will lose customers if they DON'T switch to IPv6...
No transition period? We are about fifteen years into that transition period, and it has sucked immensely with things like the requirement of man in the middle stuff like Skype just to get VoIP to work on an internet infested with NAT.
This particular problem is due to the way routing on the Internet works, where generally every router must hold routes for every prefix announced on the Internet. That system doesn't change with IPv6. Now, there might be fewer IPv6 prefixes at this time than IPv4, but intrinsically there's nothing about IPv6 that addresses the problem that all prefixes must have global visibility.
To fix this kind of problem requires changing how routing is done.
I use Friend/Foe + mod-point modifiers as a karma/reputation system.
With SDN, an infinite number of prefixes can be stored on the SDN controller, and the Internet router only needs to load prefixes into the router TCAM when there is actually a flow needed for that prefix.
We lost probably $30k in lost sales, and employees unable to do their jobs yesterday. Liquid web is going to lose a ton of customers over this. I don't know if it was their "fault," or if it was the top tier providers in their area they contract with. But as I understand it, if we had been with anyone really big who had us colocated in facilities way far away from each other, this would have been extremely unlikely.
Can anyone tell me how to set my sig on Slashdot?
It's more likely a completely new/different Internet will be designed before IPv6 is ever widely adopted. It's just too difficult to use compared to IPv4. It has too many features that add complexity when trying secure networks. Plus the addresses are long and annoying to configure, etc.
IPv6 is to IPv4 as DVD-DL is to DVD.
If they can't hear/speak IPv6, then the Internet is going to feel like a very big empty room. Everyone needs to change to the new protocol. Everywhere. And IPv4 still has to work. Everywhere.
And the problem goes away. The size of the routing tables is growing so much because every Tom, Dick, and Harry small business customer wants their own /29 block and certain ISPs like to serve those up using CIDR (are you lisetning to me, Comcast and Verizon?). This unnecessarily EXPLODES the size of the routing tables that everyone has to deal with.
Given the time between IPv6 design and the eventual global adoption of it and abandonment of IPv4, will the broader adoption of IPv6 reveal problems addressed in a future revision?
I'll admit to being willfully ignorant of IPv6 other than seeing it as enormously more complicated than IPv4, trying to solve too many problems at once. I sometimes wonder if maybe IPv6 didn't appear so complicated and different that adoption might have been increased.
Couldn't they just have added a couple of extra bytes to IPv4 to come up with something that worked like IPv4? I also wonder about an addressing scheme like IPX, where a single network address covers an entire broadcast domain and node addresses are MAC addresses plus the network address. IPX network addresses were only 8 bytes, maybe that wouldn't be future proof enough (4.2 billion networks). I'm not talking about IPX as a protocol, just the system for addressing.
The advantage is relative simplicity (no need for DHCP, network addresses are discovered and the rest is built-in), broadcast domains can scale arbitrarily large without needing to renumber -- sure you can start out every network with a /16, but often they don't and there are complications in organizations just arbitrarily shifting masks past /24, such as running into other networks in the local routing domain.
Since node addresses are locally determined, ISPs would need to only assign a network address which would allow for basically unlimited public network addresses to each subscriber.
I actually bought a new router within the last year. A "nice" Buffalo model with DD-WRT built in. Only to find out DD-WRT doesn't support native IPv6 (which my old, faulty NetGear did, go figure). They just support Toredo or other tunneled IPv6 solutions.
Man, was I disappointed.
Maxim: People cannot follow directions.
Increases in truth directly with the length of time spent explaining them
Not the fact that wifi routers degrade, you are totally right about that, but that people will replace them. I'm amazed at how shitty someone's Internet can be and they have an "Oh well, whatever," attitude about it.
A good example near and dear to me is my parents. They moved in to their current place about 7 years ago and got a cheapass Linksys router to handle their NAT and WiFi. It has been giving them enough grief for me to hear about it for at least 3 years. They are not poor, a new router is not a big deal, yet they didn't get one. So I got tired of it, and also had an easy solution: When they were visiting me this June I upgraded my WAP to a new 802.11ac one and gave them my old one, which was working great.
They still haven't installed it. It's not like they don't have time, mom is retired and dad is semi-retired, it's not like it is hard, it is much simpler to set up than their old model and they can always call me. They just haven't bothered. Their router acts up, they go reset it, and don't bother to replace it.
Another somewhat related example would be a friend of mine. He's a young guy, under 30, and quite technically savvy. He's complained to me that the Internet at his house is not meeting advertised speeds, going quite well below it. Strange, since we are both on the same ISP, and live only a couple miles from each other and my experience has been that they always are right around max. I inquire a bit more and find out he still has a DOCSIS 2 modem. Ahh ok, well that is probably the issue. Though his connection is of a speed that a single DOCSIS channel can handle (25mbps), that modem has one one channel to choose from and it could well be too loaded down by other people on the segment. So my recommendation was to get a DOCSIS 3 modem. An 8x4 modem that is compatible can be had for like $80. That should solve any speed issues since now there's a bunch of channels to choose from, and will be compatible when they bump the speeds in the future.
He didn't want to spend the money, and so just complains occasionally about the speed.
For whatever reason, there are more than a few people who will just use old, failing, technology and bitch about it rather than fix the issue.
So the "compressed IPv6 address" has the low order bits used to reflect an IPv4 address. But I thought the low order bits were going to be MAC address bits in IPv6? The two seem inconsistent.
I have no experience whatsoever with ipv6, but try to google "ipv6 ipv4 interoperability" and you will find lots of info about it.
This isn't a reason for migrating to IPv6 (although new routers with more TCAM - Ternary Content Addressable Memory) would also likely make implementing IPv6 easier.
The problem is the large number of networks that are being advertised, coupled with the number of locations that want a full BGP feed because their networks are multiply homed. Migrating to IPv6 will allow some reduction of network tables - if only because organizations with a single location that currently have multiple IPv4 networks can be allocated a single IPv6 network (and that might have a knock-on effect for organizations that are multiply homed.) It will work with organizations that are willing to tie themselves to a single ISP.
(Yes, I know that IPv6 builds in automatic address provisioning, intended to make deployment easier - but I still think that renumbering your network will be enough of a problem that there will continue to be ISP lock-in enough to encourage large organizations to get their own network numbers outside of an ISP's range.)
From the book, "Road Accidents: Prevent or Punish?"
The British road engineer J. J. Leeming, compared the statistics for fatality rates in Great Britain, for transport-related incidents both before and after the introduction of the motor vehicle, for journeys, including those once by water that now are undertaken by motor vehicle: For the period 1863–1870 there were: 470 fatalities per million of population (76 on railways, 143 on roads, 251 on water); for the period 1891–1900 the corresponding figures were: 348 (63, 107, 178); for the period 1931–1938: 403 (22, 311, 70) and for the year 1963: 325 (10, 278, 37). Leeming concluded that the data showed that "travel accidents may even have been more frequent a century ago than they are now, at least for men".
I noticed no one had mentioned LISP. I don't completely understand it, but I'll add my two cents anyway.
LISP is supposed to help with routing table exaustion and keep the global routing tables lean. It does this with a distributed database to basically map out endpoints and create tunnels around the internet. This is so no one router on the internet needs to have a full table.
In the short term for backwards compatibility, endpoints will be identified with IPv4 or IPv6 addresses, but it seems to work with any unique ID, like a serial number or GPS coordinate.
Locator/Identifier Separation Protocol (LISP)
My additional two cents...
I realize I'm risking any credibility I might have by mentioning anything related to bitcoin, but I think it's an interesting idea worth stating. Although I don't have any interest in using bitcoins as a currency, I think the underlying technology is interesting and could be useful in other applications.
The idea is for organisations to "mine" for their IPv6 allocation. They can then use their "wallet" to sign their BGP advertisements so that their peers can be certain (for various values of certain) they own that prefix. This also has the effect of decentralizing the allocation of resources, and considering the vastness of the address space of IPv6, it would be a waste of time for anyone to attempt to mine all of it and hoard it.
Really, even if you are completely ignorant about it, it does not take much more than a short reading to see how simpler IPv6 is. That's why it corrects so many issues.
The problem with IPX style local names assignment is in security. Doing it in the open, wild Internet is a certain way to destroy it. The nearest option that's actualy usable is dynamic DNS, and it's quite widspread.
Rethinking email
Because of the size if ipv6 addresses you can divide prefixes up geographically (as in graph theory, not necessarily how the world is divided).
So you get part of the address saying Europe/Netherlands/xs4all/my home. This means a router at xs4all looks at an address for USA and it knows to what interface it should be routed, a single prefix rule.
They hate the Internet and have spent twenty years stealing from the public by charging more than a fair price for their equipment. Also, they have caused much of the downtime on the Internet because they now tie licenses to hardware serial numbers to prevent companies from having spare equipment. Well, you can have the spare equipment, but they do not allow you to use it. My last employer was put out of business by cisco because cisco wouldn't give us a license to run the software we bought on our spare ASA. My employer before that went out of business because we couldn't afford to replace our ten+ year-old cisco 2501 routers at our seventy-seven restaurants because cisco charges an unfair price for routers. After not being able to process credit cards at locations because cisco didn't allow us to be able to afford replacement equipment, we had to sell out to a competitor.
cisco is the most Republican company on the planet. They refuse to sell their equipment at a reasonable price. They refuse to allow us to run the software we own on spare equipment. Finally, they refuse to fix bugs in software to force you to buy entirely new equipment.
Are there incentives of any kind for operators to think twice before making piecemeal routing advertisements? Is there any cost for multi-homing every rinky-dink company who thinks they are important enough to warrant such misuse?
Now that IPv4 resources are gone do operators pay out any penalty when they go off and start announcing random piecemeal /24's right and left?
I don't care if the penalty is simply a listing on a global wall of shame.
While IPv6 stands to reduce absolute need for disaggregation it will only be effective in doing so if there is some mechanism by which unnecessary advertisements carry a cost.
Can someone explain me how a protocol with bigger addresses and bigger routes fixes
a hardware resource problem.
I've been a Cisco networking guy for 10+ years - the 6500 series is a Distribution/Core technology for the LAN - it's definitely been milked over the years but the 4500 series is basically designed to phase it out
some of the 7600 routers (the older bricks) - I can also understand - but seriously - if you are a core internet provider, why the hell are you using a 6500 router for the BGP routing table of the internet? Put that thing in a dorm room and buy yourself an ASR 9000
RB
----------
ah honey, we're all resplendent - Bill Mallonee
This is really a different problem. We knew the BGP problem was coming, but far too many ISPs didn't bother to do the upgrades and replacements needed to address it. IPv6 adoption wouldn't have made much difference to it. At most, it would have delayed it as we neared the maximum number of IPv4 routes that older routers could handle.
"every router knowing about every destination network"
That's just not true. Your TCP/IP stack is a router and I guarantee you it doesn't know every destination network. It only knows a small set of destination networks, usually just 2. Your gateway is a router and it probably only knows 3 networks destination networks.
That's what makes IP so flexible. The end nodes can be stupid, and the intermediate nodes can be nearly as stupid. No router has full knowledge about all the networks.
If there's a problem here, it's carriers not working hard enough to simplify their networks. The system requires constant housekeeping. But dispersed knowledge, continuous chatting among intermediate routers, and every subnet being vigilant about housekeeping, is the only known algorithm that can scale to the size of the Internet.
It *WAS* 'the entire internet', barring any routers that didn't have that 512k limitation. My home internet was down from 12am to 6am yesterday, and it was definitely the edge routers (tracepath could make it up to the exit hosts for their internal network, but packets from there out were spotty. Sometimes you'd get a reply back, othertimes nothing. And this was initial on some links, then as propogation spread on all links until at some point after I went to bed they finally sorted that shit out.
Point is this was a fuckup of global proportions.
Really makes me think people aren't taking this 'decentralized network technology' seriously, given how easily major outages are generated against it for extended periods of time.
The TCAM stores IPv4 and IPv6 prefixes at same time so this issue is relevant to both families of address space.
If you use SLAAC to automatically configure an address, it does it by putting the MAC (rather, EUI-64) address in the lower 64 bits. If your address comes from something other than SLAAC then it doesn't need to have the MAC address there.
Heh, did you hear the one about the nerd who was so out of it that he expected his parents to upgrade their router?
I live and work in the UK but support offices in the US, Europe and SE Asia. Yesterday some of our network monitoring services were insisting our whole office in South Carolina was offline, despite the fact that I was at that moment screwing around with their servers remotely trying to figure out why some of our services wouldn't connect to some of our other services, pretty much bringing business completely to a halt. TWC swore up and down the fault was not with them, till eventually they acknowledged that yes, half of our businesses websites didn't work and and, yes, any traffic routed to/from BT (Britain's largest telecomm) was not reaching SC. That was yesterday, 7:30AM EST. Just now, 4:30PM EST they still have not "fixed the problem" as "not enough users have been affected." We've given up on them being useful any time soon and have routed the SC office's business-critical services through our office in Germany just to get things moving again.
I mean, I'm just IT but isn't someone at the top going to start asking these ISPs who is going to compensate them for business lost?
Nothing gets published even though I was an ex moderator. What ./ crash like google, facebook, yahoo is going to do. I bet this post gets deleted.
Even if everyone makes a serious attempt to switch to IPv6 right now, IPv4 will be around for a while. There is not enough hardware available to replace the hardware that is not able to deal with IPv4 only. I have been ready for years. I am irritated that I cannot access anything via IPv6. As for the falsehood that we will never run out of IPv6 address's, look again. There is an end. It is way out in the future, but with everything being connected to the net, including pets, the end is coming. I hope they are working on IPvSomething past 6. We will need it.
we just run a bridge to IPv4 so it looks like IPv4 to the rest of you.
-- Tigger warning: This post may contain tiggers! --
It is a problem made five times worse by the extreme high HD-ratios needed to keep IPv4 alive. If we switch to IPv6, we can go on much longer before this becomes a problem again.
It may become a problem again after IPv4 has been abandoned as the network keeps growing. Something scaling better than BGP would be nice. I predict a more scalable solution is going to need more addresses - no problem for IPv6 but would make such a scalable solution unusable with IPv4.
Do you care about the security of your wireless mouse?