Slashdot Mirror
ISP Fights Causing Netflix Packet Drops
An anonymous reader writes "We've been hearing more and more reports of ISPs throttling Netflix and other high-bandwidth services lately. The ISPs have denied it, and even Netflix itself seems to believe them. If that's the case, what's going on? Well, according to this article, the blame still lies with the ISPs. While they may not be explicitly throttling connection speeds, they're refusing to upgrade network connections as they demand more money from content distributors. For example, Netflix pays Cogent to distribute their internet traffic. Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades. 'There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. ... With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer. "Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."'"
If companies provide network access they should not be be allowed to be a content provider. Too much conflict of interest and they can concentrate on properly managing and not OVERselling their network.
by TheSpoom (715771) Uncaring Linux user here. I have nothing to add to this but please continue. *munches popcorn*
Cogent has a long history of instigating peering disputes with other networks. Normally I'd complain about Verizon's behavior but this is Cogent we're talking about. They have -zero- credibility.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Oddly enough Netflix is running well for me on Time Warner in southern California. I am no fan of Time Warner, but House of Cards (the major hot button that people seem to be using as the yard stick for performance this week) is done buffering HD by about half way through the opening credits.
The majority of the articles that I have seen all seem to mention Verizon, with little to no mention of the other ISPs. Maybe the real issue is a fight between Verizon and Cogent? If that is the case, what is the solution? Do we recommend allowing Verizon to acquire more content producers so that they stand to benefit by pushing their content back the other way, thereby restoring the in/out ratio?
I've been saying this for ages! Even mentioned this here on slashdot. Peering is peering. They are not degrading performance by configuration, they just let the link get congested. How do any of the proposed net neutrality laws address this issue? Answer is, they don't. To me that means that Net Neutrality laws are about something different than neutrality. More likely with government regulation, it becomes Net Control. With that, increased stiffing and limiting reaction to market dynamics, not improving it.
Look, for years and years, peering agreements have been based on the idea that peering between partners would have roughly equal inbound and outbound traffic patterns. When one partner starts pushing more traffic out than in, it signals an imbalance in the connection, and actually the burden.
Let me explain: internet routing is based on autonomous system (AS) hops. Cogent is an Autonomous System, so is Verizon, Level3, etc. If I'm connected to one AS, and I need to send a packet to someone on another AS, the router within my AS attempts to deliver that packet to the closest exit point for that AS. So for example, if I'm sending packets from Cogent to a user on Verizon, then Cogent will offload the transport of the packet to the closest point possible (usually in the same city). The idea being that Cogent doesn't know where the user is on verizon, so they just want to send it to verizon using an Exterior Gateway Protocol (e.g. BGP - which is AS hope based), and then verizon will use their Interior Gateway Protocol (e.g. something like OSPF) to deliver it most efficiently inside of their network.
And therein lies the problem with unequal inbound and outbound peering situations. If Cogent is 80% inbound and 20% outbound with a specific peer, that is usually shouldering the burden for most of the transport distance and cost for that packet. If it's 50/50, then the burden is the same, and they're equal peers.
In the internet routing world there are tier1 providers, tier 2 providers, etc etc based on how many peers they have. But of the tier 1's - not everyone is created equally. Cogent has Tier 1 status, and their POPs are all interconnected, but they don't have as many geographic POPs as say a Verizon. As a result, they dump packets to their peers as local as possible and those packets are routed across the internet by the peers. Their piers on the other hand, for the burden that *should* be cogents have fewer locations that they peer with cogent due to their limited # of POPs. So even for the outbound to cogent traffic, they end up shouldering a disproportionate amount of that traffic transporting it to one of cogent's pops.
And then you have companies like netflix. Netflix buys bandwidth from these low cost tier1 providers, who are low cost because they don't share the transport burden with the real tier 1s. And when congestion happens, and the real tier1's tell cogent "sorry you're out of bounds withe contract because you don't have an even inbound / outbound ratio" cogent decides not to pay the penalties for the uneven traffic patterns. Instead, they let congestion and packet drops happen. The Netflix comes along, and says "Oh yeah, this provider, they're terrible for netflix traffic - they don't upgrade their pipes." They do this knowing full well the economics of internet peering. How do they know full well? Because they don't even want to pay cogent. All the while that they're letting the tier1 take the heat for the crap peering situation, they're approaching said tier1 saying "Hey, nice network you got there. It would be a shame if someone publicized bad information about its performance. Hey, you know how you can prevent that from happening? Join open connect! All you have to do is host our hardware, provide power, and data center space, and cooling, oh and connectivity for free and your network will look great for netflix customers who won't have to suffer through this peering situation you have going on with Cogent."
And now, Netflix, not being a back bone at all, has managed to make you look bad and then tell you in order to look good you should host their hardware and give them free transport... And because /. is /. Netflix comes out looking like the victim here...
Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.
Why? A lot of people might only get internet, or faster internet anyway, BECAUSE of netflix.
If I were not streaming stuff on Netflix I might very well just use a cellular internet connection and not get cable internet at all. Netflix is helping the ISP's make money, and Netflix should gain some benefit from that fact as a result.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Why should Verizon expect to have an even in/out ratio? They sell the vast majority of their customers asymmetrical connections.
"National Security is the chief cause of national insecurity." - Celine's First Law
They SELL connections to their customers. Cogent isn't paying Verizon. The peering they're talking about is an even trade with neither company paying the other. When it ceases to be an even trade, it's time for negotiations.
It has been a while since I was in the peering game - that was back when UUNet meant something and please get off my lawn - but here's the general idea:
To provide access to the Internet - either as a content host or a content consumer - you need a full view of routes to get to all other ISPs where your sources or destinations are. To get the routes, you need "peering" with the other ISPs or you need to buy "transit" from some other ISP that gets their routes via peering or transit.
Some ISPs are, frankly, more important than others because they provide access to more subscribers or more content than others - they used to be called "Tier 1" ISPs. Peering is valuable because it's traffic you're exchanging for free that you could otherwise be charging a lot of money for. Tier 1 ISPs generally agree to peer with each other because they all need each other, and it makes economic sense for them to say they're all on an equal footing - they were "peers." The economic rationale was because Tier 1 ISPs had to pay for large national or global networks, while Tier 2 or 3 ISPs had small or regional networks and that the Tier 1 ISP was bearing most of the cost of delivery. Traffic ratios were preferred to be equal (content vs. users) for peering, because if you're a content host with one datacenter and some outbound circuits, your cost is far less than having a big national network to serve end users - so web hosting/colo provider ISPs had a harder time getting peering with the big consumer/business access providers. The Tier 2 or Tier 3 ISPs would peer with each other freely because they had equivalent footprints, etc., but the big guys knew that access to their network was extremely valuable and it would be foolish to give it away for free.
So if you're a smaller ISP, and *you* need the Tier 1 more than *they* need you, don't expect to get peering. The ISP will tell you to buy transit from them, or at least buy transit from someone else who does (and the fewer "hops" to get from you to them, the better for your customers). Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said, I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections to cut costs.
So what's happening here most likely is that Cogent has either bought transit from Verizon and doesn't want to buy more and says "peer with us, we won't buy more." Or Cogent does have peering with Verizon but VZ has said, in effect, "you are not our 'peer'" and beyond a certain amount of peering bandwidth, you should start buying." Cogent is using Netflix to try to argue that "our content is more important to Verizon customers than the other way around," and Verizon is saying, "Um, nope." I won't say who I think is right or wrong here, but this is not the first time Cogent has had peering fights with other large ISPs and I think you can see a pattern here.
"95% of all Slashdot
Sure, there is waste in the government and we all hear about it because it's our right to know. Now, how much waste is in privete industry? Ever filled out a form that you know very well will sit unread in a filing cabinet until it composts? That's waste. Ever gotten a credit card offer that you don't even qualify for? What about those crazy rules that turn a simple matter into a debacle?
Everything on up to a gulfstream for the CEO, all waste.
But when you look at actual costs for services provided, the government provided service often works out cheaper. That's why private corporations object so strenuously to government providing services even where competition is permitted.
Simple solution, Netflix should alter their client so it always echos the data back. That should balance the transfer and make Verizon very happy.