ISP Fights Causing Netflix Packet Drops

An anonymous reader writes "We've been hearing more and more reports of ISPs throttling Netflix and other high-bandwidth services lately. The ISPs have denied it, and even Netflix itself seems to believe them. If that's the case, what's going on? Well, according to this article, the blame still lies with the ISPs. While they may not be explicitly throttling connection speeds, they're refusing to upgrade network connections as they demand more money from content distributors. For example, Netflix pays Cogent to distribute their internet traffic. Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades. 'There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. ... With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer. "Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."'"

Comments

Can confirm

[schneidafunk]

I've been streaming netflix and noticed a lot of stoppage & buffering. I ended up upgrading my wireless router thinking that was the problem. Nope, it's still happening.

Re:Can confirm

[dave562]

Oddly enough Netflix is running well for me on Time Warner in southern California. I am no fan of Time Warner, but House of Cards (the major hot button that people seem to be using as the yard stick for performance this week) is done buffering HD by about half way through the opening credits.

The majority of the articles that I have seen all seem to mention Verizon, with little to no mention of the other ISPs. Maybe the real issue is a fight between Verizon and Cogent? If that is the case, what is the solution? Do we recommend allowing Verizon to acquire more content producers so that they stand to benefit by pushing their content back the other way, thereby restoring the in/out ratio?

Re:Can confirm

[msauve]

Why should Verizon expect to have an even in/out ratio? They sell the vast majority of their customers asymmetrical connections.

Re:Can confirm

[schnell]

It has been a while since I was in the peering game - that was back when UUNet meant something and please get off my lawn - but here's the general idea:

To provide access to the Internet - either as a content host or a content consumer - you need a full view of routes to get to all other ISPs where your sources or destinations are. To get the routes, you need "peering" with the other ISPs or you need to buy "transit" from some other ISP that gets their routes via peering or transit.

Some ISPs are, frankly, more important than others because they provide access to more subscribers or more content than others - they used to be called "Tier 1" ISPs. Peering is valuable because it's traffic you're exchanging for free that you could otherwise be charging a lot of money for. Tier 1 ISPs generally agree to peer with each other because they all need each other, and it makes economic sense for them to say they're all on an equal footing - they were "peers." The economic rationale was because Tier 1 ISPs had to pay for large national or global networks, while Tier 2 or 3 ISPs had small or regional networks and that the Tier 1 ISP was bearing most of the cost of delivery. Traffic ratios were preferred to be equal (content vs. users) for peering, because if you're a content host with one datacenter and some outbound circuits, your cost is far less than having a big national network to serve end users - so web hosting/colo provider ISPs had a harder time getting peering with the big consumer/business access providers. The Tier 2 or Tier 3 ISPs would peer with each other freely because they had equivalent footprints, etc., but the big guys knew that access to their network was extremely valuable and it would be foolish to give it away for free.

So if you're a smaller ISP, and *you* need the Tier 1 more than *they* need you, don't expect to get peering. The ISP will tell you to buy transit from them, or at least buy transit from someone else who does (and the fewer "hops" to get from you to them, the better for your customers). Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said, I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections to cut costs.

So what's happening here most likely is that Cogent has either bought transit from Verizon and doesn't want to buy more and says "peer with us, we won't buy more." Or Cogent does have peering with Verizon but VZ has said, in effect, "you are not our 'peer'" and beyond a certain amount of peering bandwidth, you should start buying." Cogent is using Netflix to try to argue that "our content is more important to Verizon customers than the other way around," and Verizon is saying, "Um, nope." I won't say who I think is right or wrong here, but this is not the first time Cogent has had peering fights with other large ISPs and I think you can see a pattern here.

Re:Can confirm

[msauve]

If Cogent is using Verizon for transit, yes, they should pay for that. If the peering is strictly to deliver content to Verizon's customers, that bandwidth is already being paid for by Verizon's customers.

The only time a payment for straight peering makes any economic sense is if the smaller ISP doesn't generate (either in or out) enough traffic to justify the equipment and maintenance costs of the interconnect. (Anti-competitive reasons are another thing). In the case of Cogent(Netflix)/Verizon, the existing interconnect is obviously saturated, so there's no reasonable excuse for not improving it. Verizon's customers are clearly paying for it, as measured by the amount of traffic they're trying to pull through it. As far as I can tell, this has nothing to do with transit.

Re:Can confirm

[Spazmania]

The currently accurate term for "tier 1" is "transit-free network." A network which purchases no Internet transit service is transit-free. All packets entering or leaving their network are to customers or to peers.

The distinction between a "transit" connection and a "peering" connection is that a transit connection is to "the rest of the Internet" while a peering connection is only to "you and your customers."

Network vs Content providers

[future assassin]

If companies provide network access they should not be be allowed to be a content provider. Too much conflict of interest and they can concentrate on properly managing and not OVERselling their network.

Re:Network vs Content providers

[dave562]

In your mind, where does Cogent fall on the network provider / content provider divide? They are providing network access, but they are also (as far as I know) providing the infrastructure for Netflix's CDN.

Re:Network vs Content providers

[thule]

Cogent likes to think of themselves as a pure bandwidth company. No frills bandwidth for a great price. No content, no VoIP, nothing. They have colocation data centers, but that came when they purchased a company for their network.

Re:Network vs Content providers

[Lawrence_Bird]

That isn't even the real point, though I understand where you are going with it. The real issue here is that Verizon, Comcast, Cablevision, etc all have agreed to provide their customers with X/Y data connections for a hefty monthly fee. They are refusing to do what it takes for those customers to be able to use what they have paid for. In effect, we are back to the early internet days of ISPs oversubscribing dialup lines except now it is oversubscribing routing equipment during peak hours.

Re:Network vs Content providers

[tlhIngan]

Technically, what's happening is peering agreements.

Verizon and Cogent agree to carry each other's traffic. If the flows are roughly equal, then they usually come to a "free" agreement - Verizon carriest traffic from Cogent and Cogent carries Verizon's traffic for free.

It's when flows are unequal that's when the fights happen - basically the one getting more traffic from the other starts demanding money because they're sharing more of the burden of the traffic.

So what happened here is Cogent is passing more traffic to Verizon than Verizon is passing to Cogent. This usually results in the formerly free peering arrangement to become a paid one, and Verizon starts demanding money for the unequal flow.

It turns out the one demanding party has a new tool in the shed - stop upgrading the ports so now Cogent's traffic is maxing out the port and getting dropped packets.

Sometimes the one having more traffic does nasty tricks as well - including rerouting their traffic through a more expensive link to force port upgrades (i.e., Cogent sees Verizon pays more if traffic goes through another port, so they route all traffic through that port). Or they move traffic between ports forcing port upgrades on all ports.

Re:Network vs Content providers

[TechyImmigrant]

>It's when flows are unequal that's when the fights happen - basically the one getting more traffic from the other starts demanding money because they're sharing more of the burden of the traffic.

That makes no sense. Both sides are sharing the burden, regardless of the direction of the traffic. You still have the packets on your wires. It doesn't matter if they're going left or right.

Re:Network vs Content providers

[jeffmflanagan]

Those studies are probably misleading. It takes longer to repair an underground line, but it takes an idiot to sever it, when above-ground lines are regularly severed by weather events.

Re:Network vs Content providers

[Spazmania]

Cogent is a network provider.

The divide is based on the ownership of content or the rights to distribute that content. Netflix owns rights to distribute the content. Cogent does not.

The OP was complaining about Verizon. Verizon's cable TV service is a content provider. Verizon owns the rights to distribute that content. Verizon is also a network provider, delivering Internet access service. OP implies that Verizon's conflict of interest leads them to intentionally impair Netflix's competing content service.

Re:Network vs Content providers

[Pinhedd]

No they are not. They are promising a certain link speed between the customer and the point where the customer's traffic leaves the ISPs network; if they don't deliver on that, that's a different matter entirely. They are not promising a certain level of access to services outside of the ISPs network. When external services are aggregated the marketed rate should be attainable, but aggregation involves making certain assumptions about traffic, one of which is that traffic is more or less evenly distributed across the interconnects. If thousands of customers all try to draw traffic across the same interconnect, that interconnect gets bogged down. The rest of the network is fine, and the customer should still be able to obtain whatever their normal data rate is, but only by aggregating traffic appropriately.

Re:Network vs Content providers

[schnell]

The ISP is responsible for his peering relationships - that is what makes the ISP a part of the internet.

To indulge in a bit of reductio ad absurdam but, hey, why not:

WallaWallaNet: Hello, Verizon?
Verizon: Yes, hello?
WallaWallaNet: Hi, I'm an ISP in Walla Walla Washington, and I'd like access to your subscribers.
Verizon: Super! You can buy a T3 for $1k/month for the local loop and $7k/month for Internet transit.
WallaWallaNet: Ooh, here's the problem. I host "www.awesomepixofdogslickingthemselves.com." Your subscribers want the whole Internet - including all the pictures of dogs licking themselves that they could ever want - so I think it's incumbent upon you to peer with me for only the cost of my local loop at the nearest available peering point to me. I mean, www.awesomepixofdogslickingthemselves.com is worth enough to your subscribers to forego that money you were going to charge me for transit, right?
Verizon: Excellent point! Please mention the vital importance of www.awesomepixofdogslickingthemselves.com when you call every other major ISP on the planet and ask them for peering! I just need one more piece of information from you and we can get this all set up.
WallaWallaNet: Sure, what is it?
Verizon: What color is the sky on the planet where you live?

Netflix should know better

[Spazmania]

Cogent has a long history of instigating peering disputes with other networks. Normally I'd complain about Verizon's behavior but this is Cogent we're talking about. They have -zero- credibility.

Re:Netflix should know better

[ffsnjb]

Agreed, every time I look at any various looking glass to see why something is broken on the internet, it's Cogent dropping the ball... Remember this?

Every time I have to troubleshoot a broken internet problem, and it goes up to Tier 1 land, check the looking glass... It's Cogent being stupid. As soon as this story hit... check. Sure enough, Cogent-Level3 has ~3% packet loss the last 24 hours, with stupidly high latency. Screenshot .

Re:Netflix should know better

[Shinobi]

Yeah, Cogent really is trying every dirty trick they can to go past any contract limits etc and freeload, and then they cry loudly to the media when they get told to stick to the contract.

Chromecast Vs. Roku

[TrippTDF]

I have a 1st gen Roku, and a Chromecast. When I stream Netflix with the Roku, I seem to top out at 2, maybe three quality bars. While there's no on-screen metric for the same stream in Chromecast, the picture is noticeably better. Perhaps the Chromecast is getting a higher-quality, lower-bandwidth stream, or there's some sort of throttling based on the streaming device going on?

Re:Chromecast Vs. Roku

[evilviper]

I remember reading that Netflix uses mostly uncompressed streams to TVs and TV like devices

There's no such thing as uncompressed video streaming. Uncompressed video is astronomically large, and impossible to stream over any internet connection you've ever seen.

https://en.wikipedia.org/wiki/...

1080p@24fps is about 800Mbit/sec., something even your local gigabit LAN would struggle to sustain.

What you mean is using older, less efficient, and less computationally complex (but still lossy and still highly compressed) video codecs.

Net Neutrality laws?

[thule]

I've been saying this for ages! Even mentioned this here on slashdot. Peering is peering. They are not degrading performance by configuration, they just let the link get congested. How do any of the proposed net neutrality laws address this issue? Answer is, they don't. To me that means that Net Neutrality laws are about something different than neutrality. More likely with government regulation, it becomes Net Control. With that, increased stiffing and limiting reaction to market dynamics, not improving it.

Re:Net Neutrality laws?

[Anrego]

Yeah, but it's not like Netflix is using a free service. They are paying for that bandwidth. I assume they are paying quite a bit. More importantly, someone is selling it to them.

Re:Net Neutrality laws?

[heypete]

I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

All the peering details are here. In short: they don't charge anything. They offer direct interconnects to Netflix's CDN for free, free peering at major internet exchange points, and free, Netflix-managed hardware caches to ISPs to avoid duplicate network traffic (the vast majority of traffic stays within the ISPs internal network). For the hardware caches the ISP needs only provide power and network connectivity.

There's really no reason for ISPs to wrangle with Netflix -- there's plenty of options to avoid congestion.

Re:Net Neutrality laws?

[kqs]

Very logical. In related news, many things can make you ill, not just toxic waste. Therefore, any laws which discourage dumping toxic waste on your property are not about health, and are probably just about governmental control of companies.

Peering limits are certainly used as a rough form anti-net-neutrality, but they're not ideal; they have the pinpoint accuracy of a sawed-off shotgun at 100 yards and they are very obvious. The proposed laws tend to target the subtler, better directed forms. Don't let the perfect be the enemy of the good.

Re:Net Neutrality laws?

[petermgreen]

I disagee, the root of the problem is that a teir 1 provider is vertically integrated with a last mile provider that has monopoly or near-monopoly power in some areas.

If the last mile provider was not vertically integrated with a teir 1 it would be in their interests to set up good peering with large conent providers directly (as happens here in the UK) to avoid paying upstream transit bills but because they are verticially integrated with a teir 1 and don't have to buy transit from anyone it makes sense for them to play hardball on peering. Think about it? why are cogent involved here in the first place? Most likely because Verizon refused to peer with netflix.

Re:Net Neutrality laws?

[evilviper]

Its either traffic for your clients,

Not at all. It could easily be traffic being dumped onto your backbone by a badly behaving peer. There is no means to prevent it.

or its through a peering arrangement that you already find beneficial (or it wouldn't exist).

Not "find", but "found". As in, past-tense.

Besides, these are just circular arguments, anyway. You're continually assuming some unlisted benefit, that may or very well *MAY NOT* be there. There are many cases in which what you said doesn't remotely hold true.

Natural monpolies

[bob_super]

Water, then gas, then electricity, phone, and now internet backbone are Natural Monopolies.
Private companies should not operate in Natural Monopolies because they just add overhead cost as they operate for profit.
Obviously, with the American public convinced of the lie that government is just waste, and that private companies magically reduce waste (at the bottom, for sure), we all keep on shelling massive amounts of cash for basic services. Cash that would better be invested in keeping the US moving forward, rather than lining the pockets of terrible profiteers like my 401k...

Re:Natural monpolies

[sjames]

Sure, there is waste in the government and we all hear about it because it's our right to know. Now, how much waste is in privete industry? Ever filled out a form that you know very well will sit unread in a filing cabinet until it composts? That's waste. Ever gotten a credit card offer that you don't even qualify for? What about those crazy rules that turn a simple matter into a debacle?

Everything on up to a gulfstream for the CEO, all waste.

But when you look at actual costs for services provided, the government provided service often works out cheaper. That's why private corporations object so strenuously to government providing services even where competition is permitted.

Re:as always streaming sucks, torrents rule

[Anonymous Coward]

That didn't stop the NSA

"Lopsided"?

[K. S. Kyosuke]

Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement.

I'm not quite sure I understand what's "lopsided" about that. It still flows through both of the two networks, and the companies of the receiving end are still the ones cashing in from end users, and they still very much like their users paying extra money for extra data transfers, don't they?

And this is how it's supposed to work

[Anonymous Coward]

Look, for years and years, peering agreements have been based on the idea that peering between partners would have roughly equal inbound and outbound traffic patterns. When one partner starts pushing more traffic out than in, it signals an imbalance in the connection, and actually the burden.

Let me explain: internet routing is based on autonomous system (AS) hops. Cogent is an Autonomous System, so is Verizon, Level3, etc. If I'm connected to one AS, and I need to send a packet to someone on another AS, the router within my AS attempts to deliver that packet to the closest exit point for that AS. So for example, if I'm sending packets from Cogent to a user on Verizon, then Cogent will offload the transport of the packet to the closest point possible (usually in the same city). The idea being that Cogent doesn't know where the user is on verizon, so they just want to send it to verizon using an Exterior Gateway Protocol (e.g. BGP - which is AS hope based), and then verizon will use their Interior Gateway Protocol (e.g. something like OSPF) to deliver it most efficiently inside of their network.

And therein lies the problem with unequal inbound and outbound peering situations. If Cogent is 80% inbound and 20% outbound with a specific peer, that is usually shouldering the burden for most of the transport distance and cost for that packet. If it's 50/50, then the burden is the same, and they're equal peers.

In the internet routing world there are tier1 providers, tier 2 providers, etc etc based on how many peers they have. But of the tier 1's - not everyone is created equally. Cogent has Tier 1 status, and their POPs are all interconnected, but they don't have as many geographic POPs as say a Verizon. As a result, they dump packets to their peers as local as possible and those packets are routed across the internet by the peers. Their piers on the other hand, for the burden that *should* be cogents have fewer locations that they peer with cogent due to their limited # of POPs. So even for the outbound to cogent traffic, they end up shouldering a disproportionate amount of that traffic transporting it to one of cogent's pops.

And then you have companies like netflix. Netflix buys bandwidth from these low cost tier1 providers, who are low cost because they don't share the transport burden with the real tier 1s. And when congestion happens, and the real tier1's tell cogent "sorry you're out of bounds withe contract because you don't have an even inbound / outbound ratio" cogent decides not to pay the penalties for the uneven traffic patterns. Instead, they let congestion and packet drops happen. The Netflix comes along, and says "Oh yeah, this provider, they're terrible for netflix traffic - they don't upgrade their pipes." They do this knowing full well the economics of internet peering. How do they know full well? Because they don't even want to pay cogent. All the while that they're letting the tier1 take the heat for the crap peering situation, they're approaching said tier1 saying "Hey, nice network you got there. It would be a shame if someone publicized bad information about its performance. Hey, you know how you can prevent that from happening? Join open connect! All you have to do is host our hardware, provide power, and data center space, and cooling, oh and connectivity for free and your network will look great for netflix customers who won't have to suffer through this peering situation you have going on with Cogent."

And now, Netflix, not being a back bone at all, has managed to make you look bad and then tell you in order to look good you should host their hardware and give them free transport... And because /. is /. Netflix comes out looking like the victim here...

Netflix should get benefit from desirability

[SuperKendall]

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

Why? A lot of people might only get internet, or faster internet anyway, BECAUSE of netflix.

If I were not streaming stuff on Netflix I might very well just use a cellular internet connection and not get cable internet at all. Netflix is helping the ISP's make money, and Netflix should gain some benefit from that fact as a result.

Re:Netflix should get benefit from desirability

[vux984]

Right now Netflix is getting to benefit at the expense of ISPs. [...] My issue here is that Netflix is not a good champion to use to make the case for ISPs being tightfisted when it comes to circuit maintenance.

If the ISP cannot transfer me the traffic that I am paying them to transfer me, then they need to upgrade their circuits.

If they legitimately cannot afford to do that, then they need to charge me more money.

But if they cannot afford to do that, why is their investment news a string of "profits up 40% in 2013, ARPU up 9.4%, 14.7% year over year increase in FIOS revenues, 78c EPS 2013 vs 64c EPS in 2012, ... etc...

Imagine if your neighbor turned his house / apartment into a concert venue because there were not any ordinances to prevent it. All day long, you could not find a place to park, could not have guests over because they could not park, suffered brown and black outs because the music equipment kept blowing out the local transformers, etc. How would you handle that?

My neighbor and I both subscribe to Verizon to provide "parking and electricity" in this situation. So if Verizon can't provide
"parking and electricity" to me per our agreement, because my "neighbor" is using too much, then Verizon needs to build more parking and provide more electricity. If they have to charge me and my neighbors more, so be it... but see above regarding large increases in revenues and profits... it seems Verizon is simply failing to upgrade our electricity and parking and pocketing the money we paid them to do that instead.

Re:Netflix should get benefit from desirability

[vux984]

Peering arrangements are private contracts that don't have anything to do with end users.

And if you demand your ISP gives their peering partners anything they want

I don't demand that at all.

There is a middle ground where the ISP can figure out how to deliver traffic to me without having to deliver every other ISPs traffic for free. Those peering contracts can be arbitrarily detailed.

Re:Netflix should get benefit from desirability

[Anonymous Coward]

And if you demand your ISP gives their peering partners anything they want, the peers will take massive advantage, and dump tons of their own traffic on your ISP's links, to get delivered across the country by them.

That is not possible, and would be a violation of internet engineering rules, and a breach of contract.

A peering arrangement, in contrast to a transit arrangement, only advertises routing prefixes within the AS of the eBG router, and is not obliged route any traffic for routes it has not advertised (it is obliged to route all routes it does advertise). Any sensibly configured eBG router will have IP filtering so that it doesn't deliver traffic for prefixes it hasn't advertised, and it would take deliberate manual configuration in contravention of IER to forward traffic for routing to a peer which has not advertised route availability for the traffic in question.

On the other hand, Cogent is acting as a transit provider while pretending to be a peer, which explains why Verizon is being hostile. Cogent is using an early exit strategy, whereby they route into nationwide ISPs networks at the closest available route, knowing that nationwide ISPs have internal nationwide networks for delivering traffic within their network. They could institute route filtering so each peering point only advertises routes to nearby customer edge networks, and Cogent is forced to carry traffic cross country before dropping it on Verizon and AT&T, but this would complicate network management, create potential routing failures within their network, and not be fair to other peers who may not have nationwide delivery.

Instead I think Verizon are taking the most amicable option available to them, letting Cogent saturate their peering links and expecting Cogent to build more peering links or deliver their traffic closer to it's actual destination as every transit ISP is expected to do.

We're Surrounded by Morons.

[VortexCortex]

If I gave you a decentralized network capable of surviving nuclear war, routing packets around cities that vanished in moments, and you then built a World Wide Web of Data Silos then I'd be within my rights to call you all fucking morons!

STORE AND FORWARD. How the hell can't you realize this is the decentralized solution that the Internet needed, not a centralized Server / Client cluster fuck? What is collocation? IT'S WHAT YOU GET FOR FREE WITH STORE AND FORWARD, idiots. Oh there's all that Youtube, Netflix, etc., bandwidth? Well, what if I told you I could reduce your peering bandwidth to ONE COPY of each resource? That way if your neighbor recently watched a show or cat video you could download it directly from them or the upstream cache they're connected to? A router should only ever have been one part of the node, the cache is an essential part, as part of any strategy for load distribution. Ah, but you don't need to keep a copy of each resource at each node if you index the data via distributed hash table -- it's not rocket surgery fools.

How else do you think the Space Internet will work? NASA's DTN and even old-timey HAM Packet Radio are smarter about data than the fucking web! Unlike the WWW, they use Store and Forward. HTML allows lightweight dynamic pages to pull in heavy static resources, if only assets had a <... hash="SHA-512/Base64:MDVjMjg4YmY2ZGV...hMGEx==" hmac="Base64:..."> then secure pages could pull in unsecured static resources and browsers could verify their validity without "mixed content" warnings -- Oh but that would mean the architects would have to ACTUALLY KNOW WHAT THE FUCK THEY WERE DOING. I'm surrounded by morons. I mean, you could even just use your HTTP-AUTH proof of knowledge to key your ciphers and eliminate the Certificate Authority MITM, but noooo... Morons, I Say! MORONS!

Re:We're Surrounded by Morons.

[SuricouRaven]

I really couldn't agree more. That would be a great way to do things. It wasn't built in from the beginning because it wasn't practical until recently - storage just cost too much. It isn't being built in now because the existing models work-ish, and no influential organization has been willing to lend their backing to deploying the new technology.

I did have the idea of encoding the hash addresses as a 'magic directory' in HTTP - eg, http://your.server.com/CANary/...//filename. That way any browser or software aware of the address form can run a SHA search of reachable caches, while any non-aware software (or if the cache search fails) just interpret it as an HTTP address and get the file the old way. Even if there are no caches in reachable, it'd still save a lot of IMS requests when re-visiting a website.

Making personal or household devices public cache servers is probably not the best idea though. Privacy concerns - it wouldn't take long for someone to write a script that queries a specified cache for the top thousand video hashes from the popular porn sites and see if they have it stored already.

sell is the key word. Cogent not paying Verizon

[raymorris]

They SELL connections to their customers. Cogent isn't paying Verizon. The peering they're talking about is an even trade with neither company paying the other. When it ceases to be an even trade, it's time for negotiations.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Yes, they SELL connections to their customers, so the incoming flood of traffic from Cogent has already been paid for by Verizon's customers that are trying to watch their shows on Netflix. Verizon is trying to double-dip. So what if the traffic is asymmetric? If it's that big of a problem, the continued performance problems caused by Verizon's intransigence could be solved by a massive reduction in the customer base once they find that they can't watch Netflix.

Re:sell is the key word. Cogent not paying Verizon

[msauve]

Customers PAY for those connections. Verizon's customers are paying to receive traffic from the Internet. Whether that's slashdot or Netflix doesn't matter, it behooves Verizon to deliver the service their customers are PAYING for.

Re:sell is the key word. Cogent not paying Verizon

[i.r.id10t]

so my ISP should be paying me instead of vice versa? i certainly download much more than I upload....

Re:sell is the key word. Cogent not paying Verizon

[sjames]

Every packet coming from Cogent to Verizon is because a Verizon customer paid for it. It really is as simple as that.

It only seems complicated because the kooky business people make it complicated trying to pass their costs off onto others.

Re:sell is the key word. Cogent not paying Verizon

[NormAtHome]

I know this topic has been beaten to death here, but I see it the same way you do. Verizon customers pay a toll (their monthly charge for internet access) to use Verizon's connection to the internet as a whole. No Verizon customer should have their data throttled no matter what site they are accessing as long as they are in compliance with Verizon's TOS.

Re:sell is the key word. Cogent not paying Verizon

[JesseMcDonald]

No, because the ISP owns the infrastructure and you're the intended recipient. They're not asking you to deliver packets to someone else on their behalf.

Logically, everyone should just pay for the packets they send, much like you pay (via stamps) to send an envelope. A company like Netflix would pay to have their content delivered to their customers, and would recoup the cost of "shipping" that content across the Internet in the form of subscription fees. Unfortunately, while that might work for Netflix, it would be problematic for sites serving small amounts of traffic to a large number of visitors—we still lack a practical and widespread means of micropayment.

Of course, one significant difference between the Internet and the postal service is that the postal service is required to deliver (or return) every envelope they accept—they can't just drop excess mail in the nearest incinerator when they become overloaded. At that, it would still be an improvement over the current Internet model, where not only do you have to pay for packets which were never delivered due to congestion or technical problems, you even have to pay for packets you never requested and had no opportunity to opt out of.

Re:sell is the key word. Cogent not paying Verizon

[Sancho]

And this is why I'd really like to see Internet providers become common carriers. The Internet has become much too important to let squabbles like this endanger it.

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

The thing is, private peering doesn't cost you anything.

No.... the data transferred still uses data capacity.

Lost revenue. Cogent is using Verizon's free peering to compete against Verizon, by selling the same bandwidth (sustained gigabits of data transfer) to Netflix, that Netflix would have to pay four to five times as much for if they bought an internet connection from Verizon.

In principle, to protect their revenues: the Tier1's need to ensure that they get paid by the other party connecting to their network for data coming into or leaving their network, that is using up some of their network capacity.

They either get paid via a transit fee from the party connecting, or by gaining an equivalent amount of usage of the peer's network.

Netflix vs Google

[careysb]

It seems that ISP's are so concerned with Netflix's bandwidth suck that they try to get away with throttling. What about Google? Supposedly Google's web crawlers account for the largest single chunk of Internet bandwidth. (Ok, educate me)
--
Sent from my IBM 360 mainframe

Re:Netflix offers a colo/CDN bandwidth saver for I

[sexconker]

Netflix has a program where they'll colocate some servers containing a content cache on a segment of the ISPs network so that their peering connections aren't getting beaten to death--why wouldn't these companies get involved in such a program other than as a means to squeeze more money from Netflix, their subscribers, or both.

Because you have to agree to Netflix's terms to host those things.
Everything from physical access requirements to the ol' "By the way we may host other, non-Netflix content on these things in the future, and we'll charge people for the privilege, but you'll still have to treat it as Netflix data and not expect any money for carrying it on your network".

Netflix muscled their way into favorable agreements (both with and without those storage boxes at the ISPs) when they trotted out Super HD. Now that they have a lot of those agreements in place, Netflix opened it up so (just about) everyone gets Super HD, and they aren't making any noise over it anymore.

Re: "Lopside"?

[sjames]

Simple solution, Netflix should alter their client so it always echos the data back. That should balance the transfer and make Verizon very happy.

A carrier should not be going through another carr

[horeton]

As an ISP we added a Netflix peer via "Netflix Open Connect" last week and are offloading 4Gbit per/sec at peak usage via BGP directly to Netflix rather than wasting our dia bandwidth to Cogent, Level3, or other Backbone providers. By meeting them at IXPs like the TIE and NOTA a few of many, ISPs can directly offload traffic to most major application providers for free like Netflix. Google, and Facebook just to name a few. Why Verizon wastes their DIA bandwidth for Netflix is idiotic. Verizon owns Terremark which includes NOTA. They could be directly peering with Netflix themselves. I think it's more likely they have issues keeping up with it from a cellular perspective through their own infrastructure. Going through Cogent instead of directly to Netflix defeats the point of IXPs like NOTA.

CDN/Proxy servers?

[lenne]

Netflix might be having 20000 TV-episodes and 3000 movies, but they are not evenly viewed. I bet that a caching box with even a mere 1TB disk could cover 90% of the usage.
If 10000 viewers want to see the latest House of Cards, there is no need for sending 10000 streams from one ISP to another.

Heck, even some sort of bittorrent could be used, if each viewer would allocate say 50GB for shared content.

I have a thin adsl from one ISP and a 4G to another, and a traceroute to netflix leads to two different servers, each in my country, Denmark. The adsl is even only 3 hops away.

BTW: Why do I have to type <br> to have line breaks? Why not just enter enter?