ISP Fights Causing Netflix Packet Drops

An anonymous reader writes "We've been hearing more and more reports of ISPs throttling Netflix and other high-bandwidth services lately. The ISPs have denied it, and even Netflix itself seems to believe them. If that's the case, what's going on? Well, according to this article, the blame still lies with the ISPs. While they may not be explicitly throttling connection speeds, they're refusing to upgrade network connections as they demand more money from content distributors. For example, Netflix pays Cogent to distribute their internet traffic. Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades. 'There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. ... With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer. "Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."'"

Comments

Can confirm

[schneidafunk]

I've been streaming netflix and noticed a lot of stoppage & buffering. I ended up upgrading my wireless router thinking that was the problem. Nope, it's still happening.

Re:Can confirm

[dave562]

Oddly enough Netflix is running well for me on Time Warner in southern California. I am no fan of Time Warner, but House of Cards (the major hot button that people seem to be using as the yard stick for performance this week) is done buffering HD by about half way through the opening credits.

The majority of the articles that I have seen all seem to mention Verizon, with little to no mention of the other ISPs. Maybe the real issue is a fight between Verizon and Cogent? If that is the case, what is the solution? Do we recommend allowing Verizon to acquire more content producers so that they stand to benefit by pushing their content back the other way, thereby restoring the in/out ratio?

Re:Can confirm

[Anonymous Coward]

VZ just also spent a huge chunk of money buying VZW. No spending is going on other than 'necessary'.

Re:Can confirm

[rwa2]

We could also, you know, confirm by looking at packet loss stats from Cogent to your ISP's carrier
http://internethealthreport.co...

Re:Can confirm

[Anonymous Coward]

Me too, but sometimes it works fine. I haven't figured out if I watch "battlestar gallactica" at 11pm when everyone else is asleep, or if "transformer's rescuebots" is on a particular netflix streaming server that is susceptible to all this. Does anyone know enough about netflix's arch to comment on the possibilities?

Re:Can confirm

[Anonymous Coward]

That doesn't confirm anything from this article.

Re:Can confirm

[schneidafunk]

Well that's interesting to hear, and I do have Verizon.

Re:Can confirm

[msauve]

Why should Verizon expect to have an even in/out ratio? They sell the vast majority of their customers asymmetrical connections.

Re:Can confirm

[jest3r]

Same here. For the past week or two I have been experiencing all sorts of glitches, stoppages and buffering through Netflix. My local ISP on demand service is fine though. Before last week everything was awesome!

Called my ISP about it - they said contact Netflix.

Maybe Netflix should add net neutrality to the House of Cards story arc to get the word out???

Re:Can confirm

[suutar]

I haven't been getting stoppage/buffering with Netflix, but I've been thinking I was getting lower video quality than I'm used to, which could be a less severe result of packet loss. I have definitely been getting a crapload of buffering with Amazon Instant Video, though.

Re:Can confirm

[Anonymous Coward]

Verizon's service consists of a hell of a lot more than FiOS. The vast majority of their bandwidth comes from symmetrical business fiber connections. Plenty of content is also hosted on Verizon connections as well. Prior to Netflix, there was significantly more traffic going out from Verizon's network than in.

Re:Can confirm

[schnell]

It has been a while since I was in the peering game - that was back when UUNet meant something and please get off my lawn - but here's the general idea:

To provide access to the Internet - either as a content host or a content consumer - you need a full view of routes to get to all other ISPs where your sources or destinations are. To get the routes, you need "peering" with the other ISPs or you need to buy "transit" from some other ISP that gets their routes via peering or transit.

Some ISPs are, frankly, more important than others because they provide access to more subscribers or more content than others - they used to be called "Tier 1" ISPs. Peering is valuable because it's traffic you're exchanging for free that you could otherwise be charging a lot of money for. Tier 1 ISPs generally agree to peer with each other because they all need each other, and it makes economic sense for them to say they're all on an equal footing - they were "peers." The economic rationale was because Tier 1 ISPs had to pay for large national or global networks, while Tier 2 or 3 ISPs had small or regional networks and that the Tier 1 ISP was bearing most of the cost of delivery. Traffic ratios were preferred to be equal (content vs. users) for peering, because if you're a content host with one datacenter and some outbound circuits, your cost is far less than having a big national network to serve end users - so web hosting/colo provider ISPs had a harder time getting peering with the big consumer/business access providers. The Tier 2 or Tier 3 ISPs would peer with each other freely because they had equivalent footprints, etc., but the big guys knew that access to their network was extremely valuable and it would be foolish to give it away for free.

So if you're a smaller ISP, and *you* need the Tier 1 more than *they* need you, don't expect to get peering. The ISP will tell you to buy transit from them, or at least buy transit from someone else who does (and the fewer "hops" to get from you to them, the better for your customers). Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said, I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections to cut costs.

So what's happening here most likely is that Cogent has either bought transit from Verizon and doesn't want to buy more and says "peer with us, we won't buy more." Or Cogent does have peering with Verizon but VZ has said, in effect, "you are not our 'peer'" and beyond a certain amount of peering bandwidth, you should start buying." Cogent is using Netflix to try to argue that "our content is more important to Verizon customers than the other way around," and Verizon is saying, "Um, nope." I won't say who I think is right or wrong here, but this is not the first time Cogent has had peering fights with other large ISPs and I think you can see a pattern here.

Re:Can confirm

[Anonymous Coward]

cogent is one of the largest transit networks in the world now fyi

Re:Can confirm

[msauve]

If Cogent is using Verizon for transit, yes, they should pay for that. If the peering is strictly to deliver content to Verizon's customers, that bandwidth is already being paid for by Verizon's customers.

The only time a payment for straight peering makes any economic sense is if the smaller ISP doesn't generate (either in or out) enough traffic to justify the equipment and maintenance costs of the interconnect. (Anti-competitive reasons are another thing). In the case of Cogent(Netflix)/Verizon, the existing interconnect is obviously saturated, so there's no reasonable excuse for not improving it. Verizon's customers are clearly paying for it, as measured by the amount of traffic they're trying to pull through it. As far as I can tell, this has nothing to do with transit.

Re:Can confirm

[Stormy Dragon]

Does the fact I'm a Verizon internet customer obligate them to build arbitrary infrastructure to any computer I happen to want to access? If I want to download a file from Yrui the goatherder's machine, are they now required to lay new pipe out into the wilds of Romania to reach his machine?

Re:Can confirm

[Spazmania]

The solution is: Netflix should know better than to rely on Cogent to deliver their data. Cogent has a long history of dropping packets, undersizing connections and blaming everybody else for it. Some folks will be able to stream data from Cogent and some won't. Without an alternative path to Verizon customers, this is 100% Netflix's fault.

Re:Can confirm

[Anonymous Coward]

You're not very good at trolling.

Re:Can confirm

[Spazmania]

The currently accurate term for "tier 1" is "transit-free network." A network which purchases no Internet transit service is transit-free. All packets entering or leaving their network are to customers or to peers.

The distinction between a "transit" connection and a "peering" connection is that a transit connection is to "the rest of the Internet" while a peering connection is only to "you and your customers."

Re:Can confirm

[Spazmania]

Yes and no.

In principle, you're right. Demanding payment for peering which exceeds the cost of establishing the immediate connection amounts to double-billing which is at the very least unethical.

However, we're talking about Cogent here. They aren't just demanding the right to send lots of packets to Verizon's customers. They're demanding the right to send lots of packets to Verizon's customers interfacing at the Verizon locations of their choice.

I'd have more sympathy for Verizon is they were offering to let Cogent connect at locations convenient to Verizon. But I have no sympathy for Cogent at all.

Re:Can confirm

[aztracker1]

I think it also depends on the device.. on my PC, I get a lot of buffering stops... but when I stream on my TV's built in client, no such event, and the quality is usually better too. Same internet connection.... I can only assume that different devices are hitting different servers from their PC/browser counterparts... or that the Silverlight infrastructure just sucks that much more.

Re:Can confirm

[Anonymous Coward]

This is a Cogent issue.

Netflix pays Cogent, not Verizon. The interconnects for peering between Cogent and Verizon are not free, that's why the peering agreements are based on 50/50 exchange so as to equalize the expenses between peers, Peering agreements are legal contracts with lots of terms and conditions. I'll take a guess that Cogent is in violation of the contract.

Cogent needs to spread around some of the piles of cash it's getting from Netflix and buy some new connections,

Re:Can confirm

[C18H27NO3]

I have Comcast myself.
What I find odd is that Netflix runs perfectly fine for me on both laptop and phone but Comcast's own xfinitytv streaming is a barrage of jitter and pauses.(Wired laptop, wireless phone)
It makes no sense.

Re:Can confirm

[Anonymous Coward]

"If Cogent is using Verizon for transit, yes, they should pay for that. If the peering is strictly to deliver content to Verizon's customers, that bandwidth is already being paid for by Verizon's customers."

Let's say you and your friend live in a region that Verizon broadband serves. You have a desire to do video conferences and other file sharing with your friend. You buy a broadband connection from Verizon and your friend doesn't. Is your friend entitled to a Verizon connection because you already paid for a broadband connection?

If you think this isn't a valid analogy, let's change the scenario a little bit. Let's say your friend leases space at a datacenter where Verizon also leases space. Your friend offers to connect to Verizon via 10 gigabit Ethernet and provide Verizon customers video service, but he insists on a settlement free agreement with Verizon where no one pays the other for the connection. Is Verizon required to oblige and give your friend a free connection to Verizon's network just because you want to connect to him and because you paid for a broadband connection? Maybe you and your friend would like Verizon to oblige, but that's not how the Internet market has ever worked. Your payment to Verizon does not pay for every other person wanting to connect to Verizon.

Settlement free connections are all based on mutually willing parties. Cogent has a free peering agreement with Verizon but only so long as the traffic ratios are the same or roughly the same (the industry standard is no more than 2:1 disparity). Cogent is probably well past 2:1 and Verizon could terminate the peering agreement but that would risk a dispute that Cogent is infamous for. A proper peering termination means that both parties would return to Internet transit to get to each other but the agitator in the dispute will deliberately continue sending their routes to the downed peering connection causing widespread outage. So the tactic that companies use to combat peering freeloaders who are squatting on a connection is to refuse to add more capacity and give away even more free bandwidth.

When content providers buy transit from Cogent at gutter prices, they know full well that they are getting the smallest transit capacity. Netflix knows what they are buying when they contract with Cogent to deliver their traffic. Other sites work better because they buy higher quality transit connections.

Re:Can confirm

[mysidia]

Why should Verizon expect to have an even in/out ratio? They sell the vast majority of their customers asymmetrical connections.

You only peer with another provider settlement-free, if the peering arrangement is mutually beneficial. ISPs such as Verizon are in the business of selling connectivity, not giving it away for free. If they are making settlement-free peerings where they don't benefit at least as much as the other party, then they are losing the revenue from selling connectivity, for less than the revenue would be worth in return.

Since Cogent is pushing much more data than they pull, then the peering relationship is disproportionately beneficial to Cogent.

Therefore, Cogent must pay more. It's really that simple.

Re:Can confirm

[PRMan]

All my Netflix and Olympic streaming in HD has been great on TW lately. Now if we can just keep Comcast away from my internet.

Re:Can confirm

[petermgreen]

Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said, I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections to cut costs.

AIUI cogent nowadays would be best described as a "wannabe tier 1", they are at the point where they no longer buy transit from anyone but it is widely suspected they rely on paid peering (kind of a middle ground between peering and transit) and they get into peering fights from time to time

but VZ has said, in effect, "you are not our 'peer'" and beyond a certain amount of peering bandwidth, you should start buying

And if customers understood what was going on and had access to a competitive markets of ISPs they would be saying to verizon "Sort out your little spat or we won't be your customers anymore".

Sadly many parts of the US are in the horrible situation of having their local monopoly vertically integrated with a teir 1 carrier. I believe some parts of europe have similar problems (here in the UK none of our local monopolies are vertically integrated with a teir 1).

Re:Can confirm

[petermgreen]

A proper peering termination means that both parties would return to Internet transit to get to each other

But neither verizon or cogent buy transit so if they depeer each other then there will simply be no routes.

Re:Can confirm

[mysidia]

I have been out of the Tier 1 ISP world for years - but at least historically Cogent was known as a bottom feeder of the industry. They charged dirt-cheap rates but ran a crappy network and skimped on their upstream connections.

Exactly. The Cogent rate for the same gigabits of data is probably about 70 to 80% lower than Verizon's rates. In this case... if Cogent can get unlimited free peering to Verizon's network, then it would make no sense for Netflix to pay the extra premium Verizon charges per gigabit of committed demand, for access to their network.

Cogent may host much of Netflix, but they are by no means a Tier 1. This may no longer be the case - like I said

If you read Cogent's brochures and marketing material on cogentco.com, you will see them refer to their service as a Tier 1 opcal IP network.

I believe in the firm opinion of most of the influential industry players, Cogent is not really a Tier1; they are a wannabe, almost-Tier1.

Re:Can confirm

[Anonymous Coward]

Let's say your friend leases space at a datacenter where Verizon also leases space. Your friend offers to connect to Verizon via 10 gigabit Ethernet and provide Verizon customers video service, but he insists on a settlement free agreement with Verizon where no one pays the other for the connection. Is Verizon required to oblige and give your friend a free connection to Verizon's network just because you want to connect to him and because you paid for a broadband connection? Maybe you and your friend would like Verizon to oblige, but that's not how the Internet market has ever worked. Your payment to Verizon does not pay for every other person wanting to connect to Verizon.

Verizon is not obliged to directly connect, but they are obliged to provide access from the customer to the video-service, assuming the video service has a connection via some other transit provider. And that transit provider is obliged to provide the 10gb/s of bandwidth to their customer and if the customer is constantly trying to transfer data to/from Verizon, then so be it. That transit provider will need to make sure that customer gets their 10gb of bandwidth. In the end, Verizon is going to get 10gb of data coming on their network, but it is up to Verizon to decide if it's worth directly peering instead of getting the data over another link. At some point, it's cheaper to go to the source than to get it indirectly.

Re:Can confirm

[Anonymous Coward]

Don't use wireless, especially with HD content. It is way too unreliable.

On a wired connection, I'm only seeing problems on some nights.

Re:Can confirm

[Anonymous Coward]

ditto, then I called out a Verizon tech who showed that there was no problem with my Verizon equipment or with my home network. "when you have removed all of the other possibilities then..."

So then I called Verizon back when we had slower speeds again and they said "run the Verizon approved speed test" which suddenly showed a speed increase from the test I had run just minutes before (same speed test, just not run with their servers). And showed that they could and would do something (but what?) when they chose. Or was it just coincidence? I really don't know, but when Netflix gives the message that I should just forget about a connection because my ISP speeds are too slow then there is some kind of problem.

as always streaming sucks, torrents rule

[Anonymous Coward]

so it takes an extra hour or to to get a movie, what do I care?

thanks to everyone paying for netflix and going to movies, if I wasn't poor I would totally join you in spending money on entertainment
(after medical bills, home improvements, saving for retirement, and entertainment I can't download for free)

Re:as always streaming sucks, torrents rule

[kthreadd]

Well, the fact that it's illegal is a bit of an obstacle.

Re:as always streaming sucks, torrents rule

[Anonymous Coward]

That didn't stop the NSA

Re:as always streaming sucks, torrents rule

[SuricouRaven]

It would be if copyright law was remotely enforceable online.

Re:as always streaming sucks, torrents rule

[Anonymous Coward]

no, downloading it is not illegal, sharing it with others is.

Network vs Content providers

[future assassin]

If companies provide network access they should not be be allowed to be a content provider. Too much conflict of interest and they can concentrate on properly managing and not OVERselling their network.

Re:Network vs Content providers

[dave562]

In your mind, where does Cogent fall on the network provider / content provider divide? They are providing network access, but they are also (as far as I know) providing the infrastructure for Netflix's CDN.

Re:Network vs Content providers

[thule]

Cogent likes to think of themselves as a pure bandwidth company. No frills bandwidth for a great price. No content, no VoIP, nothing. They have colocation data centers, but that came when they purchased a company for their network.

Re:Network vs Content providers

[Lawrence_Bird]

That isn't even the real point, though I understand where you are going with it. The real issue here is that Verizon, Comcast, Cablevision, etc all have agreed to provide their customers with X/Y data connections for a hefty monthly fee. They are refusing to do what it takes for those customers to be able to use what they have paid for. In effect, we are back to the early internet days of ISPs oversubscribing dialup lines except now it is oversubscribing routing equipment during peak hours.

Re:Network vs Content providers

[ducomputergeek]

We really missed the boat with not having local governments at the city/county level build the infrastructure. Where I grew up we had an electric coop. My Dad still lives there. Other than the main lines, it was all buried cables. Power outages were extreme rarities. His rates have been between $.08 - .11 per kilowatt hour. Hell a few years ago the rates went DOWN after the coop paid off some debt. In the city our rates are about .145 per kilowatt hour and increasing to .16 per kilowatt hour here soon. Last state I lived in started to force Utilities to open up their lines to competitors. My electric rates there went down from $0.18 per KW/hr to $.12 when that happened.

When I moved into the city, it was a major utility company, overhead wires, and if we got some ice or wind the power would go out.

Where my Dad lives, water & sewer is maintained by the city, and now contracts out with the county for water. His bill is about 1/3rd less than what our bill is closer to the city and our water is owned by a private utility.

Where it made sense we should have had cities or counties putting in fibre and allowed rural areas to form wireless coops. Then lease out service to whomever at a fixed cost per line.

Re:Network vs Content providers

[Anonymous Coward]

WT ever loving fuck is a wireless coop? Is that like electric fence for chickens or something?

Look man, do you think government makes up electricity by pulling it out of thier asses and then gives it away to the citiens out of their feelings of beneficience?

Man where can I get some of what you are smoking.

*This* is what government do when you give them power. Maybe not today or tomorrow, but if you give them power, this is what you get in the end.

Every. Fucking. Time. Word.

http://www.independent.co.uk/news/world/europe/ukraine-crisis-fresh-clashes-break-out-despite-president-viktor-yanukovychs-truce-with-opposition-9140428.html

But yea, computergeek says nah, this could never happen here. Sure.

Re:Network vs Content providers

[bluefoxlucid]

Studies show buried lines have more down time than overhead lines.

Re:Network vs Content providers

[Obfuscant]

We really missed the boat with not having local governments at the city/county level build the infrastructure.

No, we didn't, and this is not an example of that kind of problem anyway. It's an issue of a peering relationship that has changed from "peer" to a heavy bias towards traffic going one way. In the early days of the net peering companies decided not to try to charge each other for packets based on the idea that things would balance out. It was also harder to track that kind of stuff. It would cost more for accounting than any imbalance would cost. That's not true anymore since there are concentrated streaming video sources pushing large volumes of uncacheable content out.

Where I grew up we had an electric coop.

Where I grew up there was a coop. It's not a city or government operation, it was a coop. Customer owned. "Coop" is short for "cooperative", as in the customers cooperatively operating the company.

Re:Network vs Content providers

[Pinhedd]

The congestion isn't on the customer-facing side of the ISP's network, it's at the peering exchange with Netflix's carriers. They make no promise about that.

Re:Network vs Content providers

[suutar]

I would put Cogent on the network provider side. They store bits, they shovel bits, they don't care whether those bits encode Star Trek or Futurama or Dale and Tucker Versus Evil. (Unless they have a stake in Paramount that I haven't heard of :)

Re:Network vs Content providers

[tlhIngan]

Technically, what's happening is peering agreements.

Verizon and Cogent agree to carry each other's traffic. If the flows are roughly equal, then they usually come to a "free" agreement - Verizon carriest traffic from Cogent and Cogent carries Verizon's traffic for free.

It's when flows are unequal that's when the fights happen - basically the one getting more traffic from the other starts demanding money because they're sharing more of the burden of the traffic.

So what happened here is Cogent is passing more traffic to Verizon than Verizon is passing to Cogent. This usually results in the formerly free peering arrangement to become a paid one, and Verizon starts demanding money for the unequal flow.

It turns out the one demanding party has a new tool in the shed - stop upgrading the ports so now Cogent's traffic is maxing out the port and getting dropped packets.

Sometimes the one having more traffic does nasty tricks as well - including rerouting their traffic through a more expensive link to force port upgrades (i.e., Cogent sees Verizon pays more if traffic goes through another port, so they route all traffic through that port). Or they move traffic between ports forcing port upgrades on all ports.

Re:Network vs Content providers

[TechyImmigrant]

>It's when flows are unequal that's when the fights happen - basically the one getting more traffic from the other starts demanding money because they're sharing more of the burden of the traffic.

That makes no sense. Both sides are sharing the burden, regardless of the direction of the traffic. You still have the packets on your wires. It doesn't matter if they're going left or right.

Re: Network vs Content providers

[guruevi]

I think he was talking about coops and allowing local governments to control access to the lines. You know, like telephone and electricity. Currently the content providers/ISP's own the lines, the terminal end and the peering end and has successfully litigated against both government and coop business to take control over the tax funded lines.

Re:Network vs Content providers

[jeffmflanagan]

Those studies are probably misleading. It takes longer to repair an underground line, but it takes an idiot to sever it, when above-ground lines are regularly severed by weather events.

Re:Network vs Content providers

[jeffmflanagan]

>That's not true anymore since there are concentrated streaming video sources pushing large volumes of uncacheable content out.

No there aren't. This content is being requested by the ISPs customers, not pushed on them. Either the ISP can provide their customers with an acceptable service, or the local municipalities should step in and replace the failed ISPs.

Re:Network vs Content providers

[sjames]

Since they are selling Internet access and Netflix is in the Internet, they most certainly are promising that. The congestion is at Verizon's port so it is their responsability.

Re:Network vs Content providers

[sjames]

So the packets Cogent is sending into Verizon's network are not wanted by anyone who has paid Verizon to get them?

Re:Network vs Content providers

[Anonymous Coward]

Citation needed.

if you are going to say studies show, or studies prove, or studies suggest, or any sort of reference to a study. Provide the study you are referencing otherwise you are no better then Fox news

Re:Network vs Content providers

[Spazmania]

Cogent is a network provider.

The divide is based on the ownership of content or the rights to distribute that content. Netflix owns rights to distribute the content. Cogent does not.

The OP was complaining about Verizon. Verizon's cable TV service is a content provider. Verizon owns the rights to distribute that content. Verizon is also a network provider, delivering Internet access service. OP implies that Verizon's conflict of interest leads them to intentionally impair Netflix's competing content service.

Re:Network vs Content providers

[Anonymous Coward]

Looking at other comments, it looks like the reasoning is that the side sending the data will likely have the content either near the edge of their network already, or already on high speed links (since the content is stored at known locations). On the flip side, the side receiving the data is responsible for sending that data to any point of their network without knowing in advance where that will be, and may very well need to traverse many of their own links to accomplish the task, so receiving data is more of a burden than sending data.

That said, if I'm paying for my high speed internet from my ISP, I would like to know that they're making a best effort to make sure that I can get the most common content I want at the speed I'm paying for. After all, the receiving side is the side that requested the data in the first place.

Re:Network vs Content providers

[Kjella]

Since they are selling Internet access and Netflix is in the Internet, they most certainly are promising that. The congestion is at Verizon's port so it is their responsability.

By that logic I could set up an ISP charging $1000/MB for peering and your ISP would be forced to pay it since "it's on the Internet". It's a little bit more complicated than that, behind the scenes all the ISPs and backbone providers are negotiating how to hook themselves together. You can't demand that there be a peering point any particular place with any particular speed, it takes two to make a contract. It takes two to adjust a contract. You can't just hold one over a barrel and say "sign what that other guy is offering, no matter what".

Re:Network vs Content providers

[Lawrence_Bird]

The ISP is responsible for his peering relationships - that is what makes the ISP a part of the internet. You seem to be saying that the sports team that sells you tickets to a game has no obligation to make sure that another team shows up to compete. They sold you tickets, gave you a seat in a stadium and their own team showed up. What, you wanted to see a game? Perhaps not the best analogy but you get the drift.

Re: Network vs Content providers

[x0ra]

And you end up with regulatory capture as seen in the US with the FCC, and in Canada with the CRTC, where conflict of interest is king.

Re:Network vs Content providers

[Pinhedd]

No they are not. They are promising a certain link speed between the customer and the point where the customer's traffic leaves the ISPs network; if they don't deliver on that, that's a different matter entirely. They are not promising a certain level of access to services outside of the ISPs network. When external services are aggregated the marketed rate should be attainable, but aggregation involves making certain assumptions about traffic, one of which is that traffic is more or less evenly distributed across the interconnects. If thousands of customers all try to draw traffic across the same interconnect, that interconnect gets bogged down. The rest of the network is fine, and the customer should still be able to obtain whatever their normal data rate is, but only by aggregating traffic appropriately.

Re:Network vs Content providers

[sjames]

So you're claiming that they could just disconnect from the internet and provide connectivity only to their own servers and all is fine and dandy?

Of could it be that they're actually selling internet access?

Re:Network vs Content providers

[amorsen]

So you are saying that Cogent should have unlimited power in peering negotiations? Nice.

Re:Network vs Content providers

[sjames]

You're arguing my point for me! Verizon is the one that wants to start charging.

Your analogy doesn't hold because it wouldn't be my ISP holding up the works, it would be the other guy. We all understand that they can only be responsible for their own actions.

Re:Network vs Content providers

[sjames]

No, I would find it equally outrageous if Cogent started demanding a fee from Verizon.

Re:Network vs Content providers

[amorsen]

It is not fair for Cogent to just dump the packets at a peering far away from the destination. Peering is not just about zero cost.

Notice how it is always Cogent getting into these fights. Almost everyone else plays nicely with the other children.

Re:Network vs Content providers

[schnell]

That makes no sense. Both sides are sharing the burden, regardless of the direction of the traffic. You still have the packets on your wires. It doesn't matter if they're going left or right.

It very much does matter if you're hauling the bits from a couple data centers out to the nearest peering point, or if you're hauling them all over the US or the globe to deliver them to users in BFE, West Virginia. Hence the tension over who should pay whom.

Re:Network vs Content providers

[zippthorne]

That sounds pretty awesome. Their website doesn't look like they offer residential service, sadly. It occurs to me that my largest use of internet by volume is probably Netflix, why not get service from the same company....

Re:Network vs Content providers

[schnell]

The ISP is responsible for his peering relationships - that is what makes the ISP a part of the internet.

To indulge in a bit of reductio ad absurdam but, hey, why not:

WallaWallaNet: Hello, Verizon?
Verizon: Yes, hello?
WallaWallaNet: Hi, I'm an ISP in Walla Walla Washington, and I'd like access to your subscribers.
Verizon: Super! You can buy a T3 for $1k/month for the local loop and $7k/month for Internet transit.
WallaWallaNet: Ooh, here's the problem. I host "www.awesomepixofdogslickingthemselves.com." Your subscribers want the whole Internet - including all the pictures of dogs licking themselves that they could ever want - so I think it's incumbent upon you to peer with me for only the cost of my local loop at the nearest available peering point to me. I mean, www.awesomepixofdogslickingthemselves.com is worth enough to your subscribers to forego that money you were going to charge me for transit, right?
Verizon: Excellent point! Please mention the vital importance of www.awesomepixofdogslickingthemselves.com when you call every other major ISP on the planet and ask them for peering! I just need one more piece of information from you and we can get this all set up.
WallaWallaNet: Sure, what is it?
Verizon: What color is the sky on the planet where you live?

Re:Network vs Content providers

[zippthorne]

But how do you define "equal"? It's not just a matter of in/out - a netflix stream is a great example. There wouldn't be a stream if the individual customer didn't request it, and clearly Netflix is a driver for faster internet connections. But Netflix wouldn't have a business without delivering those streams, so they clearly have value to both entities. How do you value the traffic for the transport?

I assume the ISPs have generally figured this out. I'd think maybe railroads would be a good model - charge by distance traveled rather than balance of bits.

Re:Network vs Content providers

[Pinhedd]

Selling internet access with a particular level of general service, and selling access to a particular destination with a guaranteed or dedicated level of service are two different things. Consumers deal with the former, and get aggregate access with little to no guaranteed quality of service to any particular endpoint. Business consumers deal with the latter and pay tremendous amounts of money to get the level of service that they need.

The Internet is not just some big magic box in which traffic goes into and comes out of. Ted Steven's "series of tubes" comparison was heavily mocked by detractors, but it is a shockingly accurate comparison. ISPs provide access to the internet through a large number of backbone networks. Routing traffic from A to B and from B back to A may traverse several different networks with traffic changing hands at various exchange locations, typically located in big cities. Disconnect one particular peering arrangement and data that normally traverses that edge will simply find its way to its destination through another route, but this route will usually be less efficient to most consumers.

Right now Verizon peers directly with Cogent all across the country and that's how Netflix traffic is delivered to Verizon customers. Peering agreements are usually constructed to be fair to both parties and rarely involve actual cash payments and that is indeed the case between Verizon and Cogent. It's rare for a peering agreement to be perfectly balanced, but the growth of Netflix means that Verizon is taking far more traffic from Cogent than they are sending back to Cogent and as such are incurring the cost of accepting that uneven amount of bandwidth (which is a huge amount) while as the sender Cogent is keeping the revenue that it gets from Netflix.

Right now, Netflix accounts for almost a third of all downstream Internet traffic, but less than 5% of all upstream traffic. This is a pretty big discrepancy, wider than the usual uses which, even when they do place a large load on the network, tend to be less time sensitive and burstier.

Re:Network vs Content providers

[sjames]

My understanding is that this is over traffic ratio, not route distance. Route distance would be a reasonable point of contention if that were the problem, but since this is an existing peering, that wouls have been resolved amicably long ago.

BTW, hot potato routing is easily enough fixed without head butting by controlling what routes are announced through the peering ports.

Re:Network vs Content providers

[sjames]

I am well aware of how the net works. I have answered this elsewhere, bbut would it make Verizon happy if Netflix changed the client to echo all recieved data back? Then the peering would be well balanced. Absent that, every byte Cogent is transmitting is a byte a paying Verizon customer has solicited. Should Cogent charge Verizon a fee for all those requests coming from their network?

Re:Network vs Content providers

[Pinhedd]

Don't be facetious. Echoing the traffic would make neither Verizon nor Cogent, nor Netflix's customers happy. Verizon is under no obligation to receive every byte that a paying customer has solicited at the customer's marketed rate unless that paying customer is a business customer that has a service level agreement. The peering agreement between Verizon and Cogent is purely voluntary and Netflix is free to contract with another ISP that has better interconnects to Verizon. The requests coming from Verizon's network are part of the traffic that Verizon sends to Cogent.

Re:Network vs Content providers

[sjames]

It's not facetious, it's reductio ad absurdum. If it really is all about balanced traffic, then my solution would be perfect for everybody (It probably wouldn't bother Cogent or Netflix one bit, network connections are full duplex and symmetrical except for the last mile for residential service). That fact that it is, in fact, absurd shows that the premise it is based upon is likewise absurd.

Perhaps Cogent should contract w/ netflix to respond with a still image saying Verizon has decided it won't let you see Netflix. Perhaps that would make Verizon happy?Or they could only respond to requests from Verizon customers occasionally? Or when packets start dropping, rather than falling back to a low-res stream, put up a still image indicating that Verizon's network is oversold?

Would it be reasonable for me to request a ton of potatoes from the grocery store and then insist that they need to pay me a shipping fee for accepting them?

Re:Network vs Content providers

[Anonymous Coward]

It's a pretty good analogy, actually.

Re:Network vs Content providers

[Anonymous Coward]

I propose that there are more idiots than weather events. At least, that is my experience...

Re:Network vs Content providers

[Obfuscant]

No there aren't. This content is being requested by the ISPs customers, not pushed on them.

I didn't say the content was pushed ONTO people, it's content being pushed out the network connection. Yes, indeed, there are large volumes of network traffic being pushed out by sources like netflix that didn't exist before, and that large volume of traffic is creating unbalanced peering arrangements. It doesn't matter to the traffic balance that some ISP customer has request that large volume of traffic, it's still highly asymmetric and breaks the longstanding assumptions of peering agreements.

Either the ISP can provide their customers with an acceptable service, or the local municipalities should step in and replace the failed ISPs.

And this is completely ignoring that this problem is not being caused by the local ISPs, it is a problem IN THE PEERING at the higher levels. The ISP has no problem carrying the packets, the problem is that those packets cannot get onto the ISP network to start with. The local municipalities will NEVER be able to solve that problem because they will never be in a position to peer with anyone. They won't be large enough.

Re:Network vs Content providers

[TechyImmigrant]

In this case it is the sender sending from video streaming services (I.E. Netflix), maybe locally cached, and the receiver passing the traffic onto their one local endpoints (their customers).

A company that wasn't interested in preferring their own on-demand TV service to an internet based alternative would do the sensible thing and work with Netflix to cache locally.

Re:Network vs Content providers

[bluefoxlucid]

There's nothing misleading about spending 50% more raw hours without power per year. My power might go out once or twice a year, at the most. It went out when we had a hurricane, and that downed the underground lines too by flooding the subway tunnel they were in.

Netflix should know better

[Spazmania]

Cogent has a long history of instigating peering disputes with other networks. Normally I'd complain about Verizon's behavior but this is Cogent we're talking about. They have -zero- credibility.

Re:Netflix should know better

[ffsnjb]

Agreed, every time I look at any various looking glass to see why something is broken on the internet, it's Cogent dropping the ball... Remember this?

Every time I have to troubleshoot a broken internet problem, and it goes up to Tier 1 land, check the looking glass... It's Cogent being stupid. As soon as this story hit... check. Sure enough, Cogent-Level3 has ~3% packet loss the last 24 hours, with stupidly high latency. Screenshot .

Re:Netflix should know better

[Shinobi]

Yeah, Cogent really is trying every dirty trick they can to go past any contract limits etc and freeload, and then they cry loudly to the media when they get told to stick to the contract.

Re:Netflix should know better

[Bender Unit 22]

Bingo.
It shouldn't come as a surprise for them.
Or perhaps they should hire someone who has been in the business for some time. I feel like it is almost common knowledge. :)

Dumb design

[Anonymous Coward]

Any streaming protocol that doesn't adapt to available bandwith and allow for the odd dropped packet is broken.

Re:Dumb design

[jspoon]

Netflix does adapt to low bandwidth by the obvious contingency of showing me terrible low bitrate streams. I'm on FIOS and used to get consistent beautiful HD so I feel entitled to complain.

Re:Dumb design

[jeffmflanagan]

Dumb AC, but then if you didn't already realize that you were being dumb, you wouldn't have posted as an AC.

Chromecast Vs. Roku

[TrippTDF]

I have a 1st gen Roku, and a Chromecast. When I stream Netflix with the Roku, I seem to top out at 2, maybe three quality bars. While there's no on-screen metric for the same stream in Chromecast, the picture is noticeably better. Perhaps the Chromecast is getting a higher-quality, lower-bandwidth stream, or there's some sort of throttling based on the streaming device going on?

Re:Chromecast Vs. Roku

[sixsixtysix]

or there's some sort of throttling based on the streaming device going on?

This wouldn't surprise me in the slightest. Sometimes, they'll actively block certain devices. Just look at hbo go. I can stream on my computer or iphone app over wi-fi (or cellular), but, and this is the most retarded thing ever, it will not work on my roku because comcast and hbo go don't have a deal (which I don't get why they'd even have too, considering the previous 2 services work fine and both require linking my comcast and hbo accounts).

Re:Chromecast Vs. Roku

[Anonymous Coward]

It might be the compression type used. I remember reading that Netflix uses mostly uncompressed streams to TVs and TV like devices since they can't do heavy decoding. The Roku could be using such a stream requiring more bandwidth. If the Chromecast is able to decode at a higher compression it could be the answer as it would take less bandwidth.

Re:Chromecast Vs. Roku

[ravenscar]

Upgrade your Roku. I had the same issue. Upgrading to the newest Roku made a world of difference. I'm not sure why, but I would guess Netflix has updated their streaming protocol and that making the most of what it has to offer requires more hardware than the original Roku has on tap. If nothing else, you get a much nicer Netflix interface on the new Roku.

Side note - my solution worked for me, but might not for you. Fortunately, Rokus are fairly common so perhaps you could have a friend or neighber drop by with theirs so you could see if there is a difference.

Re:Chromecast Vs. Roku

[Anonymous Coward]

In theory the Chromecast is running WebM codecs (http://www.webmproject.org/). I wonder if one could compile and install them on a Roku box to see a similar behavior. I noticed when I set them up on my computer my Netflix streams jump to higher quality much fast and they stay there. Before using whatever the default was for my browser caused it to jump quality levels during playback. Only reason I even installed the codecs was to try them out in my own "codec pack" since I freaking hate all the existing ones out there (they are either out of date or use shitty splitters that don't provide proper chapter navigation).

Re:Chromecast Vs. Roku

[evilviper]

I remember reading that Netflix uses mostly uncompressed streams to TVs and TV like devices

There's no such thing as uncompressed video streaming. Uncompressed video is astronomically large, and impossible to stream over any internet connection you've ever seen.

https://en.wikipedia.org/wiki/...

1080p@24fps is about 800Mbit/sec., something even your local gigabit LAN would struggle to sustain.

What you mean is using older, less efficient, and less computationally complex (but still lossy and still highly compressed) video codecs.

Net Neutrality laws?

[thule]

I've been saying this for ages! Even mentioned this here on slashdot. Peering is peering. They are not degrading performance by configuration, they just let the link get congested. How do any of the proposed net neutrality laws address this issue? Answer is, they don't. To me that means that Net Neutrality laws are about something different than neutrality. More likely with government regulation, it becomes Net Control. With that, increased stiffing and limiting reaction to market dynamics, not improving it.

Re:Net Neutrality laws?

[dave562]

You are mixing apples and oranges.

Peering agreements have been the same forever. As long as there is nearly a 1:1 ratio between the providers, everything is fine. The issue comes up when one side is using more bandwidth than they are giving in return.

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs. What they should do is run the numbers and figure out what costs more; "overage" charges from Cogent, or eating the cost of paying to deploy their CDN hardware and network links to the other Tier1 ISPs.

Are you suggesting that net neutrality should address situations like this? Are you saying that it is a good idea to have the government force a business to eat the cost of supporting someone else's business model? To me, that sounds like a big fat subsidy for Netflix at the expense of everyone else.

Re:Net Neutrality laws?

[Lord Lemur]

Hasn't Netflix been offering to put their equipment inside of any ISP that will have them, since forever?

Re:Net Neutrality laws?

[Anrego]

Yeah, but it's not like Netflix is using a free service. They are paying for that bandwidth. I assume they are paying quite a bit. More importantly, someone is selling it to them.

Re:Net Neutrality laws?

[heypete]

I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

All the peering details are here. In short: they don't charge anything. They offer direct interconnects to Netflix's CDN for free, free peering at major internet exchange points, and free, Netflix-managed hardware caches to ISPs to avoid duplicate network traffic (the vast majority of traffic stays within the ISPs internal network). For the hardware caches the ISP needs only provide power and network connectivity.

There's really no reason for ISPs to wrangle with Netflix -- there's plenty of options to avoid congestion.

Re:Net Neutrality laws?

[H3lldr0p]

Except there has never been anything close to a 1:1 relationship. There couldn't have been because even since the IDSN days the up/down ratio of what we could get was always in favor of the down. So there's never been enough traffic coming from the ISPs to even approach parity.

In fact, it's been the stupid ISPs have been using this as a club against the other players. It's of their own making, and now they're choking on it. In fact, Nexflix is even willing to give ISPs servers to take the congestion away from that part of the network, something that's been consistently rejected. The ISPs are the problem. In specific their greedy, overly Wall Street focused, stupid management is the problem.

So, no. They don't get any sympathy from me in this. They aren't being held accountable by their customers, they aren't being held accountable by what little law is in place, and they certainly aren't being held accountable by the market. If they were, they'd be out of business by now.

Re:Net Neutrality laws?

[marka63]

Requiring 1:1 traffic flows is a stupid way to organise peering agreements. It doesn't matter if it is 1000:1 as long as both sides are getting benefit from the peering. If you have content you want to deliver and eyeballs that want content you peer.

As for transit providers and eyeball networks. The transit providers are being paid to deliver content and the eyeball networks are being paid to deliver content. Failing to upgrade interconnects is taking monies under false pretences and that has another name "fraud".

Re:Net Neutrality laws?

[dave562]

Except there has never been anything close to a 1:1 relationship. There couldn't have been because even since the IDSN days the up/down ratio of what we could get was always in favor of the down. So there's never been enough traffic coming from the ISPs to even approach parity.

Do you have any data to support that? On the last mile connections, there is an imbalance between up and down. On residential versus commercial, there is definitely an imbalance between up and down. But at the highest levels, the content farms and co-located customers, I thought it was pretty even.

I am pulling these examples out of my butt in an attempt to provide some context. I have no idea who has hosting agreements with who. But if you have NBA.com on ISP1 and NFL.com on ISP2, and MLB.com on ISP3, the traffic between the two would be roughly equal. Again, probably a bad example, given the different sizes of the audiences for each, the fact that the seasons do not overlap, etc. The point that I am trying to make is that most of the major ISPs all host high traffic farms, those farms are not centralized with one specific ISP, so far the most part, there is roughly 1:1 between the various providers.

Re:Net Neutrality laws?

[kqs]

Very logical. In related news, many things can make you ill, not just toxic waste. Therefore, any laws which discourage dumping toxic waste on your property are not about health, and are probably just about governmental control of companies.

Peering limits are certainly used as a rough form anti-net-neutrality, but they're not ideal; they have the pinpoint accuracy of a sawed-off shotgun at 100 yards and they are very obvious. The proposed laws tend to target the subtler, better directed forms. Don't let the perfect be the enemy of the good.

Re:Net Neutrality laws?

[MozeeToby]

Netflix is breaking the long standing status quo.

Here's what you don't understand: Cogent (on behalf of Netlix) wants to send packets to the ISP. The ISP (on behalf of their customers!) wants to receive those packets. No one is getting shafted. Both sides of the agreement are already getting what they should be wanting. Now the ISPs are coming along and saying that the peering agreements are only about upstream bandwidth? Why? The ISP's customers are the ones who requested the packets that Cogent is trying to deliver to them.

Re:Net Neutrality laws?

[firex726]

Also Netflix has made the offer available to setup CDN servers in the local providers facilities to ease the BW usage on their peers. Google uses this; but Verizon, Time Warner, and Comcast do not, who then whine about the BW use.

Re:Net Neutrality laws?

[sixsixtysix]

Isn't someone always using ~30% of ALL internet traffic? If it's not netflix, it's torrents or youtube. These 3 must use 99.999999% of all internet traffic from what people post here.

Re:Net Neutrality laws?

[compro01]

There's really no reason for ISPs to wrangle with Netflix

Verizon has two reasons to wrangle with Netflix.

Re:Net Neutrality laws?

[Obfuscant]

Except there has never been anything close to a 1:1 relationship. There couldn't have been because even since the IDSN days the up/down ratio of what we could get was always in favor of the down.

The peering relationship balance has nothing to do with your ADSL or cable modem data rates. The peering relationships are at a level where the assumption (a reasonable one for a very long time) was that the content providers and consumers were distributed around the net pretty evenly, so that a consumer on A's side of the peering connection talking to a provider on B's would be balanced out by a provider on A and a consumer on B.

The 'A' in ADSL is because the assumption for residential customers (and often written into the service agreements) is that they are mostly consumers and almost never providers. And many businesses are that way, too. Had the net grown up with everyone being a content provider and a consumer both, asymmetric last-mile service would not be the norm.

They aren't being held accountable by their customers,

Your statements would be more credible if you told us which ISPs you have cancelled service with for this issue. That's how the customers and market would hold them accountable. If what you mean is "other customers should hold them accountable, I want to keep my net connection...", well ...

Re:Net Neutrality laws?

[Anonymous Coward]

Indeed. Netflix should say "O.K, we'll stop peering and you can consume the data via. your downstream transit. How do you think that'll work out for you?" and then see what happens.

Re:Net Neutrality laws?

[evilviper]

Yeah, but it's not like Netflix is using a free service. They are paying for that bandwidth. I assume they are paying quite a bit. More importantly, someone is selling it to them.

Right, Cogent is the one who screwed-up here, dumping an unbalanced load of (what happens to be Netflix) traffic on (say: Verizon), and Cogent neither being willing to accept other traffic to balance the peering, nor paying (Verizon) for the imbalance.

If Netflix was distributing its' traffic over several different Tier-1 providers, they wouldn't be having as big of a problem with ISPs. And Netflix is being dishonest in promoting their "free" OpenConnect devices to ISPs to reduce their network traffic. Netflix knows full well that ISPs hosting the device in their data centers costs them money, and is something that other service (like Akami) have to PAY FOR the privilege of.

I'm not terribly critical of Netflix, but they are a slightly bad-actor in all of this, and are always positioning themselves as the innocent and persecuted victim.

Re:Net Neutrality laws?

[evilviper]

Netflix has made the offer available to setup CDN servers in the local providers facilities to ease the BW usage on their peers. Google uses this; but Verizon, Time Warner, and Comcast do not, who then whine about the BW use.

I'm sure ISPs are chomping at the bit for the privileged of giving Netflix free electricity and rack space in their ultra-expensive and valuable data centers... Something that other CDNs (like Akami) PAY FOR, and would object to Netflix getting for free.

Re:Net Neutrality laws?

[evilviper]

They offer direct interconnects to Netflix's CDN for free, free peering at major internet exchange points, and free, Netflix-managed hardware caches to ISPs to avoid duplicate network traffic

It's "free" only in 1984 new-speak...

ISP: "You mean I get to carry your traffic, perhaps much of it actually going to customers of other ISPs, on my network backbone for you, for FREE? And if I don't like that, I can provide you electricity and rack space in my ultra-expensive datacenter, where other CDNs pay to get space, and you don't give me dime for it? What a deal!"

Those "free" services Netflix is giving out to ISPs, are actually expensive ISP services that Netflix is hoping to TAKE, for FREEEEEEE!

Re:Net Neutrality laws?

[Anonymous Coward]

It's not a Netflix or Verizon issue. it's a Cogent problem.

Re:Net Neutrality laws?

[petermgreen]

Verizon and cogent are both transit free networks, if they depeer from each other then single homed customers of the two networks will simply not be able to communicate with each other.

Re:Net Neutrality laws?

[petermgreen]

I disagee, the root of the problem is that a teir 1 provider is vertically integrated with a last mile provider that has monopoly or near-monopoly power in some areas.

If the last mile provider was not vertically integrated with a teir 1 it would be in their interests to set up good peering with large conent providers directly (as happens here in the UK) to avoid paying upstream transit bills but because they are verticially integrated with a teir 1 and don't have to buy transit from anyone it makes sense for them to play hardball on peering. Think about it? why are cogent involved here in the first place? Most likely because Verizon refused to peer with netflix.

Re:Net Neutrality laws?

[MikeBabcock]

How long exactly have you been working with Tier-1 Internet routing and billing exactly?

Never?

That's what I thought, cause you have no idea what you're talking about.

Re:Net Neutrality laws?

[MikeBabcock]

Yes; the problem is that many local ISPs also happen to be the local cable provider and have a vested interest in not helping you cut cords.

Re:Net Neutrality laws?

[MikeBabcock]

Sadly those ISPs have this attitude only because of a lack of competition.

A smart small ISP would offer to host Netflix content and put it in their advertising that while they only have x Mbit service, they guarantee an incredible Netflix experience in full HD (or even 4k).

I know many people who basically only use the Internet for Youtube and Netflix. ISPs wanting to offer that level of service could do so at very low costs with a locally-hosted CDN server.

Re:Net Neutrality laws?

[MikeBabcock]

Carrying Netflix traffic isn't done for free, ever. That's just FUD spouted by ISPs trying to get free money. They tried doing it to Google first. Its either traffic for your clients, so you're already getting paid for delivering it, or its through a peering arrangement that you already find beneficial (or it wouldn't exist).

Re:Net Neutrality laws?

[evilviper]

Its either traffic for your clients,

Not at all. It could easily be traffic being dumped onto your backbone by a badly behaving peer. There is no means to prevent it.

or its through a peering arrangement that you already find beneficial (or it wouldn't exist).

Not "find", but "found". As in, past-tense.

Besides, these are just circular arguments, anyway. You're continually assuming some unlisted benefit, that may or very well *MAY NOT* be there. There are many cases in which what you said doesn't remotely hold true.

Re:Net Neutrality laws?

[thule]

You are mixing apples and oranges.

Peering agreements have been the same forever. As long as there is nearly a 1:1 ratio between the providers, everything is fine. The issue comes up when one side is using more bandwidth than they are giving in return.

Not entirely true. I remember reading an article years and years ago that Yahoo was only paying for half of their total bandwidth usage. At the time Yahoo was generating a lot of traffic. It helped Yahoo and the larger ISP's to bypass their expensive transit links, bypass the backbone, and connect eyeballs to content directly.

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

It is a little bit more complicated than this. Netflix uses Cogent. Cogent has pissed off other backbone providers over the years. Netflix is suffering with Verizon because of the relationship with Cogent. Netflix should see if Verizon would be interested in peering with them "directly".

What they should do is run the numbers and figure out what costs more; "overage" charges from Cogent, or eating the cost of paying to deploy their CDN hardware and network links to the other Tier1 ISPs.

Are you suggesting that net neutrality should address situations like this? Are you saying that it is a good idea to have the government force a business to eat the cost of supporting someone else's business model? To me, that sounds like a big fat subsidy for Netflix at the expense of everyone else.

I DO NOT want the government to have any say in this stuff. I would rather the market figure out the details. Yeah, there might be bumps along the road, but I would rather have that than the long arm of government regulation causing stagnation.

Natural monpolies

[bob_super]

Water, then gas, then electricity, phone, and now internet backbone are Natural Monopolies.
Private companies should not operate in Natural Monopolies because they just add overhead cost as they operate for profit.
Obviously, with the American public convinced of the lie that government is just waste, and that private companies magically reduce waste (at the bottom, for sure), we all keep on shelling massive amounts of cash for basic services. Cash that would better be invested in keeping the US moving forward, rather than lining the pockets of terrible profiteers like my 401k...

Re:Natural monpolies

[Anonymous Coward]

"Obviously, with the American public convinced of the lie that government is just waste, and that private companies magically reduce waste (at the bottom, for sure), we all keep on shelling massive amounts of cash for basic services. Cash that would better be invested in keeping the US moving forward, rather than lining the pockets of terrible profiteers like my 401k..."

lolwut?

'the lie that government is just waste'

OMFG.

http://blog.heritage.org/2014/02/06/billions-wasted-flawed-obamacare-exchanges/

"The federal taxpayers have spent over $5 billion to build Obamacare’s exchanges. Unfortunately for taxpayers, these investments have not delivered a good return and are ongoing.

Despite the muddled launch of many exchanges, the federal government is still doling out grant money, giving out over $200 million to nine states last month. Fourteen states and the District of Columbia are running their own Obamacare exchanges and they all received federal grant money to help build them—a total of over $4 billion. California alone received $1 billion to set up its exchange."

Billions bob_super, billions with a B, and do remember, this is not even healthCARE, it's an internet shopping cart, which isn't even FUCKING NEEDED if they just say "hey citizen buy health insurance that is Obamacare compliant and claim any applicable subsidy on your tax return" but no fuck you they WANT to build the exchange becuase Obama's wife has buddies that need the graft and then; the shit DONT even WORK *and* your PII is sent automatically to hackerz in Ukraine, *and* ain't no one seeing no $2500 reduction in health insurance costs (hahaaahahahahaaaa, hahahahaaaa, bwaheahaha hahahahah FUCK YOU) *and* the fucking doctors listed as in network *aint* in the fucking network.

But yea, you must be right and government is always more efficient, super bob. We'll go with that.

Re:Natural monpolies

[Anonymous Coward]

Well, no, none of those are natural monopolies. Those are state chosen monopolies because if the government did not force citizens to accept the utilities being dug under their land, some people would refuse or only permit the digging if it came with some exorbitant personal payment. This way, only the government gets the exorbitant payment, but everyone gets access to the utilities at some monopoly-declared price points.
The governments involved only have incentive to allow another provider of any utility if they are convinced that the kickbacks from two companies will actually exceed the kickbacks from one company that doesn't want competition.

Re:Natural monpolies

[Anonymous Coward]

Water, then gas, then electricity, phone, and now internet backbone are Natural Monopolies.

You forgot roads.

Re:Natural monpolies

[bluefoxlucid]

Huge massive thing with 8 billion functions screws up 1 function. See? Huge massive thing is all waste, no efficiency.

Re:Natural monpolies

[Anonymous Coward]

WT ever loving F are you talking about?

Re:Natural monpolies

[sjames]

Sure, there is waste in the government and we all hear about it because it's our right to know. Now, how much waste is in privete industry? Ever filled out a form that you know very well will sit unread in a filing cabinet until it composts? That's waste. Ever gotten a credit card offer that you don't even qualify for? What about those crazy rules that turn a simple matter into a debacle?

Everything on up to a gulfstream for the CEO, all waste.

But when you look at actual costs for services provided, the government provided service often works out cheaper. That's why private corporations object so strenuously to government providing services even where competition is permitted.

Re:Natural monpolies

[Antonovich]

Damn, why do my points always expire just before I see a comment that really needs modding up?

Sadly if this wasn't an oligopoly, it wouldn't be

[Anonymous Coward]

The sad fact is, if Verizon wanted to improve customer satisfaction they could easily join Netflix's Open Connect program: https://signup.netflix.com/openconnect
Instead they decide not to and customers are screwed, because we don't have an alternative provider available. Long term solution is for Congress to get off it's ass
and mandate a national fiber installation that will be leased out to any network provider that's willing to run an ISP.

"Lopsided"?

[K. S. Kyosuke]

Cogent has an agreement with Verizon to exchange traffic — which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement.

I'm not quite sure I understand what's "lopsided" about that. It still flows through both of the two networks, and the companies of the receiving end are still the ones cashing in from end users, and they still very much like their users paying extra money for extra data transfers, don't they?

Re:"Lopsided"?

[silas_moeckel]

It's lopsided, Verizon is more and more and eyeball network where people "consume" bandwidth they got there by pricing themselves out of most of the transit market. That puts them in a somewhat unique position that there are a tier 1 peer but should not be. They are trying to leverage the you need access to the eyeballs more then we need access to your content. Regulating Tier 1's is not easy they never really have been anything but self regulated but as they expand/develop they have a conflict of interest. Remove the conflict require the entities to only sell transit without a discount to anybody.

Re:"Lopsided"?

[Anonymous Coward]

> I'm not quite sure I understand what's "lopsided"

Huh? Cogent is dumping traffic to Verizon then demanding Verizon pay for the expensive long distance transit. In many cases, they're only paying for the intra-data center traffic then expecting Verizon to pay for the hundreds or thousands of miles to the customer. They should be buying transit rather than dishonestly demanding peering.

Re:"Lopsided"?

[Cramer]

This isn't entirely a peering or transit dispute. At the end of the day, the traffic going into verizon's network BELONGS THERE . It's traffic destined to verizon customers, that those customers specifically asked for. As a residential, consumer, broadband ISP, it's your job to keep your links from being saturated and greatly oversubscribed... or it used to be. With next to no competition, verizon knows they can use those eyeballs as leverage to make content providers (or their ISP(s)) pay through the nose to get to them. If people had an option, they'd take their money elsewhere -- but they don't. (even if you're blessed with multiple broadband options, you'll find everyone is playing their greed card.)

Let's make the (purely hypothetical) math simpler... I'm going to setup an ISP offering dialup. I buy two 2 T1's: one data, one voice. That'll work out fine as the 1.5mbps data line will easily handle 24 analog modems (24 x 40-48kbps = 960k to 1.2m) But I've sold far more than 24 accounts, so I have to add more voice lines... still one data T1, 7 voice T1's. Now we have a problem: 1.5mbps is just not going to cut it. So who should be paying for more infrastructure, "kittenpix" as the service my customers are trying to access? Or me, the idiot who's oversold the network by a factor of 4? Obviously, I'd prefer "kittenpix" buy connectivity from me to directly reach my customers, thus paying *ME* to get to my customers.

Back in the era of dialup, no one thought like that because there were many operators, and it was fairly simple (and cheap) to start a new one. That sort of thinking would drive away customers, and fuel the creation of additional ISPs. The modern broadband landscape is the exact opposite... very few players, using complex and expensive systems built on long standing protected monopolies -- land line phone, coax cable tv, and (rarely) power lines -- that they viciously protect.

This is what you get with 'net neutrality'

[roman_mir]

This is 'net neutrality' for you - if a service is eating up all of the available bandwidth, who is supposed to pay for the degradation in quality of service exactly? Net neutrality is basically a price control and we know what price controls do: create shortages of supply.

Re:This is what you get with 'net neutrality'

[AvitarX]

It's not a price control, nothing about net neutrality says they can't charge their consumers for accessing extra data, and that is who they should charge, and it shouldn't matter the source.

Where is your Network Neutrality God now?

[SuperKendall]

So how exactly is Network Neutrality going to fix the ACTUAL problems we are having instead of the problems we are pretending we have?

It's not. You can't force a company to buy more bandwidth, unless you want to expand Obamacare to mandate ISP's by a "platinum" bandwidth package from back haul providers.

Re:Where is your Network Neutrality God now?

[lilrobbie]

The funny thing is, this article is an example of the system working as intended. The packets are being dropped because Netflix data appears to be flowing upstream faster than Cogent is paying Verizon. This is not a net neutrality issue... quite simply, Cogent is trying to push more data into Verizon than they are paying Verizon to handle.

All anti-net neutrality demands are attempting to do is treat packets unequally (e.g., allow Netflix packets through but not other customers on the Cogent network). You know, we can already do that easily: If Netflix wants to improve their bandwidth to Verizon customers, they start negotiating directly with Verizon for an extra uplink. Alternatively, they can do what they are doing here, and wait for their ISP and upstream to battle it out and see who has bigger clout (e.g., will Verizon's customers change ISPs if Verizon doesn't get the bandwidth sorted out, or will Netflix simply be replaced with a faster alternative?).

Destroying net neutrality achieves nothing, and just makes it basically a given that you will one day have an "express email" delivery charge to ensure your email is delivered in under a week ;-)

Re:Where is your Network Neutrality God now?

[SuperKendall]

This is not a net neutrality issue...

I know, I totally agree.

Yet if you asked 90% of the people if they supported Net Neutrality because it would speed up Netflix for them, they would say yes. Just look at the previous story, this idea that it was simply network congestion - but MANY were more willing to believe Comcast was degrading service, and screaming for Net Neutrality Now - even though as you say it would not help the problem.

The other point is no REAL problem we've had with the internet would ever have been solved by Net Neutrality. So why make it more expensive for ISP's to operate by having to understand a bunch of new rules that do nothing?

Re:Where is your Network Neutrality God now?

[guises]

quite simply, Cogent is trying to push more data into Verizon than they are paying Verizon to handle.

The problem being that there's really no reason why Cogent should be paying Verizon any money at all. Verizon's customers have already payed for that bandwidth.

This is an end-run around network neutrality: Verizon can claim that they're staying neutral because they're slowing down all of Cogent's traffic evenly, rather than targeting Netflix specifically, but the great bulk of Cogent traffic is Netflix.

RE: "Lopside"?

[Anonymous Coward]

Peering arrangements are typically ran such that there are equal amount of sending and receiving traffic on both sides.
With the advent of Content Delivery Networks this is about impossible to achieve unless you are talking about two Tier1 ISPs exchanging traffic such as Level3 and Cogent.
Basically, Verizon are claiming that because Verizon customers want to watch Netflix, Cogent should pay them to allow the traffic.

Incentive Structures

[resistant]

Basic incentive structures are a serious problem that will only grow worse over the years. The more successful become edge businesses such as for-profit streaming video providers and monetized social networks that go heavy on large images and video advertising, the more the data carriers will enviously demand a cut of the big pie that other businesses are dividing up.

I wish I knew of a good solution to this problem that still shows reasonable respect for the free market. Perhaps it's time to give heavy federal and state tax incentives to businesses that do absolutely nothing *but* provide consistently reliable, high-speed data-transmission channels. That means no end-user content provision or end-point Internet connections for individuals, businesses or other organizations. Let the innovators scramble to implement gigabit capabilities that end-point ISPs can resell to their customers. No doubt there are sorts of problems with this idea, but my brain has nothing better right now. :/

Re:Incentive Structures

[number17]

Didn't the free market invent bandwidth caps with overage fees to cover those users who download the most content? This was due to some absurd amount of 90% of the traffic was from 5% of the users who were downloading pirated material.

Fast forward 12 years and its the average user who is sucking down that bandwidth.

Slashdot BETA

[giorgist]

What happened here ? How can I go back to the old Slashdot ?

Re:Slashdot BETA

[Anonymous Coward]

Click "parent" in a comment and soulskill will come and magically take you back to the old site...

And this is how it's supposed to work

[Anonymous Coward]

Look, for years and years, peering agreements have been based on the idea that peering between partners would have roughly equal inbound and outbound traffic patterns. When one partner starts pushing more traffic out than in, it signals an imbalance in the connection, and actually the burden.

Let me explain: internet routing is based on autonomous system (AS) hops. Cogent is an Autonomous System, so is Verizon, Level3, etc. If I'm connected to one AS, and I need to send a packet to someone on another AS, the router within my AS attempts to deliver that packet to the closest exit point for that AS. So for example, if I'm sending packets from Cogent to a user on Verizon, then Cogent will offload the transport of the packet to the closest point possible (usually in the same city). The idea being that Cogent doesn't know where the user is on verizon, so they just want to send it to verizon using an Exterior Gateway Protocol (e.g. BGP - which is AS hope based), and then verizon will use their Interior Gateway Protocol (e.g. something like OSPF) to deliver it most efficiently inside of their network.

And therein lies the problem with unequal inbound and outbound peering situations. If Cogent is 80% inbound and 20% outbound with a specific peer, that is usually shouldering the burden for most of the transport distance and cost for that packet. If it's 50/50, then the burden is the same, and they're equal peers.

In the internet routing world there are tier1 providers, tier 2 providers, etc etc based on how many peers they have. But of the tier 1's - not everyone is created equally. Cogent has Tier 1 status, and their POPs are all interconnected, but they don't have as many geographic POPs as say a Verizon. As a result, they dump packets to their peers as local as possible and those packets are routed across the internet by the peers. Their piers on the other hand, for the burden that *should* be cogents have fewer locations that they peer with cogent due to their limited # of POPs. So even for the outbound to cogent traffic, they end up shouldering a disproportionate amount of that traffic transporting it to one of cogent's pops.

And then you have companies like netflix. Netflix buys bandwidth from these low cost tier1 providers, who are low cost because they don't share the transport burden with the real tier 1s. And when congestion happens, and the real tier1's tell cogent "sorry you're out of bounds withe contract because you don't have an even inbound / outbound ratio" cogent decides not to pay the penalties for the uneven traffic patterns. Instead, they let congestion and packet drops happen. The Netflix comes along, and says "Oh yeah, this provider, they're terrible for netflix traffic - they don't upgrade their pipes." They do this knowing full well the economics of internet peering. How do they know full well? Because they don't even want to pay cogent. All the while that they're letting the tier1 take the heat for the crap peering situation, they're approaching said tier1 saying "Hey, nice network you got there. It would be a shame if someone publicized bad information about its performance. Hey, you know how you can prevent that from happening? Join open connect! All you have to do is host our hardware, provide power, and data center space, and cooling, oh and connectivity for free and your network will look great for netflix customers who won't have to suffer through this peering situation you have going on with Cogent."

And now, Netflix, not being a back bone at all, has managed to make you look bad and then tell you in order to look good you should host their hardware and give them free transport... And because /. is /. Netflix comes out looking like the victim here...

RE: And this is how it's supposed to work

[Anonymous Coward]

Verizon, Comcast, and other last mile providers are far from Tier1. Since Verizon is requesting data from Cogent, they have the obligation to either upgrade or route the traffic through another peer. I would highly doubt that many last mile providers are anything more then just that "eyeball networks" with lopsided traffic to begin with. The old peering models are essential useless when discussing a balanced peering arrangement between and CDN and Eyeball network.

Re:And this is how it's supposed to work

[sjames]

The transit distance is perfectly valid, but the up/down ratio makes no sense in that situation. Every byte a Cogent cuetomer sends is a byte a Verizon customer asked for.

Consider, if Netflix modifies their protocol so the client echos all of the data back, does that make Verizon happy? It would balance the up/down ratio neatly.

Bandwidth ratios make sense when the peers are also providing transit to other networks. It makes sense that if A and B peer with transit, they will only want to do it for free is they have roughly equal TRANSIT traffic. (that is, traffic between A and C going through B should balance with traffic between B and Z going through A).

Re:And this is how it's supposed to work

[lilrobbie]

They key problem in this mess isn't Netflix using up so much bandwidth (they aren't pumping things out, consumers are pulling things down)... it's actually Cogent. Cogent is clearly not paying to keep up with the data Netflix is consuming. Cogent needs to pay Verizon more to ship the data around (according to their peering agreement)... and should probably pass that data cost back onto Netflix which will force Netflix to find ways to balance out the bandwidth they consume.

Netflix's proposed solution is a legitimate strategy that reduces congestion and helps both Netflix AND the ISP they place with. The ISP receives less inbound traffic (usually good for their peering agreements), the ISP's customers get access to a service they desire, and Netflix is able to place some cheap servers.

Netflix is not a victim, I agree... but neither are they some type of protection racket trying to force tier1s to place their server. Netflix would be no threat if people weren't subscribing to it, so your analogy is a little off there.

Re:And this is how it's supposed to work

[Anonymous Coward]

Big networks tend to prefer MPLS-TE or GMPLS over OSPF. There is probably still OSPF lurking underneath somewhere, but it only really exists to configure routes so that LDP can distribute the labels for MPLS. The actual traffic across the network is carried is routed at the provider edge (PE) and encapsulated in MPLS, which is not routed but switched by each core router until it reaches another PE router.

The reason for this is that MPLS forwarding is massively faster and doesn't require core routers to maintain eBGP views, as they only need to know which labels map to which interface, and need not know anything about the contents of final destination of the traffic.

Of course it probably won't be long before MPLS is old-hat, and Openflow takes its place, as it requires no routing whatsoever and can be done by a dumb L2 openflow enabled switch, as long as the hardware FIB (TCAM) is large enough to contain every PE in the network.

A 48 port 10GBaseT wirespeed Openflow switch can be had for as little $30k. An eBGP PE router that can do 480Gbps costs a bunch more than that (if it even exists).

Re:And this is how it's supposed to work

[Anonymous Coward]

Yes, I would think that if Netflix altered their protocol to send back more data, they would even out the transit situation and make it easier for their tier 1.5's to negotiate with the real tier 1s. In that way, cogent would be shouldering some of the burden for the longer transit path.

I'm not saying that this is the ideal situation, but it is how it's worked pretty much since the beginning, and it worked then mostly because the large tier 1's had both ISP customers and Content Provider customers. The balance had certainly shifted to a point where half a dozen or so Tier 1's control 80% of the home customers. The largerish tier 1s also have a large # of enterprise and content provider customers - so within their networks inbound and outbound are much more balanced than say with a cogent.

But the fact of the matter is that today, cogent and companies like it act like large hosting providers who dump traffic on their local peers, and avoid having to build out the massive backhaul to all of the geographic locations themselves. This is not a matter of Verizon being dicks, this is the reality of how peering works - and how a handful of companies have built models around being the low cost providers, doing so only because they themselves shoulder the burden for a limited amount of the transport.

If cogent doesn't like the situation, they should renegotiate their contract.... But if you do it based only on transit distance the situation would be *much worse* for cogent than it is today. They'd be proverbially fucked, essentially paying verizon, att, comcast, centurylink, etc for transport instead of being a tier-1 peer. Either that, or they'd have to massively expand their platform, being local CDNs in virtually every state in the US, so that customers like netflix are always served from really short transit distances. That too would fuck them, all of the long haul transport to connect all of the POPs is not cost effective.

Also, need I remind you that while the real tier1's are holding cogent's butt to the fire, cogent has actively been huge dicks to other companies aspiring to be tier1's. Look at their past peering disputes with other companies like xo. Not to mention that they've essentially split the IPv6 internet by refusing to peer with hurriance electric - who at this point, due to their excellent v6 tunnel policies and open peering policies is one of if not the largest v6 player out there (just look at the # of prefixes they carry vs cogent, not to mention traffic).

Cogent is the bottom of the barrel, taking advantage of the situation. And Netflix knows this and they're doing the same. And the ignorant /, masses are bitching about net neutrality when this has *nothing* to do with net neutrality - in fact this is a much more net neutral situation (maintaining the status quo for peering / exchange of traffic) than anything else.

Re:And this is how it's supposed to work

[sjames]

Believe me, I am not holding Cogent up as some sort of paragon of virtue. Their sins are many (they have even billed me for 'overages' months after I disconnected from them entirely).

I just find the arguments over traffic ratios to be vacuous. Unless the peering includes mutual transit, the issue is entirely immaterial.

If a carrier doesn't want to be abused with hot potato routing, they should control what routes they advertise through their peering points.

The issue is especially silly in the case of Netflix since Verizon could opt to host a netflix cache and no longer have the inbound traffic at all.

Re: And this is how it's supposed to work

[Anubis IV]

http://en.wikipedia.org/wiki/Tier_1_network#List_of_tier_1_networks

I was about to rattle off a post agreeing with you, but I thought I'd double-check, and it turns out that both Verizon and Cogent are now considered Tier 1 providers. Caught me by surprise, since I don't recall Cogent being anything more than a Tier 2 as of a few years ago when I used to watch this stuff more often, and I thought Verizon was still a large Tier 2 as well. Turns out they're not considered that any more, however.

But yes, most last-mile providers are Tier 2 or Tier 3.

Netflix should get benefit from desirability

[SuperKendall]

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

Why? A lot of people might only get internet, or faster internet anyway, BECAUSE of netflix.

If I were not streaming stuff on Netflix I might very well just use a cellular internet connection and not get cable internet at all. Netflix is helping the ISP's make money, and Netflix should gain some benefit from that fact as a result.

Re:Netflix should get benefit from desirability

[dave562]

Do you have any hard data on how many people are actually purchasing high speed internet specifically for Netflix? I have not seen any data, but my gut feeling is that the number is a fraction of a percentage of people who have high speed connections. Would you agree that by and large, the majority of people viewing Netflix are people who already have high speed connections?

Right now Netflix is getting to benefit at the expense of ISPs. Do not get me wrong here, I do not like the way the ISPs are handling it and I think it is disingenuous that they are not keeping their circuits up like they should be. My issue here is that Netflix is not a good champion to use to make the case for ISPs being tightfisted when it comes to circuit maintenance.

Netflix partnered with Cogent. They could have partnered with Level3 or any other Tier1 provider and they would be in the same situation. The peering status quo has broken down. Cogent needs to pay to make up for the lop sided transfer ratio. Cogent will most likely pass that cost on to Netflix. Netflix will have to raise their prices to offset the cost hike. Until that happens, Netflix is able to provide artificially low rates to their customers at the expense of other ISPs. Bandwidth is not free so someone has to pay for it.

To use a real world example, when builders, either residential or commercial plan new communities, they have to do environmental impact studies. One of the essential components is usually a traffic study. If they are going to go for a high density development, like a massive multi-story apartment complex, most of the time they have to work out a deal with the city to widen nearby roads, install traffic lights, widen the sidewalks, and do all sorts of other things to mitigate the impact. Right now, the internet does not have the equivalent of that, and Netflix is benefiting from it.

Imagine if your neighbor turned his house / apartment into a concert venue because there were not any ordinances to prevent it. All day long, you could not find a place to park, could not have guests over because they could not park, suffered brown and black outs because the music equipment kept blowing out the local transformers, etc. How would you handle that?

Re:Netflix should get benefit from desirability

[0123456]

Do you have any hard data on how many people are actually purchasing high speed internet specifically for Netflix?

The only reasons I need megabit Internet are downloading games from Steam and GOG, and watching videos. Games, I don't much care if I have to leave the PC running overnight while it downloads, but streaming videos are either high enough bandwidth to play, or they're not, and will stutter and buffer or drop to low quality.

So I suspect many people are buyig high-speed Internet just to watch videos, be that Netflix, Youtube, or pirate sites.

Re:Netflix should get benefit from desirability

[vux984]

Right now Netflix is getting to benefit at the expense of ISPs. [...] My issue here is that Netflix is not a good champion to use to make the case for ISPs being tightfisted when it comes to circuit maintenance.

If the ISP cannot transfer me the traffic that I am paying them to transfer me, then they need to upgrade their circuits.

If they legitimately cannot afford to do that, then they need to charge me more money.

But if they cannot afford to do that, why is their investment news a string of "profits up 40% in 2013, ARPU up 9.4%, 14.7% year over year increase in FIOS revenues, 78c EPS 2013 vs 64c EPS in 2012, ... etc...

Imagine if your neighbor turned his house / apartment into a concert venue because there were not any ordinances to prevent it. All day long, you could not find a place to park, could not have guests over because they could not park, suffered brown and black outs because the music equipment kept blowing out the local transformers, etc. How would you handle that?

My neighbor and I both subscribe to Verizon to provide "parking and electricity" in this situation. So if Verizon can't provide
"parking and electricity" to me per our agreement, because my "neighbor" is using too much, then Verizon needs to build more parking and provide more electricity. If they have to charge me and my neighbors more, so be it... but see above regarding large increases in revenues and profits... it seems Verizon is simply failing to upgrade our electricity and parking and pocketing the money we paid them to do that instead.

Re:Netflix should get benefit from desirability

[lordofthechia]

I have not seen any data, but my gut feeling is that the number is a fraction of a percentage of people who have high speed connections.

Look at it this way:

* Netflix has over 30 Million subscribers .

* According to the US Census data, 75% of US households have internet.

* 115 Million Households in the US

Some math:

115 Million households * .75 = 86.2 Million households with internet.
31 million Netflix Subscribers / 86.2 Million Internet users = 36% of households with internet in the US use Netflix. Yes, over a third.

Now add the fact that cable companies are losing cable TV customers in the hundreds of thousands *each quarter* and you can see how more and more people are depending on Netflix (and other services) to fill their video entertainment needs. The name for the trend is cord-cutting .

So based on that I would surmise that a substandard Netflix experience would be a dealbreaker for 1/3rd of all internet customers and a larger proportion of *high speed* internet customers (since people who don't need to stream video opt for the cheaper services).

Re:Netflix should get benefit from desirability

[evilviper]

If the ISP cannot transfer me the traffic that I am paying them to transfer me, then they need to upgrade their circuits.

It's not that simple. Peering arrangements are private contracts that don't have anything to do with end users. And if you demand your ISP gives their peering partners anything they want, the peers will take massive advantage, and dump tons of their own traffic on your ISP's links, to get delivered across the country by them. When that happens, they won't be charging you a few percent more, they'll be charging you an order of magnitude more money, as your dumb sap of an ISP gets stuck delivering every other ISP's internet traffic for FREE.

Re:Netflix should get benefit from desirability

[vux984]

Peering arrangements are private contracts that don't have anything to do with end users.

And if you demand your ISP gives their peering partners anything they want

I don't demand that at all.

There is a middle ground where the ISP can figure out how to deliver traffic to me without having to deliver every other ISPs traffic for free. Those peering contracts can be arbitrarily detailed.

Re:Netflix should get benefit from desirability

[Anonymous Coward]

Arn't you Americans famous for suing and class actions?

So, Sue your ISPs, class action, demand they provide the service you paying for.

ie : unrestricted access to the internet; issues on the backbone is their problem, not yours.

Re:Netflix should get benefit from desirability

[MikeBabcock]

You really don't understand the system at all, do you?

The Internet backbone is not like your analogy at all.

There are two basic types of Internet connection (to oversimplify); there are the connections between ISPs where each provides fairly equal benefit to the other; these are often 'free' except for the hardware investment. The other is the kind where service is sold to someone who has no benefit to exchange; either a user buying primarily downlink service or a web host requiring primarily uplink service, etc.

We end up with a chart that looks a bit like this: Netflix -> ISPa <- M -> ISPb -> User.

ISPa gets money from Netflix, ISPb gets money from user. So far so good? Now ISPa and ISPb are linked to each other because its in their mutual benefit. That is to say, if ISPa didn't have this link, they wouldn't have a service to sell because part of the Internet wouldn't be as accessible to their clients. By the same token, ISPb needs the link as well. If this isn't true; they simply cancel that link and the packets route around elsewhere.

For ISPb to claim that Netflix owes them money is preposterous -- the packets flowing from ISPa to ISPb are part of a peering arrangement which only exists because its beneficial to both sides. If its isn't, the side gaining more benefit is charged money by the other side. In fact, some sites offering those 'free' mirrors for downloading software only do so to keep their peering usage up (as an aside).

Re: Netflix should get benefit from desirability

[dave562]

Thanks for the lesson. Seriously.

What am I not understanding though? The ISPs are upset with Cogent because Cogent is pushing more data than they are pulling. The equation is lopsided.

Netflix is trying to alleviate the issue by rolling their own CDN to cut down on the peering traffic.

Re:Netflix should get benefit from desirability

[Anonymous Coward]

And if you demand your ISP gives their peering partners anything they want, the peers will take massive advantage, and dump tons of their own traffic on your ISP's links, to get delivered across the country by them.

That is not possible, and would be a violation of internet engineering rules, and a breach of contract.

A peering arrangement, in contrast to a transit arrangement, only advertises routing prefixes within the AS of the eBG router, and is not obliged route any traffic for routes it has not advertised (it is obliged to route all routes it does advertise). Any sensibly configured eBG router will have IP filtering so that it doesn't deliver traffic for prefixes it hasn't advertised, and it would take deliberate manual configuration in contravention of IER to forward traffic for routing to a peer which has not advertised route availability for the traffic in question.

On the other hand, Cogent is acting as a transit provider while pretending to be a peer, which explains why Verizon is being hostile. Cogent is using an early exit strategy, whereby they route into nationwide ISPs networks at the closest available route, knowing that nationwide ISPs have internal nationwide networks for delivering traffic within their network. They could institute route filtering so each peering point only advertises routes to nearby customer edge networks, and Cogent is forced to carry traffic cross country before dropping it on Verizon and AT&T, but this would complicate network management, create potential routing failures within their network, and not be fair to other peers who may not have nationwide delivery.

Instead I think Verizon are taking the most amicable option available to them, letting Cogent saturate their peering links and expecting Cogent to build more peering links or deliver their traffic closer to it's actual destination as every transit ISP is expected to do.

Re:Netflix should get benefit from desirability

[Anonymous Coward]

If the ISP cannot transfer me the traffic that I am paying them to transfer me, then they need to upgrade their circuits.

It's not that simple. Peering arrangements are private contracts that don't have anything to do with end users. And if you demand your ISP gives their peering partners anything they want, the peers will take massive advantage, and dump tons of their own traffic on your ISP's links, to get delivered across the country by them. When that happens, they won't be charging you a few percent more, they'll be charging you an order of magnitude more money, as your dumb sap of an ISP gets stuck delivering every other ISP's internet traffic for FREE.

It's simple as this: You pay for access to the Internet and the ISP's connections are not capable of keeping up. The ISP is at fault and must find a solution in a reasonable time. If your ISP can not get a favorable peering agreement, then they should pay transit to another ISP does that have a peering connection that is capable of handling the load. Simple as that.

There are multiple ways for Verizon to get to Netflix. If Cogent is not favorable, then Verizon should stop peering with Cogent all-together, not let it remain in a degraded state. Why doesn't Verizon want to do this? Because it will cost them even more money to drop Cogent. Cogent knows this and is using it to strong-arm Verizon. This is how ALL Tier 1 negotiations work. Whomever has the most leverage, applies it and someone has to give.

Maybe Verizon should do what Level 3 does, no congested links ever. If it even approaches congestion, you either fix it or remove it, but you do not ever let it stay that way. Verizon is playing chicken while their customers suffer.

Re:Netflix should get benefit from desirability

[Anonymous Coward]

Netflix is breaking the long standing status quo. Last I checked, they accounted for ~30% of ALL of the traffic on the internet. Obviously that is going to skew the metrics, and that is why Netflix is trying to push their own CDN. I do not know the particulars there. IMO, if Netflix expects ISPs to pay for their CDN, they are on drugs.

Why? A lot of people might only get internet, or faster internet anyway, BECAUSE of netflix.

If I were not streaming stuff on Netflix I might very well just use a cellular internet connection and not get cable internet at all. Netflix is helping the ISP's make money, and Netflix should gain some benefit from that fact as a result.

I upgraded my internet connection specifically for higher quality video streaming. Nothing else I use the internet for *needs* the higher bandwidth.

Dispute between companies affecting service

[Anonymous Coward]

What are their customers, chopped liver?

Re:Obamaville. We haz it.

[Anonymous Coward]

Was that bushville 8 years ago? Clintonville 16 years ago? Greater Bushville before that? Reganville?

But yeah, homeless people are all Obama's fault.

Re:Yes, it is true!

[bobbied]

I'm a preposterous penis pumping extremist! Bow before me, Slashdot!

Dude, you are hard core..

Cogent, the UDP of ISPs

[tokencode]

Cogent is to ISPs what UDP is to network protocols.

Re:Obamaville. We haz it.

[Anonymous Coward]

imho you weigh 350+ pounds

Re: "Lopside"?

[K. S. Kyosuke]

Isn't Netflix sufficiently large to be able to afford having direct connections into multiple major ISPs? Obviously, not being US-based, I have little idea how these things work in the US (we're pretty happy with a single peering nod in our tiny little country) but it would at least eliminate the peering disputes.

It's not just Netflix Traffic

[Anonymous Coward]

It is affecting ALL traffic. I have a customer who cannot reach our servers during the afternoon because of the amount of traffic and packet loss. So the customer is getting screwed here because Verizon and Cogent are fighting it out.

We're Surrounded by Morons.

[VortexCortex]

If I gave you a decentralized network capable of surviving nuclear war, routing packets around cities that vanished in moments, and you then built a World Wide Web of Data Silos then I'd be within my rights to call you all fucking morons!

STORE AND FORWARD. How the hell can't you realize this is the decentralized solution that the Internet needed, not a centralized Server / Client cluster fuck? What is collocation? IT'S WHAT YOU GET FOR FREE WITH STORE AND FORWARD, idiots. Oh there's all that Youtube, Netflix, etc., bandwidth? Well, what if I told you I could reduce your peering bandwidth to ONE COPY of each resource? That way if your neighbor recently watched a show or cat video you could download it directly from them or the upstream cache they're connected to? A router should only ever have been one part of the node, the cache is an essential part, as part of any strategy for load distribution. Ah, but you don't need to keep a copy of each resource at each node if you index the data via distributed hash table -- it's not rocket surgery fools.

How else do you think the Space Internet will work? NASA's DTN and even old-timey HAM Packet Radio are smarter about data than the fucking web! Unlike the WWW, they use Store and Forward. HTML allows lightweight dynamic pages to pull in heavy static resources, if only assets had a <... hash="SHA-512/Base64:MDVjMjg4YmY2ZGV...hMGEx==" hmac="Base64:..."> then secure pages could pull in unsecured static resources and browsers could verify their validity without "mixed content" warnings -- Oh but that would mean the architects would have to ACTUALLY KNOW WHAT THE FUCK THEY WERE DOING. I'm surrounded by morons. I mean, you could even just use your HTTP-AUTH proof of knowledge to key your ciphers and eliminate the Certificate Authority MITM, but noooo... Morons, I Say! MORONS!

Re:We're Surrounded by Morons.

[SuricouRaven]

I really couldn't agree more. That would be a great way to do things. It wasn't built in from the beginning because it wasn't practical until recently - storage just cost too much. It isn't being built in now because the existing models work-ish, and no influential organization has been willing to lend their backing to deploying the new technology.

I did have the idea of encoding the hash addresses as a 'magic directory' in HTTP - eg, http://your.server.com/CANary/...//filename. That way any browser or software aware of the address form can run a SHA search of reachable caches, while any non-aware software (or if the cache search fails) just interpret it as an HTTP address and get the file the old way. Even if there are no caches in reachable, it'd still save a lot of IMS requests when re-visiting a website.

Making personal or household devices public cache servers is probably not the best idea though. Privacy concerns - it wouldn't take long for someone to write a script that queries a specified cache for the top thousand video hashes from the popular porn sites and see if they have it stored already.

Re:We're Surrounded by Morons.

[SuricouRaven]

Slashdot mangled my example link. Huh. Here's another example:
http://birds-are-nice.me/CANar...
Incidentally, if you can tell me what that song is, I'd be very grateful. If it's pre-1963 it's public domain in the UK and I can add it to me 'legal to use' collection.

Re:We're Surrounded by Morons.

[neo-mkrey]

Don't hold back now, tell us how you really feel.

Re:We're Surrounded by Morons.

[the eric conspiracy]

In the degenerate case (one transmission of a piece of content to a local server) store and forward is indistinguishable from having local servers.

Which is what Netflix is doing with the ISPs willing to partner with them.

Re:We're Surrounded by Morons.

[Anonymous Coward]

Amen, brother.

Re:We're Surrounded by Morons.

[sjames]

The sad thing is that Netflix makes a standing offer to provide a cache to ISPs. They will happily provide them for Verizon and greatly cut down on their external network usage.

Verizon refuses the offer.

Re:We're Surrounded by Morons.

[SuricouRaven]

It's still a very inelegent cache though - it only provides caching for a single service, at one additional level.

Re:We're Surrounded by Morons.

[Anonymous Coward]

Well, there is this working group and the introductory RFC6707, which outlines pretty much the exact problem you identify, in a much calmer and more effective manner.

Re:We're Surrounded by Morons.

[Anonymous Coward]

That's not what Netflix is doing though, it's not what anyone is doing. It's what usenet used to do back when every ISP ran a usenet server.

Netflix CDN, like every other CDN, only cache's content for the content providers under it's umbrella (typically the one's who pay). What is needed is an internet architecture with more local caches which cache content from ANY content provider. The content itself can be encrypted, and only decryptable with a key sent directly to the user form the content originator.

In fact this is the exact problem statement in RFC6707 and their is an active engineering effort to solve this problem (CDNIWG):

Content Delivery Networks (CDNs) provide numerous benefits for
      cacheable content: reduced delivery cost, improved quality of
      experience for End Users, and increased robustness of delivery. For
      these reasons, they are frequently used for large-scale content
      delivery. As a result, existing CDN Providers are scaling up their
      infrastructure, and many Network Service Providers (NSPs) are
      deploying their own CDNs. It is generally desirable that a given
      content item can be delivered to an End User regardless of that End
      User's location or attachment network. This is the motivation for
      interconnecting standalone CDNs so they can interoperate as an open
      content delivery infrastructure for the end-to-end delivery of
      content from Content Service Providers (CSPs) to End Users. However,
      no standards or open specifications currently exist to facilitate
      such CDN Interconnection.

      The goal of this document is to outline the problem area of CDN
      Interconnection for the IETF CDNI (CDN Interconnection) working
      group.

Re:We're Surrounded by Morons.

[Anonymous Coward]

Privacy is anti-social. -- Anthony David

Bad article or am I a Pollyana?

[mythosaz]

Cogent sells transit for an average price of $1.31 per megabit, though large volume users like Netflix get discounts.

Wut?

Re:Bad article or am I a Pollyana?

[amorsen]

Why do you say wut? That seems correct to me?

sell is the key word. Cogent not paying Verizon

[raymorris]

They SELL connections to their customers. Cogent isn't paying Verizon. The peering they're talking about is an even trade with neither company paying the other. When it ceases to be an even trade, it's time for negotiations.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Yes, they SELL connections to their customers, so the incoming flood of traffic from Cogent has already been paid for by Verizon's customers that are trying to watch their shows on Netflix. Verizon is trying to double-dip. So what if the traffic is asymmetric? If it's that big of a problem, the continued performance problems caused by Verizon's intransigence could be solved by a massive reduction in the customer base once they find that they can't watch Netflix.

Re:sell is the key word. Cogent not paying Verizon

[msauve]

Customers PAY for those connections. Verizon's customers are paying to receive traffic from the Internet. Whether that's slashdot or Netflix doesn't matter, it behooves Verizon to deliver the service their customers are PAYING for.

Re:sell is the key word. Cogent not paying Verizon

[0123456]

From what I understand, peering agreements are typically based on a similar amount of data going both ways, so both sides benefit from just connecting their networks together with no need for additional payments. But, if 90% more data is coming in than going out, then the network sending the data is going to have to pay the network that's receiving it. Which means Netflix will have to pay their ISP more so they can pay to deliver the data.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Which makes sense if the "sending" network is sending broadcast packets or something, but users of "receiving" network are paying "receiving" network to connect them to "sending" network, so peering should be a wash.

Re:sell is the key word. Cogent not paying Verizon

[i.r.id10t]

so my ISP should be paying me instead of vice versa? i certainly download much more than I upload....

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

i guess simply providing your customers the experience they expect isn't a benefit

Re:sell is the key word. Cogent not paying Verizon

[viperidaenz]

Why should there be a difference between sending traffic and receiving it?
The only time there is a difference in data transfer between A and B is when one of them drops packets.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

The thing is, private peering doesn't cost you anything. The receiving end is not going to send it onwards unless they're colossal idiots who can't configure a router, so it can't possibly cost them anything. So what the hell are they complaining about?

Re:sell is the key word. Cogent not paying Verizon

[sjames]

Every packet coming from Cogent to Verizon is because a Verizon customer paid for it. It really is as simple as that.

It only seems complicated because the kooky business people make it complicated trying to pass their costs off onto others.

Re:sell is the key word. Cogent not paying Verizon

[sjames]

That's what they're doing, but the posts above explain well why it is brain dead.

Re:sell is the key word. Cogent not paying Verizon

[NormAtHome]

I know this topic has been beaten to death here, but I see it the same way you do. Verizon customers pay a toll (their monthly charge for internet access) to use Verizon's connection to the internet as a whole. No Verizon customer should have their data throttled no matter what site they are accessing as long as they are in compliance with Verizon's TOS.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Yes. I believe that's it. But that was also a issue with Cogent and others 10 years ago. You could get cheap webhosting from companies on Cogent. But it wasn't exactly a secret that you got what you pay for and people knew that they had peak time issues.
So I am wondering why this comes as a surprise now. (again)

Re:sell is the key word. Cogent not paying Verizon

[JesseMcDonald]

No, because the ISP owns the infrastructure and you're the intended recipient. They're not asking you to deliver packets to someone else on their behalf.

Logically, everyone should just pay for the packets they send, much like you pay (via stamps) to send an envelope. A company like Netflix would pay to have their content delivered to their customers, and would recoup the cost of "shipping" that content across the Internet in the form of subscription fees. Unfortunately, while that might work for Netflix, it would be problematic for sites serving small amounts of traffic to a large number of visitors—we still lack a practical and widespread means of micropayment.

Of course, one significant difference between the Internet and the postal service is that the postal service is required to deliver (or return) every envelope they accept—they can't just drop excess mail in the nearest incinerator when they become overloaded. At that, it would still be an improvement over the current Internet model, where not only do you have to pay for packets which were never delivered due to congestion or technical problems, you even have to pay for packets you never requested and had no opportunity to opt out of.

Re:sell is the key word. Cogent not paying Verizon

[F.Ultra]

It might not be logical but that is how the ISPs have decided to handle peering.

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

Yes, they SELL connections to their customers, so the incoming flood of traffic from Cogent has already been paid for by Verizon's customers that are trying to watch their shows on Netflix. Verizon is trying to double-dip

NO. Verizon customers pay for a connection to Verizon's network --- you are paying for half of the equation, not the end-to-end transport; the fee paid by Verizon subscribers includes the customer's connection, but other Verizon peers and customers still have to pay in order to interconnect with Verizon's network and data sent to Verizon's other customers (unless a mutually beneficial peering arrangement can be reached that Verizon determines is either equally beneficial to both sides or benefits Verizon more).

This is not unusual, and all Large Tier1 providers behave in this manner, in order to protect their revenues and prevent loss-of-revenues due to providing free peering, when they could charge for service.

Double-dipping would be getting Cogent to pay extra, AND THEN..... limiting Netflix traffic, and reaching out to various contacts at Netflix to offer to prioritize their traffic, for an additional fee per terabyte, despite Cogent paying extra for all the paid transit of Netflix traffic through Verizon's network..

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

Which means Netflix will have to pay their ISP more so they can pay to deliver the data.

Yep. And probably one reason all the Netflix traffic is going out Cogent, is Cogent has among the lowest fees per Gigabits of data transferred for data in the industry (But a reputation for being the internet's storm gutter due to unreliability in terms of quality metrics such as jitter, packet loss, latency). So instead of buying for $X a gigabit from Verizon, Netflix buys transit from Cogent instead, at a price that is most likely about 80% lower... which no doubt saves Netflix a lot of money.

The extremely low price is only possible by exploiting and abusing free peering relationships with other Tier1s. If Cogent charged Netflix a higher, more reasonable price ---- that would be more in-line with the fees charged by Verizon, ATT, Level3, etc; it is likely, that Netflix would rationally engineer their traffic, among MULTIPLE transit providers, for best reliability across the board, so that they were more balanced among the Tier1 providers, and not pimping out Cogent links.

NETFLIX Could do this today, and they apparently choose not to ---- probably because the Netflix subscription prices would have to go up substantially.

Re:sell is the key word. Cogent not paying Verizon

[Sancho]

And this is why I'd really like to see Internet providers become common carriers. The Internet has become much too important to let squabbles like this endanger it.

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

The thing is, private peering doesn't cost you anything.

No.... the data transferred still uses data capacity.

Lost revenue. Cogent is using Verizon's free peering to compete against Verizon, by selling the same bandwidth (sustained gigabits of data transfer) to Netflix, that Netflix would have to pay four to five times as much for if they bought an internet connection from Verizon.

In principle, to protect their revenues: the Tier1's need to ensure that they get paid by the other party connecting to their network for data coming into or leaving their network, that is using up some of their network capacity.

They either get paid via a transit fee from the party connecting, or by gaining an equivalent amount of usage of the peer's network.

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

It behooves Verizon to deliver the service their customers are PAYING for.

As far as Verizon is concerned, they are delivering that service. It's not Verizon's problem, that Netflix has chosen not to pay Verizon for a connection and transit, because Cogent is so much cheaper.

Meanwhile, Cogent wants to slurp off a free peering arrangement from Verizon, to assist Cogent in competing against Verizon in selling Netflix internet connectivity to Verizon's customers.

If customers were that concerned about it, they would be calling up Netflix to complain about the problem.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

For the most part, you only do paid peering agreements for transit. Backbone ISPs are nearly always transit, so peering requires ratios, but residential customer ISPs are the destination, so no transit, thus no paid peering. Because the customer has already paid for all of the bandwidth, ISPs typically try to lower their costs by getting free bandwidth. In this case, Verizon charges the customer, then refuses to deliver until Cogent/Netflix pays up. Double Dipping.

Re:sell is the key word. Cogent not paying Verizon

[pnutjam]

What the story on Netflix caching devices? Does Verizon use them?

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

It's not the direction, it's the ratio

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Tier1 ISP do not sell to residential. Verizon as a whole owns both Tier1 and Tier2 ISPs. We're talking about the Tier2 ISP. Tier2 ISPs, which sell residential connections, always always try to get free bandwidth. They rarely turn it down because their users have already paid for that bandwidth, no point in charging yet another party for what has already been paid.

Netflix would have to pay four to five times as much for if they bought an internet connection from Verizon

Cogent is actually charging a relatively high price already, but that's because they offer other services than just bandwidth to Netflix. If Verizon would charge 4x-5x the price, then they're price gouging.

According to Cogent, Verizon is quoting Cogent a price that is 10x the industry average for transit, and Cogent isn't even buying transit, they're buying a non-transit connection. It's highway robbery. Verizon has an effective monopoly, and they're flexing their power. In the end, Verizon is trying to force Cogent into paying about 20x-40x the industry average.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Verizon is trying to charge over 10x the industry average. You may want to rethink who is trying to abuse whom.

Re:sell is the key word. Cogent not paying Verizon

[nobuddy]

Does not matter, it is still traffic in the pipe.

Re:sell is the key word. Cogent not paying Verizon

[MikeBabcock]

You obviously don't understand peering arrangements at all. That's not how the Internet works.

Re:sell is the key word. Cogent not paying Verizon

[mlyle]

Shortest exit path routing.

It's pretty much what everyone's expected to do and makes technical sense: you don't know the topology of the other guy's network, so if you have a packet for one of his customers, you should hand it over to that network at the closest possible point, rather than hauling it over your backbone and possibly having the other guy haul it back if your assumptions about costs are wrong..

But this means people that are "push" heavy use a lot less backbone resources than the provider that is receiving the traffic. Historically that has resulted in payments when peering connections have significantly asymmetric traffic flows.

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

What the story on Netflix caching devices? Does Verizon use them?

My understanding is: No.

Verizon would rather sell more data, and having a local caching device on their network, would decrease the amount of data they get to sell.

Even with local caching devices Verizon still has to do work to carry that data from the cache device across their global network, to the recipient.

As far as they're concerned --- it's more important that they get paid to carry that data, than that the packets are are delivered with the lowest latency and greatest reliability.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

Cogent Verizon SUCKS. I am a customer of both plus some others around the US. We run data over the internet back and forth between our various offices, all have at least 100mbit service. Our LA and SF have Cogent. Our Chicago office has Verizon. We average 15kb/s (yes kilobits) LA-CH and about 300kb/s SF-CH. I've called both and both blame the other. This has been happening for months. In another twist. We have Nagios running out of band on a Verizon business DSL line from our ATL office to check external availability of various services at all of our US offices. The Nagios checks constantly time out when checking our Cogent connected offices. These are simple, is port 22 there or is this web page reachable etc run once every 2 minutes and 5 no responses in a row flags it as down.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

If those content providers' data isn't delivered, the people paying those tolls, as you describe, will quit paying them at some point.

Your analogy does describe what's going on, but omits the above fact, so it doesn't defend your position like you think it does.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

If that were the case, ISP should supply free connectivity to everyone, just like the postal service delivers to everyone who bothers to install a mail box. The way things are now, I pay my ISP for a given bandwidth, and I epect them to deliver all the packages I "order" from the internet without delay or loss, much like any server operator should expect their ISP to go to some reasonable effort to deliver all packages. There should be no interconnection fees, since both ends are getting paid by their respective customers.

Re:sell is the key word. Cogent not paying Verizon

[NormAtHome]

Ok, I'll bite; how is anyone supposed to "quit paying them at some point"? In my area you have two choices, either Comcast or Verizon. I know plenty of people who use one or the other (I use Comcast myself) and no one and I mean no one likes either one. People complain endlessly about the service, the support, the cost of equipment or what seems like annual price hikes of $1 here and a dollar there. And don't get me started on Verizon, they make it so that to use the channel lineup on your cable boxes you have no choice but to use their junky Actiontec router that is not only difficult for anyone but the most advanced user to program but also many of the advanced features you find on Linksys, Netgear or Asus routers don't work properly. But for the sake of this argument, what if in the end both Verizon and Comcast throttle Netflix, what are people going to do and who do you suggest they switch to? It's a very closed market with only those two choices (in this area anyway) for internet and TV service and it's just not realistic to say "just quit paying" because you say to Verizon "You're throttling my Netflix and the service is unusable so I'm not paying you this month" and Verizon will happily say "Ok, we're turning off your service due to non-payment".

I'm standing by what I said, the customer is paying for their internet service and it's wrong for any ISP to throttle a particular site that the customer wants to use.

Re:sell is the key word. Cogent not paying Verizon

[JesseMcDonald]

There appears to be some misunderstanding. Please point out where in my previous comment I said anything related to peering relationships.

Peering is a fairly simple arrangement. You have two networks with different owners. Each network needs to send some traffic to hosts reachable through the other. The amount of traffic is about the same in each direction. Rather than paying each other roughly equal amounts to carry each others' traffic, they implement a form of clearing and split the costs instead. The alternative, when the networks are not peers, is that the smaller or more remote network purchases access from someone more centrally connected, much as end-users purchase access from their ISPs.

Re:sell is the key word. Cogent not paying Verizon

[JesseMcDonald]

I never said the current system was completely rational. There is a lot of confusion over who pays for what in comparison to something like the postal service. Still, connectivity wouldn't be completely free even with all the actual traffic paid for by the sender. The ISP needs some way of getting packets to you, and there is a continuous cost for that infrastructure even if you never receive any data. The postal service takes advantage of public roads, which amount to a subscription service you can't opt out of. The Internet at large could be thought of as a network of toll roads, but I don't see that working for the "last mile"--local roads are more of a utility service.

At a minimum, the end-user would need to pay for their own equipment (the "mail box"), a public IP address, and the connection to their ISP (the equivalent of local roads). Maintaining a connection of sufficient capacity for the expected inbound data should be the recipient's responsibility; the costs related to getting a specific packet to the recipient should be up to the sender. A residential user could expect to pay whatever the fixed cost is for an idle 20 Mbps down / 5 Mbps up connection, for example, plus the "delivery fees" for however much upstream traffic they used during that period. They wouldn't pay anything based on how much they received; the sender pays that part.

Re:sell is the key word. Cogent not paying Verizon

[mysidia]

Tier1 ISP do not sell to residential. Verizon as a whole owns both Tier1 and Tier2 ISPs.

Huh? Sure there are Tier1 carriers that have some residential networks. Verizon is one of them.

The essential definition of a Tier1 carrier is that they buy transit from nobody, and a common trait is they provide free peering or access to nobody, except other Tier1s with specific requirement such as traffic ratios, and anyone else requesting peering is told to buy transit.

We're talking about the Tier2 ISP. Tier2 ISPs, which sell residential connections, always always try to get free bandwidth.

No... every provider tries to get free data transfer, provided they can do it without losing. Even Tier1's want free bandwidth, but only if they are not sacrificing more in revenue opportunities than they are getting.

According to Cogent, Verizon is quoting Cogent a price that is 10x the industry average for transit, and Cogent isn't even buying transit, they're buying a non-transit connection.

"The industry average" is not the definition of "fair price". You can price a service 50 times the industry average, and in some cases, that price is too low, for what is actually being delivered.

Verizon has a network that nobody else has and that nobody else can get or build, and when you have that: you get to charge a premium.

Netflix streaming video service likely represents higher usage and longer paths through Verizon's network than the industry average.

Re:sell is the key word. Cogent not paying Verizon

[Anonymous Coward]

I certainly don't know the first thing about them. But, it seems to me, Netflix isn't pushing any data anywhere. It's users pulling the data. It's not Netflix's fault if Comcast can't handle the user load. It's definitely a problem for Netflix, but all the content providers are going to have the same problem. This is ISP double dipping, period. I don't even care about peering agreements. Don't see how they change the equation of: I want data from Netflix, and my ISP cannot provide it. I want data from lots of places. So, I'm supposed to have my data providers pay for my internet connection? That does not make sense to me.

Instructions

[Anonymous Coward]

Look at the URL for this page. Remove the "-beta". Hit Enter. Rejoice.

RE: RE: "Lopside"?

[Anonymous Coward]

They do offer multiple POPs which will allow direct access as well as caching servers, but due to US ISPs also being in the business of selling similar services and a lack of competition, it makes more financial sense to hamper competition and drive profits to their services. IE Verizon has Redbox which they will gladly sell to consumers.

Netflix offers a colo/CDN bandwidth saver for ISPs

[Karl Cocknozzle]

Netflix has a program where they'll colocate some servers containing a content cache on a segment of the ISPs network so that their peering connections aren't getting beaten to death--why wouldn't these companies get involved in such a program other than as a means to squeeze more money from Netflix, their subscribers, or both.

Taking The Other Side This Time

[Bob9113]

Cogent has an agreement with Verizon to exchange traffic â" which works fine until the massive amount of traffic from Netflix makes it a lopsided arrangement. Verizon wants more money from Cogent, and one of their negotiating tactics is simply to stop upgrading their infrastructure so that service degrades.

I'm a pretty rabid net neutrality guy, and as a disclaimer I've never worked for ISPs or long haul providers so I may have my head up my ass. But in this case, I'm tending to fall on the Verizon side of the argument. Peering is supposed to be a two-way street, and if it isn't, the peering is subject to negotiation. It may not be fair for Cogent to take the relatively easy money from giant, centralized Netflix outbound data handling and leave Verizon doing the more costly work of parcelling that data out to millions of endpoints without giving Verizon a piece of the action.

The problem of Verizon having a non-competitive (or reduced competition) supply of consumers to negotiate with is still there, and the outcome ultimately will be Netflix paying more for their connectivity, but at least the payment is being coupled to the right transaction.

RE: Taking The Other Side This Time

[Anonymous Coward]

Quick question. How can you have equal traffic when you have a network on one side that is basically consumers (Verizon) and a network on the other side that is basically suppliers (Cogent)? I'm sure all those DSL customers are serving up a lot of content. Considering Cogent's notorious supply of Porn and now Netflix, it should all just balance out.

Re:Taking The Other Side This Time

[Todd Knarr]

My response would be that Netflix and Cogent aren't just sending unsolicited traffic to Verizon. It's Verizon's customers who're requesting the traffic by asking to stream video from Netflix. And Verizon's being paid by those customers to deliver the traffic from Netflix. That's what that bill that Verizon sends them each month is for, after all, to pay for that customer's connectivity. If their customers are demanding more traffic than Verizon's connection with Cogent can handle, why is the fact that Verizon isn't charging their customers what the service they're demanding costs Cogent's or Netflix's problem?

Re:Taking The Other Side This Time

[Bob9113]

My response would be that Netflix and Cogent aren't just sending unsolicited traffic to Verizon. It's Verizon's customers who're requesting the traffic by asking to stream video from Netflix.

That is a really strong counterpoint. Still, though, handling Netflix's bulk traffic really is cheaper than parcelling out to all the individual customers. But then I guess Verizon should just be charging their customers more if providing the service they want is expensive. Verizon isn't going after Netflix because it is impossible to bill the customer fairly. Verizon is going after Netflix because Netflix has deep pockets and Verizon wants an additional revenue stream on the back of their limited competition market, regardless of whether it is how a theoretical perfectly efficient free market would solve it. [feigned shock]Why, they're not capitalist idealists at all, this is just a money making thing![/feigned shock]

Re:Taking The Other Side This Time

[Todd Knarr]

Why would Verizon's routing be any more expensive than Cogent's? If anything Cogent needs the more expensive routers. Router cost is a function of the bandwidth they need to handle and the size of the routing table they need to accommodate. Cogent's routers need larger routing tables because they need to handle the random agglomeration of ASes that is the Internet while Verizon's only need to handle Verizon's network which Verizon can arrange to minimize routing complexity. And Cogent's routers need to handle the entirety of Netflix's traffic all the way out to the edge while Verizon's have the traffic split early so most of them only have to handle that fraction bound for destinations "below" them. Verizon needs more routers, but the number's dependent on how they've organized their network and they'd need that many routers even if Netflix's traffic weren't there.

As for going after Netflix because they have deep pockets, my annoyance isn't about that. It's that Verizon and the rest won't admit that that's why they're doing it. They want to hem and haw and evade the question and make it out like Netflix is arbitrarily shoving this traffic at them because Netflix wants them to carry it, when in reality it's their own customers who requested the traffic and want them to carry it. I don't feel obliged to put up with the deception ISPs like Verizon are trying to pull off.

Netflix vs Google

[careysb]

It seems that ISP's are so concerned with Netflix's bandwidth suck that they try to get away with throttling. What about Google? Supposedly Google's web crawlers account for the largest single chunk of Internet bandwidth. (Ok, educate me)
--
Sent from my IBM 360 mainframe

Re:Netflix vs Google

[Kjella]

It seems that ISP's are so concerned with Netflix's bandwidth suck that they try to get away with throttling. What about Google? Supposedly Google's web crawlers account for the largest single chunk of Internet bandwidth. (Ok, educate me)

Well, if you divide everything else into very small chunks I guess that's true. However, if you exclude streaming media over http like YouTube most statistics I find shows web traffic accounts for 15-20% of total Internet traffic. The largest chunks from a very high-level perspective seems to be 1) streaming media, 2) bittorrent/p2p and 3) everything else.

Fair is fair

[Anonymous Coward]

So the ISPs want more money, that's fair as long as they pay the content providers for all the content they make available that provides the ISPs with so many customers. That's fair right?

Re: sell is the key word. Cogent not paying Verizo

[Anonymous Coward]

In the isp consumer space we pay for what we download.

So Verizon can pay for what it downloads thanks.

Supply and Demand

[recharged95]

simply to stop upgrading their infrastructure so that service degrades.

And who's going to pay for upgrading their infrastructure? Verizon might as well pass the costs upstream. And with net neutrality, we're going to see more of this.

CDN's?

[DoofusOfDeath]

Does anyone know if/why Netflix doesn't just have CDN servers located directly on Verizon's network?

It seems a little moronic to stream stuff from far away, when Netflix's streaming content is a large but basically fixed quantity of data.

Re:CDN's?

[Anonymous Coward]

Does anyone know if/why Netflix doesn't just have CDN servers located directly on Verizon's network?

Because Verizon are whining whores, and if they allowed Netflix's CDN jobs onto their network, they'd have less tantrums to throw in front of their bribed members of Congress.

Re:CDN's?

[bill_mcgonigle]

Does anyone know if/why Netflix doesn't just have CDN servers located directly on Verizon's network?

Verizon is a 'cable tv' provider through FiOS. They have incentive for Netflix to fail.

Re:CDN's?

[sjames]

Because Verizon refused their offer.

Re:CDN's?

[Anonymous Coward]

If they don't, it would seem to be Verizon's decision since Netflix generally offers free CDN ("Open Connect") boxes to ISPs. This article makes that allegation: "Verizon, Comcast, Time Warner Cable, and AT&T all want compensation from Netflix to switch to the Open Connect system" http://www.digitaltrends.com/home-theater/netflix-verizon-battle-future-internet-tv/

Re:Netflix offers a colo/CDN bandwidth saver for I

[sexconker]

Netflix has a program where they'll colocate some servers containing a content cache on a segment of the ISPs network so that their peering connections aren't getting beaten to death--why wouldn't these companies get involved in such a program other than as a means to squeeze more money from Netflix, their subscribers, or both.

Because you have to agree to Netflix's terms to host those things.
Everything from physical access requirements to the ol' "By the way we may host other, non-Netflix content on these things in the future, and we'll charge people for the privilege, but you'll still have to treat it as Netflix data and not expect any money for carrying it on your network".

Netflix muscled their way into favorable agreements (both with and without those storage boxes at the ISPs) when they trotted out Super HD. Now that they have a lot of those agreements in place, Netflix opened it up so (just about) everyone gets Super HD, and they aren't making any noise over it anymore.

Remember colos offering "Cogent-free bandwidth"?

[ebunga]

Yeah, that's still a thing, for a reason. They should seriously go the Akamai route... that way access ISPs can tout a "fully-Netflixed network!"

Re:Obamaville. We haz it.

[Anonymous Coward]

Depressions and economic collapse is a pretty easy thing to deal with

Now that I'm not tied to a job, and obamacare takes care of my medical needs, I'm free to pursue my passions while collecting welfare and voting a straight democratic ticket. Everyone in America should follow my example. It would be utopia!

The real problem is our growing dependency on the

[Anonymous Coward]

It doesn't matter how fast a file downloads if you get to keep it and play it locally.

The cloud!

[Anonymous Coward]

What about Dropbox? What about OneDrive? What about Office365? What about YouTube? Aren't all of these things being used more often by more people? Why are the ISPs blaming Netflix? Why don't they upgrade their shit?

Perfect way to even the odds

[Anonymous Coward]

Netflix should just set their player to echo back the feed from the client device to Netflix. They could tell cogent to null route the data and just ignore it. That would "even up" the ratio closer to 1:1, while causing MASSIVE upload bandwidth for the ISP. That would probably get them to change their tune pretty quick.

What's stopping Netflix

[MoronGames]

From doing some kind of opt-in peering arrangement, where movies were cached locally for a while after being watched on some devices with the storage capability to do so? While they're cached, they can help stream to peers on the same network along with Netflix's servers, in some type of arrangement similar to bittorrent. That way ISPs could degrade Netflix however they wanted but end user performance would still be acceptable in some cases.

Re:What's stopping Netflix

[Anonymous Coward]

Netflix already offers ISPs free dedicated caching appliances through their "Open Connect" program.

Re: "Lopside"?

[sjames]

Simple solution, Netflix should alter their client so it always echos the data back. That should balance the transfer and make Verizon very happy.

Yet another peering fight

[TFoo]

Peering fights are as old as the Internet itself. Just economics. Nothing to see here, move along. Vote with your wallets and pick ISPs that do a better job with their peering arrangements.

Re:Yet another peering fight

[petermgreen]

Vote with your wallets and pick ISPs that do a better job with their peering arrangements.

The problem is here we have a case of a teir 1 provider vertically integrated with an access provider that carries local monopolies or effective monopolies in some areas. AIUI in the US it has been ruled that carriers don't have to share fiber or cable networks so the only networks that have real ISP competition are the crappy old DSL ones (IIRC in some areas these are even being actively removed to force people onto the single ISP fiber networks).

"Demand more money from content distributors"

[nurb432]

And content consumers ( us ).

Can confirm

[Streetlight]

...with little to no mention of the ISPs.

Check out the fairly active thread over at DSL reports vis a vie Comcast subscriber's problems with Netflix including some non answers by a Comcast employee: http://www.dslreports.com/foru...

Verizon Bussines Fios here NY

[Anonymous Coward]

And you are right, Friday night streaming on NetFlix lag? good luck, on regular 1.5 h movie, it will usually freeze 10-20 times and on standard resolution few times. Line is not loaded and ping to other routes are fine. There is definitely something going around.

Used To Have That Problem With Sprint

[Greyfox]

I used to have Sprint between me and an online game I was playing at the time. If it ever ground to a halt, I could be pretty sure Sprint was to blame. A quick traceroute would usually show one of their routers crapping packets on the floor. If we had actual competition, you could shop around for an ISP that sucks less. Since Netflix and their customers can pretty much be sure that's not going to happen, their best bet is to just increase their buffers and expect there to be wait times. They could just put all their movies on some sort of... car full of magtapes... and deliver them directly to the customer that way. Sure the latency would be higher, but it's always been difficult to beat the bandwidth of a car full of magtapes!

How 'bout BitTorrent?

[guruevi]

If they got such problem with peering, maybe instead of streaming, they should keep the stuff within the networks. How many people aren't simultaneously streaming the 'popular' stuff. They're already maintaining a cache of ~512MB-2GB and Silverlight uses another 1GB of RAM. They can simply use h264/VP9 and give people that want to run 'supernodes' on their connection a discount.

Peer to Peer and end the problem

[Bruha]

If Netflix leveraged all the devices out there to store pieces of all the movies then they could get content inside the networks. Sadly because there's no unlimited anymore it would be difficult to deploy this model anymore.

Not Just Netflix

[organgtool]

I was occasionally having problems accessing YouTube videos over my FIOS connection. I had 75 Mbps down, but YouTube videos sometimes failed to play after about 5% of the video had downloaded or they would only download at 144p (it reminded me of RealVideo from the 90s). I read reports from other users that showed up to 50% packet loss over FIOS to YouTube servers. Those users also reported that if they VPNed into their work network, they were able to view YouTube videos just fine. That was the straw that broke the camel's back and I switched to Comcast and cut the television cord. While their service isn't quite as good, I got a really good deal. The only thing that annoys me is that when I VPN into work, my connection goes from 30 Mbps to 750 Kbps. Since I primarily use the command line, that is tolerable, but it does piss me off that this wasn't well-communicated to me before I signed up.

I couldn't disagree more.

[Anonymous Coward]

It's called a conflict of interest. In legal terms, if a judge or a lawyer have a conflict of interest in a case, they *HAVE TO* recuse themselves from the case.

In the Internet world, if a bandwidth provider is also a content provider, I find that to be a clear conflict of interest, with clear and present danger to the internet.

Time Warner / AT&T / Verizon should not be allowed to buy into or own in any way any form of content provisioning, period.

If they want to provide content, they need to get out of the bandwidth business.

The best thing that could happen would be for all of the fiber / copper / cables / radio waves used for internet connectivity to be pulled into common carrier status. Then the *providers* and ISPs pay into a public pool to keep increasing bandwidth and connectivity, while limiting what they can charge to get onto the internet because all of the mom and pop ISPs can now connect for the same low low price as the big boys do - more competition equals lower prices for everyone - fewer profits for assholes who got us into this mess to begin with.

Same should be done for cellular networks - then we'll see cell phone plans that are like $50.00 a month (for an entire family, and all of their devices), including all taxes, fees, blah blah blah for unlimited everything at maximum data rates 24x7.

Long gone will be the day where they give away voice and text (which uses 10 x the bandwidth of a facebook / netflix app) while charging 50 bucks a gig of data.

I cannot wait to see these big money laundering companies eating shoe-leather with their fat profit margins cut to the bone.

eh - wrong

[Anonymous Coward]

Once we know for a fact that the peers cannot traffic shape, it will be simple to map out the congested areas and point out the culprits who are overselling their capacity.

A few quick law changes and the companies that are overselling will *HAVE TO* increase capacity to be able to handle 150% of what they are selling, increasing capacity as they sell.

Simple metrics posted to a single FCC website will keep the providers honest, or if they try to game the system, they'll be outed by the common carrier reporting.

Quick, simple and overall will stop the profit mongering of the peers. If they refuse to increase their bandwidth, they lose the license to carry customers on the common internet infrastructure after the FCC declares their network common carrier.

Don't forget the up and coming netflix competitor

[Anonymous Coward]

Verizon owns that little bit of providership, in their attempt to wage content war.

it CAN cost a lot to move data cross country. who

[raymorris]

> it can't possibly cost them anything.

A server in New York wants to stream data to a user in California. Someone has to pay for a nationwide fiber network to move the traffic. Both Cogent and Verizon have presence in New York and California. Who carries the packets from New York to California? Very often, that's what peering disputes come down to.

  In general, neither party wants to carry the traffic across the country. Cogent wants to instantly hand the traffic to Verizon right there in the same building where they got the traffic from Netflix, so they don't have to carry the packets more than 100 meters. That's reasonable to them - they are delivering the packets to the company they are addressed to. Verizon would want to receive California packets in California. When Cogent is charging Netflix for transit, it's reasonable for Verizon to ask Cogent to provide that transit to California. Both have reasonable positions. They'll negotiate a mutually acceptable arrangement after first staking out their starting positions.

Firsthop is a natural monpoly not the backbone

[Anonymous Coward]

Cogent has competition, Verizon doesn't

Use VPN

[Anonymous Coward]

I have been using VPN to overcome this problem, based on this article http://thevpn.guru/netflix-streaming-problems-verizon/ if you use VPN or SmartDNS your traffic bounces behind the Verizon control servers and it wont get affected by the BS Verizon is pulling.

Customers are not paying for Cogent traffic

[Anonymous Coward]

Verizon's end-customers pay for being hooked up with the Verizon network with a certain bandwidth. They don't pay for receiving Cogent traffic at this bandwidth. So the end customers are not paying for it.
If Cogent pushes more traffic into Verizon, and as a consequence Verizon has to upgrade their connections, then somebody has to pay for it. The costs should at the very least be equally shared by both parties. It's interesting though who of the two has the bigger economical interest.

Cogent - wanting to satisfy netflix, by enabling them to get more customers
Or
Verizon - wanting to keep and acquire more (netflix-addicted) end-customers

not so simple. customers pay for traffic not servi

[zerostar]

Verizons end-customers are paying for being hooked up with the Verizon network at a certain bandwidth. They are NOT paying for having Cogent traffic routed to them at all time at this bandwidth at all time. That would be a quality-of-service agreement. If Verizon needs to upgrade their links because of traffic coming from Cogent, then Cogent should pay a share of the upgrade cost. It's interesting though, who of this two players has the greater need to keep the end-customers happy: Cogent trying to please Netflix OR Verizon competing with other ISPs also connected with Cogent

Why no netflix caches in verizon?

[Anonymous Coward]

Well, when Cogent is involved, the packet loss is _always_ their fault. They're the trolls of Internet Transport (ISP of ISPs).

But the poor quality of service for Netflix users in Verizon is Verizon's fault.

Netflix has a very powerful caching infrastructure, which Verizon can host in several places on their network. They're far more than big enough to warrant it, in fact they are big enough to warrant the full cache cluster (which caches 99,9999% of the netflix content). And it wouldn't even cost Verizon a lot to deploy those, since the cache hardware itself as well as all of its maintenance is covered by Netflix.

That would reduce Verizon's Netflix peering traffic with Cogent to very manageable levels (something like one terabyte/day, and the caches are smart enough to refresh at the low-demand hours).

And it *works*. How else do you think we deal with Netflix overseas?

Here's the "server p0rn" and details on the Netflix CDN cache nodes you can host at your ISP:
https://signup.netflix.com/openconnect

A carrier should not be going through another carr

[horeton]

As an ISP we added a Netflix peer via "Netflix Open Connect" last week and are offloading 4Gbit per/sec at peak usage via BGP directly to Netflix rather than wasting our dia bandwidth to Cogent, Level3, or other Backbone providers. By meeting them at IXPs like the TIE and NOTA a few of many, ISPs can directly offload traffic to most major application providers for free like Netflix. Google, and Facebook just to name a few. Why Verizon wastes their DIA bandwidth for Netflix is idiotic. Verizon owns Terremark which includes NOTA. They could be directly peering with Netflix themselves. I think it's more likely they have issues keeping up with it from a cellular perspective through their own infrastructure. Going through Cogent instead of directly to Netflix defeats the point of IXPs like NOTA.

ISP co-op

[Anonymous Coward]

Just like energy co-ops, why can't communities band together to create ISP co-ops? What are the hurdles?

Re:ISP co-op

[PPH]

Butt hurt on the part of the telecom/cable companies.

All that underground wind

[Ultra64]

must be knocking them out.

CDN/Proxy servers?

[lenne]

Netflix might be having 20000 TV-episodes and 3000 movies, but they are not evenly viewed. I bet that a caching box with even a mere 1TB disk could cover 90% of the usage.
If 10000 viewers want to see the latest House of Cards, there is no need for sending 10000 streams from one ISP to another.

Heck, even some sort of bittorrent could be used, if each viewer would allocate say 50GB for shared content.

I have a thin adsl from one ISP and a 4G to another, and a traceroute to netflix leads to two different servers, each in my country, Denmark. The adsl is even only 3 hops away.

BTW: Why do I have to type <br> to have line breaks? Why not just enter enter?

Re:CDN/Proxy servers?

[Steve Hamlin]

Netflix has a local caching solution for ISPs: Netflix OpenConnect Appliance, and it's over 100TB of content (refreshing 5TB per night). 4U chassis, Intel Sandy bridge CPU, 32GB RAM, 1TB solid state, 100TB+ spinning (36x3TB), FreeBSD, nginx, 2x10Gbps fiber

Overview

In-depth Deployment Guide

Galacial pace is pretty fast these days

[arvindsg]

With Cogent and Verizon fighting, the upgrades are happening at a glacial pace

http://science-beta.slashdot.o...

Re: sell is the key word. Cogent not paying Verizo

[r_jensen11]

Isn't Netflix using Amazon's hosting solutions, though? If so, then the decision to use Cogent would come down to Bezos & Co.

Capital costs + regulation

[bussdriver]

Reality is that if you get around the capital costs, the power of the monopolies in the area is flexed. Then regulations become a problem; the regulators themselves are all rooting for you (I've known some) but when the pressure comes in... forget it. Then you have a fight on your hands. They'll go above you and get the state to pass a law against any town doing their own ISP. Or they'll create regulations that make it costly to enter the market or they'll spread around lies to make life miserable. If you get going they'll find every possible misstep or anything that can be misconstrued and make sure every official hears about it so much they'll want to act just to end the harassment.

It can be done but it is not easy; nowhere near as easy as it should be and it would be if the public were competent citizens and supported decent officials at positions of real power.

Best thing one can do is try to build up underground momentum so that when the beasts have awoken you have a chance.

Mod parent up.

[bussdriver]

Mod parent up.
I think the best fitting term is FOOL, not moron.

Re: sell is the key word. Cogent not paying Verizo

[mysidia]

Isn't Netflix using Amazon's hosting solutions, though? If so, then the decision to use Cogent would come down to Bezos & Co.

We know Netflix makes extensive use of the Amazon cloud.

It doesn't really matter though --- Netflix is ultimately responsible.

Do you think it matters if Netflix opens the ticket with Cogent, or if Netflix opens the ticket with Amazon, who in turn opens tickets with Cogent?

Of course... Verizon customers are also in a position to open tickets. If enough Verizon customers open tickets, then I suspect, they will have to do something.

But because of Verizon's massive size, you either need a bunch of high dollar-value Verizon clients opening tickets, OR tens of millions of residential users, with enough intelligence and research to get past level 1 support screeners and get a "real" ticket open.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Buffering

[jaq1an]

This explains all the buffering I've been experiencing with Netflix on UPC here in Ireland

120%?

[Vrtigo1]

They must be using some kind of new-fangled routers because the ports on the ones I have only go to 100%. Bad wording there.

Re:Netflix offers a colo/CDN bandwidth saver for I

[Karl Cocknozzle]

Yes, that's what co-location is: Somebody else pays you for physical access to your site for long-term deployment of equipment. So the "physical access" requirement isn't exactly some sort of "evil scheme" netflix invented to screw over Comcast.

This part is nonsensical:

Everything from physical access requirements to the ol' "By the way we may host other, non-Netflix content on these things in the future, and we'll charge people for the privilege, but you'll still have to treat it as Netflix data and not expect any money for carrying it on your network".

1) They already charge people to access their service now, and in a way that apparently harms Comcast/ISPs in general, so we have zero difference from the status quo--the ISPs already have accepted this as "normal" and I don't see how they can ever change that without essentially erasing the entire Internet and starting over.

2) If Netflix hosts other people's data on those systems... so what? It's to Comcast's benefit--the more content that users stream that way (as opposed to over their expensive peering links) the happier their customers will be.

3) Comcast already gets money to carry all of this data--they get it from their subscribers. They're caterwauling for a double-dip opportunity--the right to bill not just for bandwidth to users, but for the same bandwidth again to companies providing content.

Re:Netflix offers a colo/CDN bandwidth saver for I

[sexconker]

Yes, that's what co-location is: Somebody else pays you for physical access to your site for long-term deployment of equipment. So the "physical access" requirement isn't exactly some sort of "evil scheme" netflix invented to screw over Comcast.

This part is nonsensical:

Everything from physical access requirements to the ol' "By the way we may host other, non-Netflix content on these things in the future, and we'll charge people for the privilege, but you'll still have to treat it as Netflix data and not expect any money for carrying it on your network".

1) They already charge people to access their service now, and in a way that apparently harms Comcast/ISPs in general, so we have zero difference from the status quo--the ISPs already have accepted this as "normal" and I don't see how they can ever change that without essentially erasing the entire Internet and starting over.

2) If Netflix hosts other people's data on those systems... so what? It's to Comcast's benefit--the more content that users stream that way (as opposed to over their expensive peering links) the happier their customers will be.

3) Comcast already gets money to carry all of this data--they get it from their subscribers. They're caterwauling for a double-dip opportunity--the right to bill not just for bandwidth to users, but for the same bandwidth again to companies providing content.

You don't get it.

Netflix muscled these agreements and boxes onto ISPs by shouting "THIS ISP DOESN"T SUPPORT SUPER HD!!!!" from the rooftops. "SUPER HD" is just a decent-quality 8 Mbps stream, and it was never about ISPs supporting the stream, it was about Netflix granting them access to it. This required ISPs to agree to one-sided contracts in Netflix's favor. If ISPs didn't capitulate, they were branded as not supporting "SUPER HD" and users were actively encouraged by Netflix to bitch at the ISPs. Then once Netflix felt like they had enough coverage with these agreements and boxes, they opened up "SUPER HD" to (almost) every ISP.

The boxes and agreements are favorable to Netflix, not the ISPs. The ISPs would normally be paid to carry that traffic (or be able to account for it when discussing peering agreements). Now they pay to store, power, and deliver Netflix's content, and it's removed from the table when discussing peering agreements. On top of that Netflix can store non-Netflix content on those boxes, and ISPs would be unable to treat that content and its associated bandwidth/power/space/etc. costs differently than the rest of the content on those boxes.

Netflix weaseled their way into favorable shit and pissed of a lot of PHBs. Netflix did the same thing with their by-mail service. Once faced with impending shipping cost increases, they actively decimated the by-mail service.
Once those contracts are up the bandwidth wars will start. They've got nowhere to go this time, so they're going to do all they can to bolster the importance of their content in the 3-5 years (my guess) those contracts are good for, which means diversifying services to users as well as allowing 3rd-party, non-Netflix content to fly under their banner.

Expect Netflix-branded competitors to shit like Youtube, Dropbox, and Twitch by the end of 2016.

Re: "Lopside"?

[Anonymous Coward]

Not back, out. Make the client part of the CDN.