Slashdot Mirror

A CDN inside your network SAVES you money by allowing you to use bandwidth that would have been spent streaming the content from the source to your customer on other content. In the case of Netflix it would have saved you A LOT of money. So yes I don't think Netflix was unreasonable in thinking "Hey we're asking to do this FOR you and our mutual customers for free and you want to CHARGE us to save you money and give your customers a better experience?" The ISP's in question have an extreme conflict of interest because they want you to buy their inflated VOD services instead of using Netflix. They also oversell their service such that if all their customers were to use the bandwidth they pay for at the same time it would cripple their network and even if it didn't their peering connection to the backbone is already saturated with less than half the traffic they sold. These blogs over at Level 3 are very enlightening.

A CDN inside your network SAVES you money by allowing you to use bandwidth that would have been spent streaming the content from the source to your customer on other content. In the case of Netflix it would have saved you A LOT of money. So yes I don't think Netflix was unreasonable in thinking "Hey we're asking to do this FOR you and our mutual customers for free and you want to CHARGE us to save you money and give your customers a better experience?" The ISP's in question have an extreme conflict of interest because they want you to buy their inflated VOD services instead of using Netflix. They also oversell their service such that if all their customers were to use the bandwidth they pay for at the same time it would cripple their network and even if it didn't their peering connection to the backbone is already saturated with less than half the traffic they sold. These blogs over at Level 3 are very enlightening.

I suspect they are all fast enough but your ISP may not like the VOD competition.

I suspect they are all fast enough but your ISP may not like the VOD competition.

I've posted this before but it's still probably relevant. Basically the US ISP's sell a huge pipe to their customer but the pipe to the internet is saturated 24/7 because they have refused to upgrade their back end.

I've posted this before but it's still probably relevant. Basically the US ISP's sell a huge pipe to their customer but the pipe to the internet is saturated 24/7 because they have refused to upgrade their back end.

This is a little dated so things might have changed but there is evidence their current customers can't get what they're paying for.

It's still hard to tell what's going on here because both parties only seem interested in depicting the other as evil.

Level3 blames Verizon. There's some nasty stuff going on there if what they say is true. But none of it uniquely affects Netflix, it affects all Level3 customers. This is the kind of network management that the FCC has declined to intervene in.

They claimed that Verizon literally unplugged half the connections between two networks that are only half congested.

Presumably, if they added more links, the connection to that Netflix server (among other Level3 customers) would go up, but it couldn't do any better than double. (Bitrates might more than double now that there's no packet retransmissions, but it probably wouldn't increase an order of magnitude, which is what the VPN user claims.)

If a VPN really is faster, then Netflix clearly has access to unused capacity via other routes that they're not providing to customers. That is, the VPN is just doing the routing that Netflix isn't doing.

Either that, or the VPN customer is accessing the same Netflix server, which would make the VPN story is a lie, because VPN, of course, doesn't let you blast through network congestion.

Regardless, none of the accusations claim Net Neutrality violations.

> Verizon decided they wanted a piece of the action, and degraded service. It's been shown that it was intentional, and not the result of insufficient hardware.

Level3 and Verizon both disagree with you. They both agree that the 7 interconnect routers became saturated as the Netflix traffic increased, and both have published diagrams explaining it for laymen. Links are below. Anyone,who knows how to use traceroute can confirm that fact. The only question is whether Netflix or Level3 will pay to upgrade that connection.

http://publicpolicy.verizon.co...

http://blog.level3.com/open-in...

Whether Netflix is using Verizon as an ISP or if they are peers is of course the crux of their dispute. Looking at the definition of "peer" in the dictionary, they don't seem to meet the definition.

I'm not sure if you are unfamiliar with the situation or just have a portfolio full of ComCast stock but There is some very insightful material over at Level 3 that paints a clear picture of just who is ganging up on who. And Netflix isn't the only Internet site that suffers from these ISP's greed. They just happen to be the largest. And as far as their CDN how is relieving the congestion on your network not a payment in and of itself? Netflix traffic from the backbone would decrease 90% once they are allowed to put their CDN inside your network leaving you free to use it for other traffic. If Netflix is causing the congestion then allowing them to put their CDN in is a win/win but ISPs want Netflix to lose so their on demand service can pick up those customers. If you truly want the truth then read the articles in the link.

You're a lying or dumb son of a bitch. Check out Level3's response to this bullshit, with easy for dumbasses like you to understand diagrams. There's a rack of routers sitting there on both sides and Verison refuses to add a few more for a few thousand dollars (chump change) or to even run wires to the open ports between the two router racks. Level3 would even offered to pay for the hardware AND the installation. In exchanges like these it's quite common that when there's some congestion you just add more hardware. Verison refuses to connect more hardware unless you're anyone but a carrier of Netflix bandwidth.

Fucking idiot. It really is that simple. "Hey, there's a choke point here we could just run a few more switches" "No, fuck you, I want Netflicks to pay us a fortune for what should just be the normal business operations that ISPs do. Not to mention we can then complain about congestion so Verison's customers can expect a rate hike for every bloody BAUD we allow to trickle in."

You're over thinking it. Follow the gods damned money.

This is exactly what Level 3 claims.

http://blog.level3.com/global-...

The above uses the Los Angeles interconnect as an example. Four 10-GB ethernet connections between Level 3 and Verizon and they are saturated.

Level 3 wants to add additional 10-GB connections (and even offered to buy the router cards, cable and do the install :-) ). Verizon refuses.

"So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion."

It *could* be that Verizon realizes that currently, as configured, the Verizon network is running nicely (as the nice diagram shows). And that doubling the amount of traffic by doubling the interconnect to Level 3 (and thereby satisfying their customers demand for Netflix VOD) would cause a meltdown.

But in any respect, Verizon is selling a service and not providing it. Netflix is paying for transit TO Verizon and Verizon's customers are paying for transit (to Verizon) for transit from Level 3. Verizon needs to charge their customes what they need to provide the service.

Verizon was called-out for refusing to do it, simultaneously refusing to increase their L3 peering (for free!) to compensate, and then claiming Netflix was the cause of degraded Netflix streaming: http://blog.level3.com/global-...

Ps, http://blog.level3.com/global-...

The fix was actually a couple of network cards + cable, which Level 3 offered to sort out.

> It may get Level3 to drop the "symetric vs asymetric" part of their argument, but it won't change the amount of traffic going from the ISP to back-bone provider.

FWIW, that is only sort-of L3's argument.

They have this concept of "bit-miles" which wasn't obvious to me at first. The idea is that instead of pricing by the bit that crosses the exchange point the metric should be bits multiplied by the distance the bit moves. Here's a cut-n-paste from a recent L3 posting:

(sorry the apostrophes are missing, they are using a 16-bit character for a superscript 1 as apostrophes and slashdot eats that)

This is fiber. I don't have Verizon myself, but in general everything people complain about in regards to ISPs goes away once you're fiber. They'd have to have some pretty serious congestion issues for FiOS to start having trouble.

It matters not how fast your download speed from your ISP is if said ISP's connection to the content you are requesting isn't able to deliver it.

Along that same line though, I've no idea why they had asymmetric on fiber to begin with. The point to ADSL (Asymmetric DSL) has to do with crosstalk on the copper lines in the DSLAM. This isn't an issue, at all, for Fiber. So it makes little sense to have asymmetric fiber service other than for marketing purposes.

Consumer ISP's are all about getting content to you. They don't want you throwing up a server at your house to stream data to the ethers. They want you to stream media from them. So much so most have U NO RUN SERVER clauses in their TOS. An asynchronous connection allows them to advertise higher bandwidth "download" speeds and keeps those nasty server runners with paltry pipes to get their filth up to the internet.

http://blog.level3.com/global-...

"A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.

That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.

Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers."

The link to the LEVEL3 Blog post has been blocked...but modifying the link a bit pulls it back up again. Blog post reproduced below (with relevant image here)

VerizonÃ(TM)s Accidental Mea Culpa
Mark Taylor / 23 hours ago

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns NetflixÃ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by VerizonÃ(TM)s own paying broadband consumers.

His explanation for NetflixÃ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know itÃ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But hereÃ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as VerizonÃ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatÃ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network Ã" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3Ã(TM)s network interconnects with VerizonÃ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So letÃ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of th

The link to the LEVEL3 Blog post has been blocked...but modifying the link a bit pulls it back up again. Blog post reproduced below (with relevant image here)

VerizonÃ(TM)s Accidental Mea Culpa
Mark Taylor / 23 hours ago

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns NetflixÃ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by VerizonÃ(TM)s own paying broadband consumers.

His explanation for NetflixÃ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know itÃ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But hereÃ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as VerizonÃ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatÃ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network Ã" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3Ã(TM)s network interconnects with VerizonÃ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So letÃ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of th

Then how do you explain the Level 3 data? The major ISPs got caught red-handed throttling Netflix traffic until the extortion was paid (Comcast in this case). Days later everything was running smooth as a baby's ass. So how can you seriously make an argument that all the blame lies on Netflix' shoulders when the ISP's customers are paying for the bandwidth to receive the content?

Let's say there was a burden. If the ISPs aren't willing to upgrade their networks then their business model is the problem, not how the internet works. And according to the data it looks like the ISPs infrastructure isn't that bad off anyway, they were simple messing with the traffic to extort payments from content providers.

TL;DR: WTF are you talking about?

http://blog.level3.com/global-...

Are you seriously suggesting that congested ports -> Netflix pays for their own direct interconnects -> uncongested ports somehow proves that Netflix was being throttled? Because, frankly, it suggests the opposite to me (i.e. moving lots of traffic to a different interconnect freed up capacity on the original). Your own link shows the general congestion: see this graph.

You can, quite easily, make the argument that Comcast (or Verizon, or whoever the peer in question is) let that situation fester until it resulted in their "winning" a new customer (Netflix) from level3, but certainly not that their traffic was being treated differently from anyone else's.

This is standard in these contract negotiations. I work with them from time to time.
"Pay up or we're shutting you off!"
well... we're working on some stuff...
*throttle to 20%*
woaaaa there guys, we were just kidding, where do we sign?

You might not like it, but that's HOW it's done in the industry. The ISPs treat each other in the same way. I'd agree that the FCC should regulate this market better. Software vendors do the same sort of things.

Then how do you explain the Level 3 data? The major ISPs got caught red-handed throttling Netflix traffic until the extortion was paid (Comcast in this case). Days later everything was running smooth as a baby's ass. So how can you seriously make an argument that all the blame lies on Netflix' shoulders when the ISP's customers are paying for the bandwidth to receive the content?

Let's say there was a burden. If the ISPs aren't willing to upgrade their networks then their business model is the problem, not how the internet works. And according to the data it looks like the ISPs infrastructure isn't that bad off anyway, they were simple messing with the traffic to extort payments from content providers.

TL;DR: WTF are you talking about?

http://blog.level3.com/global-...

Are you seriously suggesting that congested ports -> Netflix pays for their own direct interconnects -> uncongested ports somehow proves that Netflix was being throttled? Because, frankly, it suggests the opposite to me (i.e. moving lots of traffic to a different interconnect freed up capacity on the original). Your own link shows the general congestion: see this graph.

You can, quite easily, make the argument that Comcast (or Verizon, or whoever the peer in question is) let that situation fester until it resulted in their "winning" a new customer (Netflix) from level3, but certainly not that their traffic was being treated differently from anyone else's.

This is standard in these contract negotiations. I work with them from time to time.
"Pay up or we're shutting you off!"
well... we're working on some stuff...
*throttle to 20%*
woaaaa there guys, we were just kidding, where do we sign?

You might not like it, but that's HOW it's done in the industry. The ISPs treat each other in the same way. I'd agree that the FCC should regulate this market better. Software vendors do the same sort of things.

Then how do you explain the Level 3 data? The major ISPs got caught red-handed throttling Netflix traffic until the extortion was paid (Comcast in this case). Days later everything was running smooth as a baby's ass. So how can you seriously make an argument that all the blame lies on Netflix' shoulders when the ISP's customers are paying for the bandwidth to receive the content?

Let's say there was a burden. If the ISPs aren't willing to upgrade their networks then their business model is the problem, not how the internet works. And according to the data it looks like the ISPs infrastructure isn't that bad off anyway, they were simple messing with the traffic to extort payments from content providers.

TL;DR: WTF are you talking about?

http://blog.level3.com/global-...

Are you seriously suggesting that congested ports -> Netflix pays for their own direct interconnects -> uncongested ports somehow proves that Netflix was being throttled? Because, frankly, it suggests the opposite to me (i.e. moving lots of traffic to a different interconnect freed up capacity on the original). Your own link shows the general congestion: see this graph.

You can, quite easily, make the argument that Comcast (or Verizon, or whoever the peer in question is) let that situation fester until it resulted in their "winning" a new customer (Netflix) from level3, but certainly not that their traffic was being treated differently from anyone else's.

Then how do you explain the Level 3 data? The major ISPs got caught red-handed throttling Netflix traffic until the extortion was paid (Comcast in this case). Days later everything was running smooth as a baby's ass. So how can you seriously make an argument that all the blame lies on Netflix' shoulders when the ISP's customers are paying for the bandwidth to receive the content?

Let's say there was a burden. If the ISPs aren't willing to upgrade their networks then their business model is the problem, not how the internet works. And according to the data it looks like the ISPs infrastructure isn't that bad off anyway, they were simple messing with the traffic to extort payments from content providers.

TL;DR: WTF are you talking about?

http://blog.level3.com/global-...

Are you seriously suggesting that congested ports -> Netflix pays for their own direct interconnects -> uncongested ports somehow proves that Netflix was being throttled? Because, frankly, it suggests the opposite to me (i.e. moving lots of traffic to a different interconnect freed up capacity on the original). Your own link shows the general congestion: see this graph.

You can, quite easily, make the argument that Comcast (or Verizon, or whoever the peer in question is) let that situation fester until it resulted in their "winning" a new customer (Netflix) from level3, but certainly not that their traffic was being treated differently from anyone else's.

Then how do you explain the Level 3 data? The major ISPs got caught red-handed throttling Netflix traffic until the extortion was paid (Comcast in this case). Days later everything was running smooth as a baby's ass. So how can you seriously make an argument that all the blame lies on Netflix' shoulders when the ISP's customers are paying for the bandwidth to receive the content?

Let's say there was a burden. If the ISPs aren't willing to upgrade their networks then their business model is the problem, not how the internet works. And according to the data it looks like the ISPs infrastructure isn't that bad off anyway, they were simple messing with the traffic to extort payments from content providers.

TL;DR: WTF are you talking about?

http://blog.level3.com/global-...

> Comcast's peering connection to Level 3 has been saturated (over 90% capacity) 24/7 for over a year now

Got a source on that? Not that I doubt you, just looking to back up that claim.

While he doesn't come right out and say the name of any specific ISP Mark Taylor VP of Content and Media at Level 3 points his finger at 5 major US ISP's that have been saturated for over a year and refuse to upgrade their connection. Take that revelation and combine it with this graph which shows 8 Major ISPs and the relative speed with which Netflix traverses them and the 5 companies he references become pretty clear. Granted the graph does originate from Netflix so grain of salt and all that but I'm inclined to believe the data.

You can whine and bitch and proclaim all you want it still doesn't make it so. Your ISP sells you "up to x mbps" meaning they can provide anything they want and may occasionally actually give you the top speed you are paying for but unless your contract has a best minimum speed clause you aren't guaranteed anything faster than their slowest connection speed. And if you haven't already read this and this do yourself a favor and take the time. It is enlightening.

You can whine and bitch and proclaim all you want it still doesn't make it so. Your ISP sells you "up to x mbps" meaning they can provide anything they want and may occasionally actually give you the top speed you are paying for but unless your contract has a best minimum speed clause you aren't guaranteed anything faster than their slowest connection speed. And if you haven't already read this and this do yourself a favor and take the time. It is enlightening.

If Verizon wanted to offer the best experience to their Netflix subscribing users they would allow Netflix to install a streaming server in their server farm. This would save Verizon money and prevent the throttling that happens at the peer junction.

To illustrate imagine 40% of Verizon ISP customers are streaming a movie from Netflix. Without the streaming server the entire 40% have to traverse the backbone which Verizon pays a tier A provider like Level 3 for. Now Verizon, like most USA ISPs oversells the capacity they can accommodate because they don't expect everybody to use their full bandwidth portion simultaneously so to save money they also under purchase back end peering connections so that 40% of traffic just slammed all the connection going from Verizon to their tier A provider slowing traffic for everyone trying to access a connection not on Verizon's network. If you add the streaming server inside Verizon's network that 40% of traffic never leaves Verizon's infrastructure thus negating the need to upgrade their back end connection to accommodate the load. Netflix simply sends any new content to the streaming server when it becomes available. Now this scenario SAVES Verizon/Comcast/etc. money but they insist Netflix pay for the privilege of putting the server inside their network. The only reason they would pass up the opportunity to save money is if they also had a streaming service that competes with Netflix which could potentially make them more than they would save. VOD (Video on Demand) and RedBox Instant are just such services. This is why ISPs should not be content providers.

If you read the followup post from Level 3 the average utilization across all of its peers is 36%. Level 3 has 51 entities they peer with. Of those 12 are currently saturated. Of the 12, 6 are working with Level 3 to upgrade. 5 of the remaining 6 are major US ISP's who have been saturated at 90+% 24/7 for over a year and refuse to upgrade their circuit. This is not just affecting Netflix traffic this affects all of that ISP's traffic to its customers. You say correctly that peering is not free but turn around and say it's OK for them to sell something they don't own (oversell capacity) and then charge a third party to cover the extra cost despite the fact you got paid for something you can't deliver. You also rightly said deploying infrastructure is expensive but the cost to maintain it is substantially less so the existing infrastructure costs have gone down but customer costs have gone up with no upgrades planned but a customer base that has expanded. No matter how you try to spin it ISP's are double dipping at best and committing outright fraud at worst. If they cannot sustain the all you can eat buffet they need to switch to per bit billing as much weeping and gnashing of teeth as that will cause and charge the biggest users the largest fees.

It is Comcast creating the bottleneck and it is done deliberately.

Not exactly. They aren't really creating the bottleneck (that's being done by their customers), they're just being negligent in expanding their peering capacity to alleviate it.

Or you could make the case that they are attacking the bottleneck from both the supply and demand sides instead of just the supply side. By charging Netflix a peering fee, they are increasing the price of Netflix which reduces demand for data from Netflix, partially eliminating the bottleneck. Then they use the revenue to increase their bandwidth to Netflix, further eliminating the bottleneck. The demand curve illustrates how this all works.

"One person" may only stream one video at a time, but "people" as a whole may stream thousands or tens of thousands of videos all at the same time, and that's what creates the bottleneck in the peering connection. These same "people" are the "people" who currently stream videos over Comcast et.al. and create the peering bottleneck between Comcast and Level 3.

It is Comcast creating the bottleneck and it is done deliberately. They want you to believe it is Netflix that has the problem but they could have solved it for their entire customer base for ~$30K according to Level 3. And Netflix offered to host their own servers inside of Comcast's network which would eliminate the bottleneck altogether but Comcast refused instead demanding tribute before allowing more Netflix traffic.

Which is all relatively pointless when their connection to the backbone is fscked on purpose to keep your streaming choppy.

Well according to Level 3 they refuse to upgrade their circuits now anyway so it wouldn't make a whole lot of difference to the end user.

1) The problem is manufactured. The largest American ISPs are purposely congesting their networks so that it appears that the content providers are swamping them. With proper infrastructure investment they wouldn't be. See the Level 3 blog post from a few days ago:
http://blog.level3.com/global-...

They want their cake and eat it too. They charge the consumer for bandwidth (which the price is kept artificially high by limiting their networks, and by lack of competition). Then they want to charge for peering arrangements to pay for hardware upgrades. Not only that, they save capex costs since there's no incentive to upgrade their overall infrastructure...

2) There is too much fuzziness in the FCC proposed regulations for pricing the fast lane. Only that the schedule is commercially reasonable. In fact, the FCC can't dictate pricing policies for entities that aren't common carriers.

With wishy-washy terms, the startup who wants to provide a higher-bandwidth service will get screwed. There's no way a for-profit company can't take advantage of the "gift" of opaque pricing and still be fair to their shareholders. Similarly, it opens a Pandora's box for finding a million loopholes to start messing around with tolls on the slow lane too.

Bottom line is that they need to be common carriers, end of story. To preserve an open internet, they can't extract revenue from both the transport and content sides. That's what they desperately want and what started this whole mess when the Bush administration declared the internet as an "information service". Then the Obama administration continues the charade by appointing lobbyists to the FCC.

HTH.

Well, I do use Netflix, and have had their service non-stop for nearly a decade. I used to have 8 discs at a time, but for the past few years have only used their streaming service. I am outraged at both companies for even considering the peering arrangement. (If your not familiar, google: "netflix comcast deal" they are also working on deals with other ISPs all of which hurt network neutrality. Want to know why? Click my links below!)

Here is some food for thought:
A bunch of cable tv channels were dropped from directv a few times fairly recently during contract disputes. If level 1 providers like cogent and level3 take a card from big cables deck; by offering offending ISPs just had a taste of dark fiber and all customers jumping ship they might change their tune. I'm sure other ISPs could build a decent network with all those new customers. If you hold their customers hostage and they will definitely come to the table or go out of business, either would be good. We can not allow money hungry last mile monopoly to continue to drive internet speeds downward while erasing net neutrality.

If you are a netflix customer I urge you to please: CALL THEM AND DEMAND THEY CANCEL THE COMCAST AND OTHER ISP DEALS. WE CAN NOT STAND FOR THEM HURTING NET NEUTRALITY NOR CAN WE AFFORD TO PAY THEM TO PUT SERVERS IN EVERY SINGLE ISPs DATACENTER. I CALLED AND DEMANDED THAT THEY SEND A MESSAGE TO THEIR CEO TO GOOGLE LEVEL3 http://blog.level3.com/global-... and COGENTs http://www.cnet.com/news/cogen... STANCE ON THE SITUATION.

Spread the word, please, I beg you as a longterm netflix customer and fan who loves this (normally) innovative and forward thinking company.

I couldn't agree more with this comment. If it is OK for netflix to pay network providers for peering/colocation, then we should be able to get reimbursed for hosting a netflix node even as a customer.

Here is some food for thought:
A bunch of cable tv channels were dropped from directv a few times fairly recently. If level 1 providers take a card from big cables deck; by offering offending ISPs just had a taste of dark fiber and all customers jumping ship they might change their tune. I'm sure other ISPs could build a decent network with all those new customers. If you hold their customers hostage and they will definitely come to the table or go out of business, either would be good. We can not allow money hungry last mile monopoly to continue to drive internet speeds downward while erasing net neutrality.

If you are a netflix customer I urge you to please: CALL THEM AND DEMAND THEY CANCEL THE COMCAST AND OTHER ISP DEALS. WE CAN NOT STAND FOR THEM HURTING NET NEUTRALITY NOR CAN WE AFFORD TO PAY THEM TO PUT SERVERS IN EVERY SINGLE ISPs DATACENTER. I CALLED AND DEMANDED THAT THEY SEND A MESSAGE TO THEIR CEO TO GOOGLE LEVEL3 http://blog.level3.com/global-... and COGENTs http://www.cnet.com/news/cogen... STANCE ON THE SITUATION.

Spread the word, please, I beg you as a longterm netflix customer and fan who loves this (normally) innovative and forward thinking company.

The two comments above were exactly my thought on the subject. Level 3 has responded in the comments (on the blog in the second link of the news article) that many of their peering agreements do not have many of the contractual restrictions we assume they do. A bunch of cable tv channels were dropped from directv a few times. If level 1 providers take a card from big cables deck; by offering offending ISPs just had a taste of dark fiber and all customers jumping ship they might change their tune. I'm sure other ISPs could build a decent network with all those new customers. If you hold their customers hostage and they will definitely come to the table or go out of business, either would be good. We can not allow money hungry last mile monopoly to continue to drive internet speeds downward while erasing net neutrality.

If you are a netflix customer I urge you to please: CALL THEM AND DEMAND THEY CANCEL THE COMCAST AND OTHER ISP DEALS. WE CAN NOT STAND FOR THEM HURTING NET NEUTRALITY NOR CAN WE AFFORD TO PAY THEM TO PUT SERVERS IN EVERY SINGLE ISPs DATACENTER. I CALLED AND DEMANDED THAT THEY SEND A MESSAGE TO THEIR CEO TO GOOGLE LEVEL3 http://blog.level3.com/global-... and COGENTs http://www.cnet.com/news/cogen... STANCE ON THE SITUATION.

Spread the word, please, I beg you as a longterm netflix customer and fan who loves this (normally) innovative and forward thinking company.

Why don't you read what someone at Level 3 has to say about the issue?

"Level 3 has 51 peers that are interconnected in 45 cities through over 1,360 10 Gigabit Ethernet ports"
"The average utilization across all those interconnected ports is 36 percent."
"A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers."
"the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content."
"Five of those congested peers are in the United States and one is in Europe."

Five major US ISP's all deliberately refusing to upgrade their interconnect. How many "major" ISP's do you know of in the US?

(Sorry, a properly grammatical title would not fit in the space allotted)

Netflix & Level 3 Only Telling Half The Story, Won’t Detail What Changes They Want To Net Neutrality

In a fairly deep and interesting article over at StreamingMedia.com, Dan Rayburn argues that there is more to the story here and that neither Netflix nor Level 3 are giving us their proposed solutions. He goes through both the Netflix and the Level3 blog posts, taking them apart very carefully.

It is not a network neutrality problem, but rather a business problem. Worthwhile read.

I am routinely a bit confused as to why datacenters aren't predominantly located in places with colder climates. Free cooling from the outside during the winter and whatnot. Is there simply a lack of infrastructure to make an ultrahigh-bandwidth line out to...say...northern Montana?

Basically, yeah.

A lot of the networks have expanded where there's people: early networks grew up around universities and government facilities (often located in or near major population centers), companies later grew up (or migrated to) where the tech people were, and things more or less grew organically from there.

Take a look at Level3's network map for the US: there's a lot of facilities in areas where there's a lot of population: SF Bay Area, Los Angeles, New England, etc. Florida has more than I would otherwise expect, but it's not too surprising. There's basically nothing in the Dakotas and very little in other sparesely-populated areas.

One may well be able to get some network connections in remote areas, but it'll likely be expensive, inconvenient, and from a considerably less-diverse group of network providers than one might get in, say, Ashburn VA or the Bay Area.

Level 3 has a blog post on this subject. They point out that the amount of data that a user consumes is mostly irrelevant because an ISPs costs are driven by the peek data rate not the total amount of data transferred.

But if, as appears to be the case, Comcast threatens to resolve this by targeting video traffic specifically (which in practice means netflix), then they're in the wrong.

Except that it isn't the case, as far as I can tell. It seems that Level3 is playing us all for suckers - their press release doesn't indicate that Comcast is specifically targeting Internet video, it only refers to "Internet online movies and other content", which basically means "packets".

Level3 is trying to spin this as a network neutrality debate, when it's pretty clearly not. This is a peering dispute.

Level3 DOES NOT have to interconnect with Comcast to access Comcast subscribers. Comcast is not Tier 1, which means that there are alternate indirect routes between Comcast and Level3. However, not interconnecting with Comcast would create huge traffic imbalances that could jeopardize Level3's other peering relationships.

are clutching at straws, when all you're really after is faster BitTorrent.

The simple answer is: Download is for clients, Upload is for servers.

... and there's the other shoe. You must be a media shill, right? You don't give a damn about what a user is capable of doing, you only care about there being a gatekeeper somewhere where all bits have to pass in order for people to communicate.

And how could you possibly argue that position unless you work for the only people that network topology could possibly benefit?

Unless you've got a good reason you want residential properties to suddenly be servers, making things a *lot* more difficult on the security and management front...

And what would you know about "security and management"? Speaking as Senior sysadmin of a residential ISP, I am in charge security and management. And do you know what straws I see to grasp at from this position? 70% of our user base, bless their hearts, can't tell their computer from their monitor and think their web browser is named "Foxfire". Yet an overlapping 30% I have personally assisted in configuring realtime video conferencing, running security cameras and IP-based security systems at their homes, running VPN tunnels and remote desktop from their office to their home connections, and diagnosing trouble playing online games including hosting their own servers.

What you don't seem to get, Dot, is that the backbone of the internet itself is decentralized. We run a tier 1 network and follow transit prices like the goddamned stock market. Today we might be routing most of our outbound traffic through Level 3, tomorrow we slurp all the inbound requests to our colo through 360 Networks because they've bumped up their link budget.

Who gives a damn about Bittorrent when the ultimate decentralized protocol is BGP? Do you really think the internet would have scaled to the sizes we see today if all transactions had to route through some centralized point of failure controlled by a media monopoly?

Didn't you learn anything when Darpa's rag-tag network of networks dwarfed both AOL and Compuserv's walled gardens over the span of a couple of years in the early nineties? That was only possible because every hobbiest bulletin board could simply link to the cloud (yeah, that is the correct use of the term) and lend access to everyone who was already dialing into them.

But the hub-star configuration simply isn't the way to go. It doesn't scale, and like you it discourages participation and expects everyone to be neither seen nor heard. If that were the future, then America Online wouldn't be Offline already and we'd all still be watching television instead of trading up to hulu, slingbox, and youtube.

But we all know you're really just trolling, don't pretend otherwise. If you really believed what you were saying, you would stop pushing ascii characters the wrong way up your cable to say things on slashdot. That, obviously, takes greater upload capability than a simple TCP ACK.

The last thing I want is a level 5 dwarf (haha) providing me my OS.

What about a level 3 little person providing your business with an Internet connection?

Heh, sorry for posting so many times: a follow-up. A couple traceroutes show that I'm being bounced directly from Level3 to MIT. If there are no other middle-men (none as I can see) that would make MIT a Tier 2 network which, as far as I can tell, would allow them to lease their backbone to whomever they want, but IANAL. http://www.level3.com/legal/acceptable_use_policy.html They do prohibit illegal activity, which could be a problem in public wifi.

Uhh, that is the answer dude. Comcast's problem isn't with IP Transit. Comcast's problem is the same problem faced by all cableco's -- they have a shared last mile. I don't know what speeds they offer, but I do know that my Roadrunner connection is 5.0/384. At 5.0 it takes less then nine customers to completely max out the downstream on a DOCSIS channel. The only real solution to this problem is to split the network into smaller nodes so less people are sharing the bandwidth on them.

You are in the dark if you think the problem is just the last mile. On the other end the ISP has to buy the bits and bandwidth. Fixing the last mile simply exceeds the capacity of the ISP's connection. Fixing the ISP's connection is only part of the cost. Buying the extra bits is the big expense. Who do you think Comcast peers with and how do you think they get a connection to the backbone? Do a trace to Google for example. Comcast in my area connects to Level 3 net in Seattle.

http://www.level3.com/
http://en.wikipedia.org/wiki/Level_3_Communications

If you think fixing the ISP bandwidth problem is simply upgrading the last mile, you are mistaken. The growing bill for bits from Level 3 is the problem.
Level 3's primary focus is selling service to organizations with large bandwidth requirements, such as telecom carriers, cable TV operators, universities, web hosting companies, and to other, smaller ISPs, often known as Tier 2 carriers.

They sell bandwidth to Comcast. For Comcast to buy more bandwidth without increasing revenue is poor business planning. Many people only see the last mile problem and fail to look upstream.

Great proposal. For $60/month form the subscribers, provide service that cost you $150/month per subscriber. Good answer. ISPs buy bits. Buying a bigger pipe and greater monthly traffic isn't free.

University of Oregon had to deal with just this issue in the early Napster days.
http://scout.wisc.edu/Projects/PastProjects/net-news/00-01/00-01-25/0001.html

Oregon State University became concerned
about the program before the RIAA suit, when systems
administrators noticed that Napster was consuming 5 percent of
the school's bandwidth. Napster activity could have pushed the
school over its $75,000 yearly budget for bandwidth, says Oregon
State's vice provost for information services Curt Pederson,
noting that the school's bandwidth usage would double every 90
days if not controlled.

How many times do you propose letting the annual budget of $75,0000 double every 90 days as Peer to Peer catches on.
The bill doubles regularly while not adding a single new paying customer. This is not a good business plan.

This heavy use is created by only about 20% of the population and is growing. It's chop time or die. Comcast having to buy bits is faced directly with the rising cost of exploding bandwidth use. The throttling the universities had to do to maintain IT budgets is why most universities web pages load like they are served on a dial-up connection. They buy limited bandwidth which is mostly saturated in the evenings. The RIAA when nailing college students only download a few songs from the A list from the student. The big reason for this is it would take forever to download the entire A list through a saturated edu connection to the ISP.

Check with any university student regarding the speeds they get on campus. It's nothing to write home about.

Dude! Save your company some money and go with Level 3's Managed Modem service. http://www.level3.com/559.html

Level-3 and Cogent are not all that differnet in size. They both cover the US from coast to coast. They both go to Europe. Level-3 goes to Hawaii, but that is the only real footprint difference.

Cogent has fewer US POPs, but still hits all four corners. Cogent claims 80G backbones. Level-3 claims 110G backbones. Both networks have real 24x7 operations centers and skilled operators.

You cannot deny that there is a differnce in size between the two companies, but it is pretty easy to describe both as "Tier-1" networks.

If you read the Level-3 peering policy statement:

http://www.level3.com/1511.html

(which is where the Level-3 bw numbers come from, so they might be a little old), it would look on the surface that peering with Cogent is perfectly reasonable.

The complaint of Level-3 is twofold. The first one is, "we are bigger, so you should pay us". It is true that L3 is bigger, but whether they can throw that around and make Cogent pay is not 100% clear. When you buy from Level-3, you "expect" to be able to talk to Cogent. I am a Level-3 customer and L3 did not imform me that they were cutting these lines off. I was pissed. I knew that L3 could legally do what they wanted, but I thought it inappropriate for them to do it without giving their customers notice. Level-3 chopped off 45 million IP addresses when they de-peered Cogent. Even if L3 thought it was Cogents "fault", it was still L3 that threw the switch.

One other issue of this is that some major suppliers buy from L3 and don't multi-home. Thus, RoadRunner cable modems lost contact with all Cogent sites. This left RR answering the phones from customers. I am sure it cost RR a lot of hassle, and probably some real money. I cannot imaging that RR was not threatening L3 with "we are going to multi-home and you will lose 60%+ of our traffic if you don't get this working now". L3 had a lot to lose.

This is the real problem with Level-3 taking action. Whether Level-3 is right or wrong, it's customers will still blame L3. L3 did enter into peering with Cogent. L3 did initiate the termination of the peering arrangement. In doing so, L3 did cutoff a large chunk of the internet to it's own customers. Image if Level-3, on it's home page, prominently stated that you were not buying connection to the "whole" internet. No one would buy service from them. Level-3 gets customers because, on the whole, we trust Level-3 to work hard to keep us connected. This trust is what L3 is selling. The de-peering has damaged this trust and will, in the end, cost L3 a lot more money than they were trying to extract from Cogent.

The only real, technical issue between L3 and Cogent is traffic symmetry. Cogent sells to a lot of hosters. L3 sells to a lot of end-users (cable modems, dial-up, etc.). Thus more bytes move from Cogent to L3 than from L3 to Cogent. If the two networks only touched at one point, this would still be fair. After all, it takes a L3 customers to request a Cogent porn page. But the peering between L3 and cogent happens at multiple points. Thus, you get the "hot potato" case where traffic from Cogent dumps to L3 early (and vice versa, traffic from L3 dumps to Cogent early). This means more Cogent bytes travel on L3's network than on Cogents network. This is presumably what L3 wants Cogent to pay for. While this may be a "real" traffic issue, I suspect L3 wants more from Cogent than is actually reasonable. What would be interesting would be for L3 to advertise its routes to Cogent differently at each peering point. Thus Cogent would carry it's own outbound traffic over more of it's own network and the "hot potato" effect would not happen. Perhaps BGP does not work that way, but it should be able to. I suspect L3 was more interested in "monetizing" it's relationship than in making it "fair".

I also just noticed that L3 is starting to compete with Cogent on price. It used to be that the $3000/mo for a 100mbit

Cable modem users and DSL users are unaffected (unless Cogent is their provider's only upstream, and I haven't heard of any that are in that situation). They can use the vast majority of interesting services on the 'net just fine. The vast majority of high-traffic web sites are on multi-homed connectivity. There *are* popular sites on cruddy connectivity, but they'll learn quick with this, as they should. Never keep all your eggs in one basket. That lesson is as old as the commercialized Internet

Listen, You seem to keep responding while ignoring what I'm actually saying, so i'm going to spell it out to you.

Customers of some ISPs that have routes out both to the L3 side and Cognent side CAN NOT access any Cognent controlled networks (AS174). In some cases it has to do with not knowing another route to that network. In other cases it has to do with Cognent blocking a path they just don't want used. Case A is Level3's issue, Case B is Cognents. Either way, the downstream guy is screwed.

Look. Here's me trying to get to Level3 side:

1 1 ms 1 ms 1 ms 192.168.2.1
2 9 ms 7 ms 8 ms 10.33.0.1
3 13 ms 15 ms 8 ms pos0-0-nycmnyb-rtr1.nyc.rr.com [24.29.97.93]
4 9 ms 8 ms 9 ms 24.29.97.25
5 9 ms 9 ms 10 ms so-6-1.car2.Weehawken1.Level3.net [63.208.104.5]
6 12 ms 9 ms 9 ms ge-7-0-0.mp2.Weehawken1.Level3.net [4.68.125.141]
7 14 ms 14 ms 18 ms as-3-0.bbr2.Washington1.Level3.net [4.68.128.206]
8 22 ms 14 ms 13 ms ae-22-54.car2.Washington1.Level3.net [4.68.121.115]
9 24 ms 16 ms 14 ms 4.79.228.26
10 14 ms 14 ms 17 ms 66.249.95.123
11 15 ms 16 ms 15 ms 64.233.174.130
12 18 ms 16 ms 16 ms 216.239.48.110
13 15 ms 16 ms 15 ms 216.239.37.99

And here's me trying to get to something on the Cognent side:

Tracing route to vpn.google.com [66.28.250.25]
over a maximum of 30 hops:

1 1 ms 1 ms 1 ms 192.168.2.1
2 8 ms 9 ms 10 ms 10.33.0.1
3 9 ms 9 ms 8 ms pos0-0-nycmnyb-rtr1.nyc.rr.com [24.29.97.93]
4 9 ms 10 ms 9 ms 24.29.97.25
5 8 ms 8 ms 9 ms pos2-0-nycmnya-rtr2.nyc.rr.com [24.29.101.253]
6 9 ms 9 ms 10 ms pop2-nye-P13-3.atdn.net [66.185.141.37]
7 10 ms 9 ms 10 ms bb2-nye-P1-0.atdn.net [66.185.151.66]
8 9 ms 10 ms 9 ms pop1-nye-P1-0.atdn.net [66.185.151.51]
9 9 ms 9 ms 9 ms Verio.atdn.net [66.185.139.150]
10 9 ms 11 ms 10 ms p16-0-1-3.r21.nycmny01.us.bb.verio.net [129.250.3.48]
11 21 ms 16 ms 15 ms p16-1-2-2.r21.asbnva01.us.bb.verio.net [129.250.4.27]
12 * * * Request timed out.

The fact that RoadRunner is sending my packets via ATDN via Verio to get to AS174 shows me that the pinned route RR previously had (ie, all traffice for cognent side, haul via Verio which Cognent bought) is still up, but Cognent is actively blocking the traffic. If they didn't block it, we wouldn't know they were depeered and this would be a non-story. Now, I can't tell you that previously the data was backhauled via the AS3356 (Level3) network, but this is my guess. I just don't have any tracerts from then.

But not that Cognent is the only bad guy in this, Level3 has no advertised routes to AS174. Check http://www.level3.com/LookingGlass/ :

Show Level 3 (New York, NY) BGP routes for 38.9.51.20
No matching routes found for 38.9.51.20.

And from what I read on NANOG they are filtering advertisments of the AS174 routes from reaching anyone on their side. So even if you could route through L3 to Sprint to get to Cognent, you wouldn't know.

As for the home users who are whining about this, y'all need to get a life and just g

...it is dishonest for Level-3 to blame Cogent for this exclusively. Level-3 had a peering arrangement with Cogent for a long time. If you look at Level-3's interconnection policy page:

        http://www.level3.com/1511.html

It still looks like Cogent and Level-3 could peer under these terms. It was Level-3 that pulled the plug, not Cogent.

Quick quote from the introduction to these terms:

It is Level 3's view that IP transit is the rule and that settlement free interconnection or "peering" should be the exception when backbones interconnect.