Assuming that's true, Netflix probably should have bought transit directly on those networks instead of cutting corners by going with crappy providers that either didn't want to properly size their paid peering links or were willing to imbalance settlement free links.
Yep. If they kept it up, an investigation by someone neutral would show that Netflix was the bad guy in all of this. They chose shitty Tier1 ISPs for a reason and it had nothing to with providing a good experience for their customers.
I like the argument from Netflix that ISPs shouldn't let peer links congest. I guess every ISP has to give other ISPs infinite bandwidth without charging for it.
Settlement free peering is what made the internet what it is, but Netflix has profits to make and they've chosen to save money by going with transit providers that will play the victim when ISPs don't tolerate settlement free peering link imbalances or refuse to upgrade paid peering links without being paid.
If what Level 3 said is true, then they would have had something actionable in Verizon not activating links they're been paid to activate.
The author also makes the assumption that Netflix can't possibly be putting traffic on peer links just because they have a CDN. In that case, the 100% saturation from Level 3 to Verizon makes no sense. L3 doesn't dispute the 100% thing except to say that Verizon needs to enable the links they've supposedly been paid to enable.
Really? It actually cleared up when Netflix did the right thing and bought transit on the network(s) they wanted heavy access to.
It's also a weird coincidence that this issue never came up with other streaming companies - Youtube, Amazon, Hulu, to name a few. I guess Comcast and AT&T were favoring them too, right?
No, it wasn't something that happened. Comcast was managing traffic from Cogent, an ISP that was sending far more traffic one than the other; something they did to numerous other ISPs who also had problems with this.
There are two types of peering - settlement free wherein you exchange equal amounts of traffic, or paid peering where you pay for the links you want to the other ISP. Some, possibly all, settlement free peering agreements will have payment options for excess traffic that flows one way.
Cogent wanted paid peering style bandwidth upgrades at settlement free prices. Comcast, and all the others, were right to fight this.
Any of us would have to pay to put a server in our ISP's datacenter. I don't see why Netflix should get a free ride simply because they're popular.
Smaller ISPs probably appreciated it because they were on paid peering links that were getting saturated with Netflix traffic coming from an upstream peer. The alternative to them was pay to upgrade their links.
Cogent, the main ISP culprit in this, was on a settlement free link with Verizon, Comcast, Level 3, Sprint, AT&T, etc. that they abused repeatedly by sending far more traffic towards those ISPs than they received. THIS is what drew the ire of these companies. That it was a direct (VoD) competitor for most causing it only added salt to the wound.
I hate that this issue has made me repeatedly defend cable companies, especially fucking COMCAST, but they were not wrong to uphold the settlement free peering agreement, and were not wrong to expect to be paid for transit on their network.
A lot of the people here don't seem to understand what would happen if ISPs split so that the ones serving residential and small business customers were separate from the ones generating the content and those 'server' ISPs basically got unlimited peering bandwidth.
The only money to maintain the internal network would come from the residential customers and not the people generating the traffic. I don't look forward to extremely expensive service with low data caps and/or slow speeds and high latency.
I don't care what someone claimed in another thread.
The only way it would cost Comcast for traffic flowing across their peering points was if it was transiting their network to another settlement free peer and significantly upsetting that traffic balance.
If this had occurred, Comcast would have been well within their rights to drop the link altogether.
Though, the only real way this would happen (outside of an outage) was if Netflix was altering their routing policies to send traffic destined for certain IP blocks out their Cogent link with the intent that it transit Comcast for ISP X on the other side instead of a peer closer to that ISP.
Aside from temporary outages, traffic from Tier 1 ISP should not be transiting another Tier 1 to get to a third.
Yep, and this is the fault of Level 3 for not managing their traffic properly.
Amazon can also sue Level 3 for not maintaining SLAs. BUT, any large company that depends on streaming for a sizeable portion of its revenue is not going to leave details like this to chance the way we're led to believe Netflix did.
Perhaps it's time to question why a large streaming provider has chosen to buy transit from an ISP that peers with their customers and not from the customer's ISP directly like other streamers did. The reason is that they got a better deal from that other ISP who is then crying foul when the 'customer' ISP upholds the peering agreement and doesn't see fit to allow another ISP to get the benefits of a paid peering agreement from a settlement free one.
The analyses out there either covered throttling or dropping of peer links from Cogent, and at least one suggested that it was Cogent throttling Netflix at Cogent's egress so as to allow other traffic to go across that link.
If it's a settlement free peering link, traffic between them is supposed to be equal; you handle as much traffic from me as I handle from you. When you start sending 10 times more traffic my way than I send to you, it's time for you to manage your traffic better so as to maintain the balance, for me to throttle your traffic towards me to match my traffic to you, or for you to pay for the excess.
The idea that one ISP can collect all the business customers and demand unlimited bandwidth (via peer links) from another ISP that has the residential customers is ridiculous and will lead to nothing but bad things - an end to settlement free peering, data caps, and/or higher rates.
There are two ways ISPs interconnect - settlement free where they exchange equal amounts of traffic, or PAID where the side that is continually sending excess traffic pays the other ISP. Level 3 wanted paid peering style bandwidth expansion on a settlement free peering connection and Comcast wasn't having it.
I remember one of their blogs where they talked about how they were willing to pay for the hardware to expand the other side of the link and Comcast still wasn't biting. Level 3 appeared to be missing the point on purpose, since they were on the other side of that dispute with Cogent.
Comcast is under not obligation to allow another ISP free access to its network. Comcast gets paid, OR the other ISP handles the same amount of traffic FROM Comcast as is sent TO Comcast.
This would be a breach of contract if PizzaHut was their customer, a breach of the (paid or settlement free) peering agreement if Pizza Hut was a customer of another ISP.
I wonder why neither Netflix or Cogent ever brought legal action against the ISPs that were supposedly singling out Netflix for throttling.
The articles covering the issue show ISPs like AT&T, Verizon, Comcast, Level 3, etc. either dropping settlement free peer links due to traffic imbalance or throttling the whole thing to bring the inbound levels inline with the outbound. Usually this affected all of the customers of that ISP (Cogent).
Cogent is, at best, grossly irresponsible for allowing its settlement free link to go that badly imbalanced that the other side has to take such serious action.
They know what they were doing, as does Netflix. Netflix chose the shittiest provider it could to cut costs and then pretended not to know that the provider was abusing settlement free links.
I want to start an ISP that sells internet access super cheap to ultra-high volume customers and then manipulate uninformed rubes into fighting the ISPs I connect to into upgrading their peer links, and eventually their internal links, to handle all the traffic my customers are generating. One NN advocate would see to it that I only have to pay for the actual costs of the interconnect hardware.
I don't have to pay to build and maintain a residential/business size/scale internet service provider, nor do I pay much at all to access those networks. It's win-win for me and my customers.
On the other hand, ALL of the customers of those ISPs get to cover the ever increasing costs of upgrading the ISPs internal network to handle the added bandwidth demands of a few of them, so it's not so great for them.
Neither of those articles really refuted what I said.
Yep, all those companies were risking their settlement free agreements with Cogent and/or violating paid peering agreements with Cogent by deliberately throttling connections. In reality Cogent was irresponsibly allowing settlement peer links to go heavily unbalanced to the point that the other side was throttling the whole thing to maintain the ratios or simply shutting down links.
Things like this are interesting: "In the past, if two networks transferred so much data between themselves that they were about to exceed the capacity of their connection, they would have gotten in touch to solve the problem. As M-Lab notes in its report, “[T]he traffic that flows through these interconnections is the lifeblood of the Internet—nearly all of the value of the Internet comes from the exchange of traffic, even when the ISPs involved are fierce competitors.” The engineers would have worked out a solution to open the access network’s door to the outside world more broadly. And they would split the minor costs of doing this upgrade—a $300 piece of fiber, a $10,000 souped-up router. A January 2013 OECD report found that 99.5% of Internet interconnection agreements at Internet Exchange Points happen without any formal contracts; engineers easily make deals to share the very low cost of trading traffic between networks in the same building."
The key here is that the ISPs transfer data between themselves, not one ISP transferring fifty times as much data towards an ISP than it receives.
"In the past, requests for upgrades were routinely granted. Now, suddenly, upgrades are impossible without painful negotiations over fees that have no perceptible relationship to the cost of making the upgrade—and Comcast and the other eyeball networks are making no promises about restraining themselves in the future."
The upgrades were for easy and routinely granted when the equally exchanged traffic hit certain thresholds. When one side is the cause of the imbalance, they are the ones that need to pay for it. The alternative is forcing all customers of an ISP to pay for the demands of some while also effectively subsidizing the business model of Netflix.
Netflix had a reason for choosing Cogent and it had nothing to do with ensuring the best experience for their customers.
I'll point out again here that this didn't happen to Hulu, Youtube, Amazon, etc.
As for that second article, ISPs are not obligated to give free datacenter space or network access to anyone, especially not a previously abusive user. Did Netflix offer to pay for the rack space and transit they wanted, or were they expecting another free ride?
I think they won't because it will only cause huge legal headaches for them.
Disrupting their own customers is a lawsuit waiting to happen. Disrupting the customers of a paying peer is also a lawsuit waiting to happen. Disrupting the customers of a settlement free peer is grounds to terminate the agreement, which hurts the interfering side more than they're hurting the customers they were interfering with.
It makes some sense for network operators to have throttled p2p connections in the past due to their affect on the network Torrents actually do cause performance problems on DOCSIS nodes, especially large ones.
Can you actually show proof that Verizon throttled Netflix?
In other words, you think an ISP would openly attempt to extort a company that is one of the following: A customer of theirs - lawsuit A customer of a paying peer - lawsuit from the peer A custom of a settlement free peer - possible lawsuit from peer, risk of ending settlement free peering.
The DNS redirection on its own would be grounds for a lawsuit since the ISP and beneficiary company would be effectively hijacking the domain name Pizza Hut registered.
Slashdot Mirror
Assuming that's true, Netflix probably should have bought transit directly on those networks instead of cutting corners by going with crappy providers that either didn't want to properly size their paid peering links or were willing to imbalance settlement free links.
Strange that Hulu, Amazon, Youtube, etc.,didn't have this problem, only Netflix.
Yep. If they kept it up, an investigation by someone neutral would show that Netflix was the bad guy in all of this. They chose shitty Tier1 ISPs for a reason and it had nothing to with providing a good experience for their customers.
Then the problem was a contractual one, not a net neutrality one.
I like the argument from Netflix that ISPs shouldn't let peer links congest. I guess every ISP has to give other ISPs infinite bandwidth without charging for it.
Settlement free peering is what made the internet what it is, but Netflix has profits to make and they've chosen to save money by going with transit providers that will play the victim when ISPs don't tolerate settlement free peering link imbalances or refuse to upgrade paid peering links without being paid.
If what Level 3 said is true, then they would have had something actionable in Verizon not activating links they're been paid to activate.
The author also makes the assumption that Netflix can't possibly be putting traffic on peer links just because they have a CDN. In that case, the 100% saturation from Level 3 to Verizon makes no sense. L3 doesn't dispute the 100% thing except to say that Verizon needs to enable the links they've supposedly been paid to enable.
Feel free to negotiate a paid peering agreement then.
Stop blocking traffic and hitting vehicles that wish to get by and this won't happen.
Really? It actually cleared up when Netflix did the right thing and bought transit on the network(s) they wanted heavy access to.
It's also a weird coincidence that this issue never came up with other streaming companies - Youtube, Amazon, Hulu, to name a few. I guess Comcast and AT&T were favoring them too, right?
No, it wasn't something that happened. Comcast was managing traffic from Cogent, an ISP that was sending far more traffic one than the other; something they did to numerous other ISPs who also had problems with this.
There are two types of peering - settlement free wherein you exchange equal amounts of traffic, or paid peering where you pay for the links you want to the other ISP. Some, possibly all, settlement free peering agreements will have payment options for excess traffic that flows one way.
Cogent wanted paid peering style bandwidth upgrades at settlement free prices. Comcast, and all the others, were right to fight this.
Any of us would have to pay to put a server in our ISP's datacenter. I don't see why Netflix should get a free ride simply because they're popular.
Smaller ISPs probably appreciated it because they were on paid peering links that were getting saturated with Netflix traffic coming from an upstream peer. The alternative to them was pay to upgrade their links.
Cogent, the main ISP culprit in this, was on a settlement free link with Verizon, Comcast, Level 3, Sprint, AT&T, etc. that they abused repeatedly by sending far more traffic towards those ISPs than they received. THIS is what drew the ire of these companies. That it was a direct (VoD) competitor for most causing it only added salt to the wound.
I hate that this issue has made me repeatedly defend cable companies, especially fucking COMCAST, but they were not wrong to uphold the settlement free peering agreement, and were not wrong to expect to be paid for transit on their network.
So much this.
A lot of the people here don't seem to understand what would happen if ISPs split so that the ones serving residential and small business customers were separate from the ones generating the content and those 'server' ISPs basically got unlimited peering bandwidth.
The only money to maintain the internal network would come from the residential customers and not the people generating the traffic. I don't look forward to extremely expensive service with low data caps and/or slow speeds and high latency.
I don't care what someone claimed in another thread.
The only way it would cost Comcast for traffic flowing across their peering points was if it was transiting their network to another settlement free peer and significantly upsetting that traffic balance.
If this had occurred, Comcast would have been well within their rights to drop the link altogether.
Though, the only real way this would happen (outside of an outage) was if Netflix was altering their routing policies to send traffic destined for certain IP blocks out their Cogent link with the intent that it transit Comcast for ISP X on the other side instead of a peer closer to that ISP.
Aside from temporary outages, traffic from Tier 1 ISP should not be transiting another Tier 1 to get to a third.
Yep, and this is the fault of Level 3 for not managing their traffic properly.
Amazon can also sue Level 3 for not maintaining SLAs. BUT, any large company that depends on streaming for a sizeable portion of its revenue is not going to leave details like this to chance the way we're led to believe Netflix did.
Perhaps it's time to question why a large streaming provider has chosen to buy transit from an ISP that peers with their customers and not from the customer's ISP directly like other streamers did. The reason is that they got a better deal from that other ISP who is then crying foul when the 'customer' ISP upholds the peering agreement and doesn't see fit to allow another ISP to get the benefits of a paid peering agreement from a settlement free one.
The analyses out there either covered throttling or dropping of peer links from Cogent, and at least one suggested that it was Cogent throttling Netflix at Cogent's egress so as to allow other traffic to go across that link.
You're forgetting about ISP peering arrangements.
If it's a settlement free peering link, traffic between them is supposed to be equal; you handle as much traffic from me as I handle from you. When you start sending 10 times more traffic my way than I send to you, it's time for you to manage your traffic better so as to maintain the balance, for me to throttle your traffic towards me to match my traffic to you, or for you to pay for the excess.
The idea that one ISP can collect all the business customers and demand unlimited bandwidth (via peer links) from another ISP that has the residential customers is ridiculous and will lead to nothing but bad things - an end to settlement free peering, data caps, and/or higher rates.
There are two ways ISPs interconnect - settlement free where they exchange equal amounts of traffic, or PAID where the side that is continually sending excess traffic pays the other ISP.
Level 3 wanted paid peering style bandwidth expansion on a settlement free peering connection and Comcast wasn't having it.
I remember one of their blogs where they talked about how they were willing to pay for the hardware to expand the other side of the link and Comcast still wasn't biting. Level 3 appeared to be missing the point on purpose, since they were on the other side of that dispute with Cogent.
Comcast is under not obligation to allow another ISP free access to its network. Comcast gets paid, OR the other ISP handles the same amount of traffic FROM Comcast as is sent TO Comcast.
What other companies should be given free or almost free space in ISP datacenters? Why?
Netflix was trying to further reduce their costs by avoiding their own cut rate ISP and just getting a free ride on Comcast's network.
What costs was Comcast incurring that the caching server/CDN node would have alleviated?
This would be a breach of contract if PizzaHut was their customer, a breach of the (paid or settlement free) peering agreement if Pizza Hut was a customer of another ISP.
I wonder why neither Netflix or Cogent ever brought legal action against the ISPs that were supposedly singling out Netflix for throttling.
Citation needed.
The articles covering the issue show ISPs like AT&T, Verizon, Comcast, Level 3, etc. either dropping settlement free peer links due to traffic imbalance or throttling the whole thing to bring the inbound levels inline with the outbound. Usually this affected all of the customers of that ISP (Cogent).
Cogent is, at best, grossly irresponsible for allowing its settlement free link to go that badly imbalanced that the other side has to take such serious action.
They know what they were doing, as does Netflix. Netflix chose the shittiest provider it could to cut costs and then pretended not to know that the provider was abusing settlement free links.
I want to start an ISP that sells internet access super cheap to ultra-high volume customers and then manipulate uninformed rubes into fighting the ISPs I connect to into upgrading their peer links, and eventually their internal links, to handle all the traffic my customers are generating. One NN advocate would see to it that I only have to pay for the actual costs of the interconnect hardware.
I don't have to pay to build and maintain a residential/business size/scale internet service provider, nor do I pay much at all to access those networks. It's win-win for me and my customers.
On the other hand, ALL of the customers of those ISPs get to cover the ever increasing costs of upgrading the ISPs internal network to handle the added bandwidth demands of a few of them, so it's not so great for them.
Neither of those articles really refuted what I said.
Yep, all those companies were risking their settlement free agreements with Cogent and/or violating paid peering agreements with Cogent by deliberately throttling connections. In reality Cogent was irresponsibly allowing settlement peer links to go heavily unbalanced to the point that the other side was throttling the whole thing to maintain the ratios or simply shutting down links.
Things like this are interesting:
"In the past, if two networks transferred so much data between themselves that they were about to exceed the capacity of their connection, they would have gotten in touch to solve the problem. As M-Lab notes in its report, “[T]he traffic that flows through these interconnections is the lifeblood of the Internet—nearly all of the value of the Internet comes from the exchange of traffic, even when the ISPs involved are fierce competitors.” The engineers would have worked out a solution to open the access network’s door to the outside world more broadly. And they would split the minor costs of doing this upgrade—a $300 piece of fiber, a $10,000 souped-up router. A January 2013 OECD report found that 99.5% of Internet interconnection agreements at Internet Exchange Points happen without any formal contracts; engineers easily make deals to share the very low cost of trading traffic between networks in the same building."
The key here is that the ISPs transfer data between themselves, not one ISP transferring fifty times as much data towards an ISP than it receives.
"In the past, requests for upgrades were routinely granted. Now, suddenly, upgrades are impossible without painful negotiations over fees that have no perceptible relationship to the cost of making the upgrade—and Comcast and the other eyeball networks are making no promises about restraining themselves in the future."
The upgrades were for easy and routinely granted when the equally exchanged traffic hit certain thresholds. When one side is the cause of the imbalance, they are the ones that need to pay for it. The alternative is forcing all customers of an ISP to pay for the demands of some while also effectively subsidizing the business model of Netflix.
Netflix had a reason for choosing Cogent and it had nothing to do with ensuring the best experience for their customers.
I'll point out again here that this didn't happen to Hulu, Youtube, Amazon, etc.
As for that second article, ISPs are not obligated to give free datacenter space or network access to anyone, especially not a previously abusive user. Did Netflix offer to pay for the rack space and transit they wanted, or were they expecting another free ride?
I think they won't because it will only cause huge legal headaches for them.
Disrupting their own customers is a lawsuit waiting to happen.
Disrupting the customers of a paying peer is also a lawsuit waiting to happen.
Disrupting the customers of a settlement free peer is grounds to terminate the agreement, which hurts the interfering side more than they're hurting the customers they were interfering with.
It makes some sense for network operators to have throttled p2p connections in the past due to their affect on the network Torrents actually do cause performance problems on DOCSIS nodes, especially large ones.
Can you actually show proof that Verizon throttled Netflix?
In other words, you think an ISP would openly attempt to extort a company that is one of the following:
A customer of theirs - lawsuit
A customer of a paying peer - lawsuit from the peer
A custom of a settlement free peer - possible lawsuit from peer, risk of ending settlement free peering.
The DNS redirection on its own would be grounds for a lawsuit since the ISP and beneficiary company would be effectively hijacking the domain name Pizza Hut registered.
In other words, you can't explain how a company would actually go about charging a pizza place more than another while justifying it.