First Detailed Data Analysis Shows Exactly How Comcast Jammed Netflix

An anonymous reader writes John Oliver calls it "cable company f*ckery" and we've all suspected it happens. Now on Steven Levy's new Backchannel publication on Medium, Susan Crawford delivers decisive proof, expertly dissecting the Comcast-Netflix network congestion controversy. Her source material is a detailed traffic measurement report (.pdf) released this week by Google-backed M-Lab — the first of its kind — showing severe degradation of service at interconnection points between Comcast, Verizon and other monopoly "eyeball networks" and "transit networks" such as Cogent, which was contracted by Netflix to deliver its bits. The report shows that interconnection points give monopoly ISPs all the leverage they need to discriminate against companies like Netflix, which compete with them in video services, simply by refusing to relieve network congestion caused by external traffic requested by their very own ISP customers. And the effects victimize not only companies targeted but ALL incoming traffic from the affected transit network. The report proves the problem is not technical, but rather a result of business decisions. This is not technically a Net neutrality problem, but it creates the very same headaches for consumers, and unfair business advantages for ISPs. In an accompanying article, Crawford makes a compelling case for FCC intervention.

Comments

Yes it is a peering problem ...

[jschultz410]

and not a net neutrality issue thankfully.

Settlement free peering between tier 1 carriers only happens when the flow of traffic is roughly balanced between the contracting peers.

When one peer is pushing a lot more traffic onto the other network, then that usually goes out the window and the pusher is required to pay the receiving network. Otherwise, networks would be monetarily incentivized to unload traffic they should carry on their own networks onto their peers' instead.

Re:Yes it is a peering problem ...

[Burdell]

Traffic balance is not the primary measure these days (from what I understand), it is just an economic decision. However, the Netflix case is interesting, because they were essentially used as a leverage tool by Cogent against the other carriers. Cogent has a long history of trying to get settlement-free peering, not meeting contract terms (whatever they are), getting dropped, and then blaming the other side. They have long wanted to be a settlement-free "tier 1" provider (which is a nebulous term, but go with it), but have generally not been. They sell bandwidth often at below-market rates in order to attract customers to leverage against the other "tier 1" providers. They saw Netflix on the rise and grabbed them, apparently selling bandwidth much cheaper than any other backbone (possibly at a loss even) in order to leverage settlement-free peering contracts out of other providers.

Any network engineering with a clue knows that you never buy bandwidth only from Cogent (or even Cogent and one other provider), because you _will_ get disconnected from somebody when Cogent gets in another peering dispute.

Re:Yes it is a peering problem ...

[Anonymous Coward]

Sensible?? More like nonsensical and completely misses the point of the article. As a great many people have pointed out, this is not unsolicited traffic, but traffic requested by Comcast's customer. So where on earth do you get the idea that

"When one peer is pushing a lot more traffic onto the other network, then that usually goes out the window and the pusher is required to pay the receiving network."

has ANY relevance to this story? Given what has been reported as the "trivial" costs of upgrading the interconnection points, by what other argument do you think the ISP's are in the right? Given that they are being paid by their customer's for internet access?

Re:Yes it is a peering problem ...

[Obfuscant]

Why does imbalance matter?

Because when peering agreements were created, the assumption was that trying to keep track of how much data you wanted us to carry and you keeping track of how much we gave you to carry was not necessary because we'd be charging each other the same amount if we did keep track.

Large amounts of data going one way breaks that basic tenet of peering. Now it makes sense to charge the other guy for data they want you to handle.

But this view ignores the most important point, that Comcast has explicitly promised its customers "internet access" at an advertised speed.

No. From here:

Performance Starter: Offer ends 01/04/15. Restrictions apply. Not available in all areas. Limited to new residential customers. Requires subscription to Performance Starter Internet service. Equipment, installation, taxes and fees, including regulatory recovery fees, Broadcast TV Fee (up to $3.50/mo.), Regional Sports Fee (up to $1/mo.) and other applicable charges extra, and subject to change during and after the promotion. After applicable promotional period, regular rates apply. Comcastâ(TM)s current monthly service charge for Performance Starter Internet is $49.95 (pricing subject to change). Service limited to a single outlet. May not be combined with other offers. Actual speeds vary and are not guaranteed.

Emphasis mine. The same emphasized text appears in the details for all three residential service levels. My statement stands: they did not promise you 24/7 full-rate access to anyplace off their net.

There are no pharmaceutical-like disclaimers during those commercials

There are when you actually go to sign up. And common sense tells you that they cannot guarantee those speeds to every site on the planet. They can't even guarantee those speeds to every site on the Comcast network. That's why they don't.

If Comcast says you're paying for "10Mbps internet", the assumption is that you get the advertised speed to the entire internet, provided there are no technical limitations outside of Comcast's control.

That's what some people assume, but that's not backed up by the service agreement.

It's not even backed up by common sense. Suppose you buy the Blast service and get 105Mbps download. You want to connect to my system and I've got Performance Starter (6 Mbps down, God knows what it is up). You ain't getting anywhere near 105Mbps from my stream. Even trying to connect to your next door neighbor who has the same service, you ain't getting faster than his upload allows. If you think Comcast could promise anything faster, then you must think they'll upgrade MY service to Blast for free because they promised YOU that you'd get data from me that fast, and you're paying them for my data at that speed.

Re:Common Carrier

[Burdell]

That would have zero impact. This is like the telephone company in city A have 96 channels to the telephone company in city B, but then 100 people try to make calls. Only some of them will go through, and that's a capacity issue, not regulated by Common Carrier status. They are not discriminating based on callers or anything, they are just "decliining" to upgrade capacity. In some cases, that could be regulated by state PUCs/PSCs, but AFAIK it is not normally. It is just up to the two carriers to reach an agreement.

This type of thing happened a lot in the early dialup ISP days, when telecom deregulation spawed a lot of CLECs that had to connect to ILECs to carry calls. The ILECs structured the contracts with settlement money for to flow to the destination of a call (thinking most of the CLEC calls would be _to_ ILEC users), but then the CLECs went and got all the dialup ISPs to move modem banks to them. Suddenly all the calls went _to_ the CLECs, and the ILECs had to pay (some did not and went to court instead).

Re: Multiple CDN contracts?

[bill_mcgonigle]

Netflix is its own CDN - they will give, for free, one or more caches to any ISP, causing any one movie to transit the ISP's nonfree network connections only once.

But this is about competition for video services, not caching.