First Detailed Data Analysis Shows Exactly How Comcast Jammed Netflix
An anonymous reader writes John Oliver calls it "cable company f*ckery" and we've all suspected it happens. Now on Steven Levy's new Backchannel publication on Medium, Susan Crawford delivers decisive proof, expertly dissecting the Comcast-Netflix network congestion controversy. Her source material is a detailed traffic measurement report (.pdf) released this week by Google-backed M-Lab — the first of its kind — showing severe degradation of service at interconnection points between Comcast, Verizon and other monopoly "eyeball networks" and "transit networks" such as Cogent, which was contracted by Netflix to deliver its bits. The report shows that interconnection points give monopoly ISPs all the leverage they need to discriminate against companies like Netflix, which compete with them in video services, simply by refusing to relieve network congestion caused by external traffic requested by their very own ISP customers. And the effects victimize not only companies targeted but ALL incoming traffic from the affected transit network. The report proves the problem is not technical, but rather a result of business decisions. This is not technically a Net neutrality problem, but it creates the very same headaches for consumers, and unfair business advantages for ISPs. In an accompanying article, Crawford makes a compelling case for FCC intervention.
Bennett Haselton is a frequent contributor, and I tend to hold-off judgement on these things until I read his fine points on the topic. I was recently particularly moved by his work on line queues at Burning Man. It changed my entire life. I now piss sitting down.
"In an accompanying article, Crawford makes a compelling case for FCC intervention."
That won't work unless it comes with a check with seven digits attached to it.
The world's burning. Moped Jesus spotted on I50. Details at 11.
Once again, a call for net neutrality will ensue. All we really need is for the FCC to call them Common Carriers and apply the age old law.
It has already been applied to Telecoms and Utilities, just apply it to the ISP's and be done with this crap.
http://en.wikipedia.org/wiki/C...
Stuff like this is why I think Net Neutrality discussions miss the mark - you're not going to fix the problem that way, you're only going to cause the cable companies to achieve the same throttling through other technical means. Trying to close technical loopholes via the lawmaking process requires a body of law the size of the tax code.
The fundamental problem is that companies with a legally-granted monopoly for delivering high-speed internet are also allowed to sell content. In a free market, that wouldn't bother me - competition would sort it all out. But "last mile" is about as far from a free market as you can get in most of the country these days, and so we get this as a result.
Last mile needs to become a public utility. Let vendors compete for my business, and I'll pick "just a pipe" or a content company or whatever mix fits my needs.
Socialism: a lie told by totalitarians and believed by fools.
and not a net neutrality issue thankfully.
Settlement free peering between tier 1 carriers only happens when the flow of traffic is roughly balanced between the contracting peers.
When one peer is pushing a lot more traffic onto the other network, then that usually goes out the window and the pusher is required to pay the receiving network. Otherwise, networks would be monetarily incentivized to unload traffic they should carry on their own networks onto their peers' instead.
Yeah, I'm sure netflix just goes around dumping truckloads of data on the information superhighway just at random, and picked on Comcast like a bully.
Oh wait, every one of those streams were requested from users of Comcast's network who thought that those awesome 150mbit internet speeds comcast advertised were real.
I should be able to use my bandwidth any way I please! It's freedom of speech! They can't throttle netflix! George Bush would be rolling in his grave!
This is a ploy from Obama to help spread misinformation about ebola (Obola) on his path to further to destroy the country!!1!
Engineer - "Hey Boss, we need some cash to upgrade the connection to these networks."
Boss - "What?! We just upgraded those connections a couple years ago"
Engineer - *rolls eyes* "Well the link is saturated, looks like lots of people watching online video... Netflix comes in over this connection so it makes sense"
Boss - "First they take our subscribers now they're forcing us to upgrade our equipment... well fuck em!"
Engineer - "Waaah?"
Boss - "You heard me, fuck em!"
Engineer - "But... our customers will get terrible service when they try to watch Netflix, or do anything else on that network for that matter"
Boss - "Exactly!"
The traffic isn't transiting Comcast going to another network. It's going to a Comcast subscriber who wants to watch a movie. So, yes, the subscriber is requesting a movie and the data is being delivered to them. There's no other route to the subscriber than through their ISP.
The Very First Honest Cable Internet Provider
I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
Netflix is its own CDN - they will give, for free, one or more caches to any ISP, causing any one movie to transit the ISP's nonfree network connections only once.
But this is about competition for video services, not caching.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Replace the Boss' last line with:
Boss: "Oh, boo hoo! What're they gonna do? Leave us and go to another ISP? We've got a monopoly in the area! They want Internet? They need to come to us. Besides, if we make Netflix look bad, maybe people will dump them and pay us $200 a month for cable again!"
My sci-fi novel, Ghost Thief, is now available from Amazon.com.
This is a very salient point. Netflix already has these arrangements with other ISPs. Only Comcast, AT&T, and Verizon (surprise surprise) refused to host local caching servers. Of course, this precedes their demands for more money because, "Waaaahhhh...they're stealing our customers, they need to pay!".
Netflix tried to be the better entity (within reason) and were told, in no uncertain terms, "Go fuck yourself."
Yay, free market!
*sigh*
"Helping to keep you two steps ahead of the Thought Police!"
That's quite a loaded statement.
Business buggery is not the only reason you might not get the advertised speed.
First of all, ISPs advertise "up to" a specific speed, which means that's the maximum bandwidth you're allowed.
It doesn't state or imply that you'll receive that speed consistently.
It means that, assuming the network is capable of that speed, if you were capable of getting higher speeds you'd be capped at that speed.
"First of all, ISPs advertise "up to" a specific speed, which means that's the maximum bandwidth you're allowed.
It doesn't state or imply that you'll receive that speed consistently."
Well, how about I pay "up to" the amount they want, then? I can throw that (in small print) on the back of a check I send them, to make it all legal and such.
Actually netflix offered to foot the bill for upgrading the bandwidth - it's literally a couple cross-connects in a datacenter, maybe a fiber card or two.
Oh, and netflix ALSO offers to drop a server in your datacenter *free* which caches all the common netflix streams. This reduces the internet bandwidth demands by something like 90+% since it lives within the ISP's datacenter and just needs to download each stream once.
But the last line is exactly the point. The ISPs are also TV providers and they don't want you to have a good netflix experience. If they can passively let that happen...well of course they will. No one can accuse them of taking any action to damage your netflix streaming...it's their complete inaction that's resulting in it.
You can get rich if you own a politician, but you have to be rich to buy one in the first place.
except CDN's have paid ISP's for years for hosting costs. netflix refuses to pay and wants their CDN hosted for free
"Similarly, M-Lab's data shows that the problem wasn't a lack of bandwidth on the part of the ISPs (i.e., no actual technical congestion), because those same ISPs had no problem connecting to a different transit provider, Internap. So the only logical culprit was the interconnection points. There was more than enough bandwidth to go around. There's just the single bottleneck of the interconnection border router (which, again, is trivially simple to get rid of by opening up more ports)."
link
XKCD:Xeric Knowledge Comically Dispen
All kinds of wrong. They will provide a server box to be housed in the ISP containing their most popular content FOR FREE.
Peter predicted that you would "deliberately forget" creation 2000 years ago...
Oh wait, every one of those streams were requested from users of Comcast's network who thought that those awesome 150mbit internet speeds comcast advertised were real.
150mbps isn't very fast. Did you mean 150Mbps?
The issue is that people think the 150Mbps "last mile" speed was some kind of promise of speed to every other site on the planet. I can request a data stream from a service that needs 100Mbps to function reliably, but that doesn't mean I'm guaranteed 100Mbps from end to end. If an ISP promises you that, you know they're lying. If you assume the last mile speed applies to every connection you make anywhere, then you're the one who's being foolish, not the ISP being dishonest.
Think twice next time you wonder why you aren't getting your advertised speed...
And another article about Comcast throttling Netflix without any background or context.
Backbone providers have peering agreements. Usually, if both backbone providers (i.e. Comcast & Verizon) are sending close to the same traffic between each other, the peering agreement is free for both parties. In the Netflix case, Netflix went with a small backbone provider, likely due to cost. The problem is that the backbone provider they chose sends way more traffic than they accept. Typically, this type of peering agreement means that the smaller backbone provider (i.e. Cogent) pays fees to the larger backbone provider (Comcast). It's my understanding that Cogent wouldn't or couldn't pay these fees, so Comcast throttled them.
Because Cogent couldn't or wouldn't pay the fees and customers were complaining, Neflix agreed to pay the peering fees for Cogent. Though, this isn't how the media or Netflix presents it.
I hate Comcast as much as anyone. I think that they are essentially a monopoly that takes advantage of their customers by increasing prices where there is little to no competition (a toothless FCC doesn't help). But, in this one instance, it's my opinion that Comcast had a good case against the provider used by Netflix and that, by selecting the lowest cost provider and possible knowingly selecting one that is a bit sketchy on the peering side, Netflix had some responsibility.
This is complete nonsense. Google and Akamai both place hardware just inside ISP borders at no cost. They don't pay the ISPs to do this, the ISPs are saving a shit-ton on transit and are happy to set aside a little space and electricity for the hardware. This includes Comcast. Try doing a traceroute to 23.79.61.240 for example, this is one of at least a dozen Akamai servers sitting in a Comcast cage in Atlanta. Akamai is paying zero dollars for that.
Netflix offered the same arrangement and Comcast said no, the only reason being that Netflix sells a competing product to Comcast's own video on demand. It's anticompetitive behavior, period, plain and simple.
from wikipedia
Franchise fees are fixed at a maximum of 5% of gross revenues. So how do municipalities maximize revenues from franchise fees? By maximizing cable company gross revenues. And how do municipalities maximize cable company gross revenues? By creating monopolies! By awarding exclusive license to one provider to extract monopolist profits from the public.
Note that there is nothing inherently wrong with permitting local governments to charge cable companies fees. That is justifiable to the extent that local governments incur costs of infrastructure repair with damage from cable installation. All that is needed is a single addendum to the law, one prohibiting local governments from creating monopolies. The law could simply mandate that municipalities must offer franchise licenses to all ISPs if they offer licenses to one and that all licencees must be be charged at the same rate.
The only reason we have cable monopolies in the U.S. is because the Cable Communications Act of 1984 created that perverse incentive. Other countries without such laws have much faster service at much lower prices.
If federal law permitted local governments to do this sort of thing with groceries, computers and cars we would have regional monopolies for those products as well. Be grateful that your town council is not permitted to sell grocery, computer and car franchises.
Ceci n'est pas une signature.
Except you are confusing a transit and a consumer endpoint. Transit providers normally peer, but an endpoint is going to have more traffic coming in then going out because their consumers are requesting it, ALWAYS, but this is the first time they have been able to pressure people into these types of agreements.
Peering agreements between transit providers is fine, but not when an endpoint bullies a service providor.
When you cant win, ad hominem.
No, I'm not confusing transit and endpoints.
You are right that the end point connection isn't one sided. That's why we all pay monthly fees to our provider to pay for bandwidth. Our provider, though, has peering agreements with other providers. If this peering is unequal, some of the money that we pay our provider goes to pay for peering fees. In the case of Netflix, their provider refused to pay the peering fees. If our provider screws up, we suffer. We then have a choice to change providers. Netflix chose to pay Comcast instead (probably the cheaper option, factoring termination fees, etc.). But... they did have other options...
The problem is that the backbone provider they chose sends way more traffic than they accept.
And consumer ISPs give asymmetric speeds most of the time with EULAs that forbid running servers. It's pretty obvious that they'll accept more data than they send by design, so it's unreasonable for their peering agreements to assume symmetric transfers.
When a toll road sells 10.000 people a monthly pass for a lane each, and then offers up to 5 lanes at a time during peak hours due to "maintenance", and yet has another 9,995 lanes constructed, uncluttered, and for sale in advertisements to other bidders I would certainly complain.
Inheritance is the sincerest form of nepotism.
The requestor pays. That was the old system. They are now finding out ways of blackmailing the sender into paying. My fees to my ISP pay for all the bandwidth needed to get the bits from Netflix to me. Unless my ISP blackmails Netflix.
Learn to love Alaska
Since when were fibre cables, $20000 optics, Switch ports, and 40-Gigabit port licenses free when the link is turned off?
Not free, but Cogent is willing to pay these costs itself. Verizon and Comcast won't take Cogent's offer; they want to charge Cogent an arguably excessive markup on top of Cogent's costs