Slashdot Mirror


First Detailed Data Analysis Shows Exactly How Comcast Jammed Netflix

An anonymous reader writes John Oliver calls it "cable company f*ckery" and we've all suspected it happens. Now on Steven Levy's new Backchannel publication on Medium, Susan Crawford delivers decisive proof, expertly dissecting the Comcast-Netflix network congestion controversy. Her source material is a detailed traffic measurement report (.pdf) released this week by Google-backed M-Lab — the first of its kind — showing severe degradation of service at interconnection points between Comcast, Verizon and other monopoly "eyeball networks" and "transit networks" such as Cogent, which was contracted by Netflix to deliver its bits. The report shows that interconnection points give monopoly ISPs all the leverage they need to discriminate against companies like Netflix, which compete with them in video services, simply by refusing to relieve network congestion caused by external traffic requested by their very own ISP customers. And the effects victimize not only companies targeted but ALL incoming traffic from the affected transit network. The report proves the problem is not technical, but rather a result of business decisions. This is not technically a Net neutrality problem, but it creates the very same headaches for consumers, and unfair business advantages for ISPs. In an accompanying article, Crawford makes a compelling case for FCC intervention.

21 of 243 comments (clear)

  1. I'd like to hear Bennett Haselton's opinion by Anonymous Coward · · Score: 5, Funny

    Bennett Haselton is a frequent contributor, and I tend to hold-off judgement on these things until I read his fine points on the topic. I was recently particularly moved by his work on line queues at Burning Man. It changed my entire life. I now piss sitting down.

  2. Sigh... by MightyMartian · · Score: 5, Insightful

    "In an accompanying article, Crawford makes a compelling case for FCC intervention."

    That won't work unless it comes with a check with seven digits attached to it.

    --
    The world's burning. Moped Jesus spotted on I50. Details at 11.
    1. Re:Sigh... by Em+Adespoton · · Score: 5, Insightful

      And that won't really work because an even larger check from the cable companies will always follow. Hell they most likely just give them a credit card with an unlimited balance that draws right from the cable companies' customers' bank accounts.

      FTFY

  3. Common Carrier by Anon-Admin · · Score: 5, Insightful

    Once again, a call for net neutrality will ensue. All we really need is for the FCC to call them Common Carriers and apply the age old law.

    It has already been applied to Telecoms and Utilities, just apply it to the ISP's and be done with this crap.

    http://en.wikipedia.org/wiki/C...

    1. Re:Common Carrier by Burdell · · Score: 4, Informative

      That would have zero impact. This is like the telephone company in city A have 96 channels to the telephone company in city B, but then 100 people try to make calls. Only some of them will go through, and that's a capacity issue, not regulated by Common Carrier status. They are not discriminating based on callers or anything, they are just "decliining" to upgrade capacity. In some cases, that could be regulated by state PUCs/PSCs, but AFAIK it is not normally. It is just up to the two carriers to reach an agreement.

      This type of thing happened a lot in the early dialup ISP days, when telecom deregulation spawed a lot of CLECs that had to connect to ILECs to carry calls. The ILECs structured the contracts with settlement money for to flow to the destination of a call (thinking most of the CLEC calls would be _to_ ILEC users), but then the CLECs went and got all the dialup ISPs to move modem banks to them. Suddenly all the calls went _to_ the CLECs, and the ILECs had to pay (some did not and went to court instead).

    2. Re:Common Carrier by Jason+Levine · · Score: 5, Insightful

      It's more like Phone Company A has 100 channels to Phone Company B. They notice that there is a constant level of 99 calls between A and B. They also see that 25 calls are constantly being made to Joe's Pizzeria who uses Phone Company B for phone service. Phone Company A gets upset that Joe's isn't paying them (Phone Company A) for this traffic, so they refuse to add more channels. (Even though doing so would be inexpensive to do.) So calls to Joe's Pizzeria begin being dropped and Joe's customers get mad that they can't get through. Joe's finally signs an agreement with Phone Company A paying them money and suddenly the calls go through just fine.

      This is extortion plain and simple. Add in the fact that the ISPs doing this have an Internet monopoly/duopoly in their areas and also tend to provide video services - that Netflix competes against - and you have extortion plus the use of a monopoly to crush competition in another market. This deserves swift and severe action to show the ISPs that this is NOT to be tolerated. Unfortunately, at best we'll get a strongly worded statement and perhaps a fine that Comcast will make back in the time it took me to write this comment.

      --
      My sci-fi novel, Ghost Thief, is now available from Amazon.com.
  4. WHy net neutrality doesn't work by lgw · · Score: 5, Insightful

    Stuff like this is why I think Net Neutrality discussions miss the mark - you're not going to fix the problem that way, you're only going to cause the cable companies to achieve the same throttling through other technical means. Trying to close technical loopholes via the lawmaking process requires a body of law the size of the tax code.

    The fundamental problem is that companies with a legally-granted monopoly for delivering high-speed internet are also allowed to sell content. In a free market, that wouldn't bother me - competition would sort it all out. But "last mile" is about as far from a free market as you can get in most of the country these days, and so we get this as a result.

    Last mile needs to become a public utility. Let vendors compete for my business, and I'll pick "just a pipe" or a content company or whatever mix fits my needs.

    --
    Socialism: a lie told by totalitarians and believed by fools.
    1. Re:WHy net neutrality doesn't work by nine-times · · Score: 4, Insightful

      The fundamental problem is that companies with a legally-granted monopoly for delivering high-speed internet are also allowed to sell content.

      I agree with this part of your post, at least, and have been making the same argument for years. If the companies providing the infrastructure were not making money from selling content, and were only serving as "dumb pipes", then their business incentive would be in pushing customers toward higher-bandwidth (and therefore more expensive) connections. In that business model, companies that can provide content to saturate slow connections become very important, and so it seems likely that they would be falling all over themselves to provide a better connection to Netflix.

      Instead, the Infrastructure providers have no incentive to increase content availability, because any piece of available content becomes competition for the content that they are trying to sell. That's a bad system. Unless you have an effective regulatory system, the ISPs will find ways to push towards a walled garden AOL-style internet, charging for access outside of the walled garden.

      However, I don't think this is an example of "net neutrality" missing the mark. Net neutrality is a concept, and divorcing infrastructure providers from content providers is one way in which net neutrality could be promoted.

    2. Re:WHy net neutrality doesn't work by nehumanuscrede · · Score: 5, Interesting

      So the fix sounds easy, but gets complicated quickly. :|

      Break them up. Don't think about it, threaten or consider it. To borrow a phrase from Nike: JUST DO IT

      Force the companies who provide the bandwidth to split from the part of the company who sells content so there isn't any grey area issues. In order for the bandwidth side of the house to remain competitive, they'll need to upgrade their network to ensure they're delivering what their customers want. This also prevents the content folks from tweaking the network to ensure their own services get priority over competing ones. Taken to an extreme, and without some sort of oversight, the big players can absolutely destroy the competition in this manner.

      They've already proven they can't be trusted to do the right thing themselves. Time to step in with a big stick and start swinging.

      Of course, true competition needs to be in place to help fuel that fire otherwise we'll still end up with crappy throughput with no realistic choices of switching to another provider. We really need to end this whole regional mono / duopoly thing.

      On top of this, we'll need to purge the Congressional votes-for-contributions types to make sure the legislation has a fair chance of happening.

  5. Yes it is a peering problem ... by jschultz410 · · Score: 4, Informative

    and not a net neutrality issue thankfully.

    Settlement free peering between tier 1 carriers only happens when the flow of traffic is roughly balanced between the contracting peers.

    When one peer is pushing a lot more traffic onto the other network, then that usually goes out the window and the pusher is required to pay the receiving network. Otherwise, networks would be monetarily incentivized to unload traffic they should carry on their own networks onto their peers' instead.

    1. Re:Yes it is a peering problem ... by Burdell · · Score: 4, Informative

      Traffic balance is not the primary measure these days (from what I understand), it is just an economic decision. However, the Netflix case is interesting, because they were essentially used as a leverage tool by Cogent against the other carriers. Cogent has a long history of trying to get settlement-free peering, not meeting contract terms (whatever they are), getting dropped, and then blaming the other side. They have long wanted to be a settlement-free "tier 1" provider (which is a nebulous term, but go with it), but have generally not been. They sell bandwidth often at below-market rates in order to attract customers to leverage against the other "tier 1" providers. They saw Netflix on the rise and grabbed them, apparently selling bandwidth much cheaper than any other backbone (possibly at a loss even) in order to leverage settlement-free peering contracts out of other providers.

      Any network engineering with a clue knows that you never buy bandwidth only from Cogent (or even Cogent and one other provider), because you _will_ get disconnected from somebody when Cogent gets in another peering dispute.

    2. Re:Yes it is a peering problem ... by Anonymous Coward · · Score: 5, Insightful

      When one peer is pushing a lot more traffic onto the other network, then that usually goes out the window and the pusher is required to pay the receiving network.

      They're not pushing the traffic - the other network is pulling it. Netflix's traffic is not unsolicited - every goddamned packet is being sent in response to a specific request from the other network's customers, and it's not fucking transit - the Netflix packets will terminate within the receiving network. Are you seriously arguing that Comcast should be paid by Netflix because they're carrying gigabytes of Netflix traffic their own fucking customers requested?

    3. Re:Yes it is a peering problem ... by Obfuscant · · Score: 4, Informative

      Why does imbalance matter?

      Because when peering agreements were created, the assumption was that trying to keep track of how much data you wanted us to carry and you keeping track of how much we gave you to carry was not necessary because we'd be charging each other the same amount if we did keep track.

      Large amounts of data going one way breaks that basic tenet of peering. Now it makes sense to charge the other guy for data they want you to handle.

      But this view ignores the most important point, that Comcast has explicitly promised its customers "internet access" at an advertised speed.

      No. From here:

      Performance Starter: Offer ends 01/04/15. Restrictions apply. Not available in all areas. Limited to new residential customers. Requires subscription to Performance Starter Internet service. Equipment, installation, taxes and fees, including regulatory recovery fees, Broadcast TV Fee (up to $3.50/mo.), Regional Sports Fee (up to $1/mo.) and other applicable charges extra, and subject to change during and after the promotion. After applicable promotional period, regular rates apply. Comcastâ(TM)s current monthly service charge for Performance Starter Internet is $49.95 (pricing subject to change). Service limited to a single outlet. May not be combined with other offers. Actual speeds vary and are not guaranteed.

      Emphasis mine. The same emphasized text appears in the details for all three residential service levels. My statement stands: they did not promise you 24/7 full-rate access to anyplace off their net.

      There are no pharmaceutical-like disclaimers during those commercials

      There are when you actually go to sign up. And common sense tells you that they cannot guarantee those speeds to every site on the planet. They can't even guarantee those speeds to every site on the Comcast network. That's why they don't.

      If Comcast says you're paying for "10Mbps internet", the assumption is that you get the advertised speed to the entire internet, provided there are no technical limitations outside of Comcast's control.

      That's what some people assume, but that's not backed up by the service agreement.

      It's not even backed up by common sense. Suppose you buy the Blast service and get 105Mbps download. You want to connect to my system and I've got Performance Starter (6 Mbps down, God knows what it is up). You ain't getting anywhere near 105Mbps from my stream. Even trying to connect to your next door neighbor who has the same service, you ain't getting faster than his upload allows. If you think Comcast could promise anything faster, then you must think they'll upgrade MY service to Blast for free because they promised YOU that you'd get data from me that fast, and you're paying them for my data at that speed.

  6. Net neutrality is a constituional right! by Anonymous Coward · · Score: 5, Funny

    I should be able to use my bandwidth any way I please! It's freedom of speech! They can't throttle netflix! George Bush would be rolling in his grave!

    This is a ploy from Obama to help spread misinformation about ebola (Obola) on his path to further to destroy the country!!1!

  7. Re:How many engineers does it take to screw netfli by MozeeToby · · Score: 5, Insightful

    Engineer - "Hey Boss, we need some cash to upgrade the connection to these networks."
    Boss - "What?! We just upgraded those connections a couple years ago"
    Engineer - *rolls eyes* "Well the link is saturated, looks like lots of people watching online video... Netflix comes in over this connection so it makes sense"
    Boss - "First they take our subscribers now they're forcing us to upgrade our equipment... well fuck em!"
    Engineer - "Waaah?"
    Boss - "You heard me, fuck em!"
    Engineer - "But... our customers will get terrible service when they try to watch Netflix, or do anything else on that network for that matter"
    Boss - "Exactly!"

  8. This is kind of relevant... by Svartalf · · Score: 5, Funny
    --
    I am not merely a "consumer" or a "taxpayer". I am a Citizen of the State of Texas
  9. Re: Multiple CDN contracts? by bill_mcgonigle · · Score: 5, Informative

    Netflix is its own CDN - they will give, for free, one or more caches to any ISP, causing any one movie to transit the ISP's nonfree network connections only once.

    But this is about competition for video services, not caching.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  10. Re: Multiple CDN contracts? by killfixx · · Score: 5, Insightful

    This is a very salient point. Netflix already has these arrangements with other ISPs. Only Comcast, AT&T, and Verizon (surprise surprise) refused to host local caching servers. Of course, this precedes their demands for more money because, "Waaaahhhh...they're stealing our customers, they need to pay!".

    Netflix tried to be the better entity (within reason) and were told, in no uncertain terms, "Go fuck yourself."

    Yay, free market!

    *sigh*

    --
    "Helping to keep you two steps ahead of the Thought Police!"
  11. Re:How many engineers does it take to screw netfli by torkus · · Score: 5, Insightful

    Actually netflix offered to foot the bill for upgrading the bandwidth - it's literally a couple cross-connects in a datacenter, maybe a fiber card or two.

    Oh, and netflix ALSO offers to drop a server in your datacenter *free* which caches all the common netflix streams. This reduces the internet bandwidth demands by something like 90+% since it lives within the ISP's datacenter and just needs to download each stream once.

    But the last line is exactly the point. The ISPs are also TV providers and they don't want you to have a good netflix experience. If they can passively let that happen...well of course they will. No one can accuse them of taking any action to damage your netflix streaming...it's their complete inaction that's resulting in it.

    --
    You can get rich if you own a politician, but you have to be rich to buy one in the first place.
  12. End the ISP monopolies by Jodka · · Score: 4, Interesting

    from wikipedia

    Franchise fees are governed under Section 622 of the Cable Communications Act of 1984.[2] Section 622, states that municipalities are entitled to a maximum of 5% of gross revenues derived from the operation of the cable system for the provision of cable services such as Public, educational, and government access (PEG) TV channels.

    Franchise fees are fixed at a maximum of 5% of gross revenues. So how do municipalities maximize revenues from franchise fees? By maximizing cable company gross revenues. And how do municipalities maximize cable company gross revenues? By creating monopolies! By awarding exclusive license to one provider to extract monopolist profits from the public.

    Note that there is nothing inherently wrong with permitting local governments to charge cable companies fees. That is justifiable to the extent that local governments incur costs of infrastructure repair with damage from cable installation. All that is needed is a single addendum to the law, one prohibiting local governments from creating monopolies. The law could simply mandate that municipalities must offer franchise licenses to all ISPs if they offer licenses to one and that all licencees must be be charged at the same rate.

    The only reason we have cable monopolies in the U.S. is because the Cable Communications Act of 1984 created that perverse incentive. Other countries without such laws have much faster service at much lower prices.

    If federal law permitted local governments to do this sort of thing with groceries, computers and cars we would have regional monopolies for those products as well. Be grateful that your town council is not permitted to sell grocery, computer and car franchises.

    --
    Ceci n'est pas une signature.
  13. Re:Their answer to oversubscription as well by thaylin · · Score: 4, Interesting

    Except you are confusing a transit and a consumer endpoint. Transit providers normally peer, but an endpoint is going to have more traffic coming in then going out because their consumers are requesting it, ALWAYS, but this is the first time they have been able to pressure people into these types of agreements.

    Peering agreements between transit providers is fine, but not when an endpoint bullies a service providor.

    --
    When you cant win, ad hominem.