Slashdot Mirror


Verizon's Accidental Mea Culpa

Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).

22 of 390 comments (clear)

  1. But scarcity! by StikyPad · · Score: 5, Insightful

    If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?

    1. Re:But scarcity! by StikyPad · · Score: 5, Informative

      Verizonâ(TM)s Accidental Mea Culpa
      Mark Taylor / 18 hours ago
      David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflixâ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizonâ(TM)s own paying broadband consumers.

      His explanation for Netflixâ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

      The diagram then shows this one little bar, suggestively color-coded in red so you know itâ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

      But hereâ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizonâ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatâ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.

      So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

      Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3â(TM)s network interconnects with Verizonâ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

      So letâ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the con

    2. Re:But scarcity! by Anonymous Coward · · Score: 5, Informative

      Google's cached copy...

      http://webcache.googleusercontent.com/search?q=cache:DBHDyx7n4D0J:blog.level3.com/global-connectivity/verizons-accidental-mea-culpa/+&cd=1&hl=en&ct=clnk&gl=us

    3. Re:But scarcity! by ffsnjb · · Score: 5, Insightful

      In what world do you live that Level3 is a "much smaller ISP"? Level3 is a global tier 1 ISP, FFS.

      --
      "Why do you consent to live in ignorance and fear?" - Bad Religion
    4. Re:But scarcity! by StikyPad · · Score: 5, Insightful

      Apples to oranges. Level3 and Cogent aren't last-mile providers; they're Tier 1 backbone providers. Tier 1 providers have things like peering agreements -- last mile providers do not. Last mile providers are (and sell) unbalanced connections, so it's impossible for them to ever have "peers."

      A better way of thinking of it is that Verizon should be representing the interests of its customers, because Verizon is the gateway between the customers, and the rest of the internet. It's not doing that job -- it's trying to play both sides against each other. This is what middlemen do, of course, and they're entitled to do it, but as long as they have a monopoly (which they do), then there should be limits, oversight, and accountability.

    5. Re:But scarcity! by Shados · · Score: 5, Insightful

      The problem is still the lack of competition in the market. If everyone had the choice between 4-5 ISPs, considering the popularity of Netflix, consumer ISPs would be paying Level 3 truckloads of money to ensure Netflix works flawlessly...and the roles may even be reversed (where Level 3 tries to gouge Verizon, since they'd know Verizon would have no choice or lose a ton of customers).

      But since there isn't any competition, Verizon takes their own customers hostages...

    6. Re:But scarcity! by EvilSS · · Score: 5, Funny

      It looks like the Level 3 post has been pulled. It goes to their 404 page which has a link to recent posts which lists the very post linked in the article.....and the recent post link ALSO takes you to a 404.

      Strange, the link works fine for me. Your ISP isn't Verizon by chance is it?

      --
      I browse on +1 so AC's need not respond, I won't see it.
    7. Re:But scarcity! by profplump · · Score: 5, Insightful

      Why don't I want city-owned fibre? I'm a big fan of city-owned roads and city-owned sewer pipes.

    8. Re:But scarcity! by Anonymous Coward · · Score: 5, Interesting

      [...] Level3 and Cogent aren't last-mile providers; they're Tier 1 backbone providers. Tier 1 providers have things like peering agreements -- last mile providers do not. [...]

      Except that Verizon Enterprise (formerly Verizon Business) is also a Tier 1 backbone provider. Different part of the company, but the behavior does appear to be a conflict of interest, of exploiting the Verizon's ISP (last-mile) business actions (failing to resolve congestion to L3) to make a competitor (L3) to Verizon Enterprise (formerly UUnet, AS 701 / 702 / 703) less desirable to Level 3 customers, namely Netflix.

      Arguably, Verizon is abusing its ISP customers as pawns in making a competitor to one its Enterprise IP business less desirable, in a very anti-competitive fashion.

  2. No excuses left by Anonymous Coward · · Score: 5, Insightful

    Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.

  3. PR needs to talk to tech by MrDoh! · · Score: 5, Insightful

    Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.

    --
    Waiting for an amusing sig.
  4. Connect with a VPN by LinuxFreakus · · Score: 5, Interesting

    Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.

    1. Re:Connect with a VPN by i.am.delf · · Score: 5, Interesting

      You can also escape this bottleneck using an IPv6 tunnel to he.net.

    2. Re:Connect with a VPN by Anonymous Coward · · Score: 5, Insightful

      According to tfa, they actually aren't throttling. Throttling implies that they are deliberately shaping traffic inside their network to limit your bandwith.

      What they are really doing is deliberately creating a bottleneck at key peering locations through negligent inaction when it comes to upgrading infrastructure.

      Small difference, I know, but very important when you actually talk about throttling, and likely the argument they would make if the FCC took them to court over it.

  5. Re:In Verizon's defense by Taeolas · · Score: 5, Funny

    Actually, they did. Verizon has just yet to deliver. Apparently they don't expect to deliver until the end of the year in any case.

    Which this article seems to implies it takes Verizon a year to send a technician to 7 cities to connect up a few cables between routers. (And / or maybe install a couple of cards). Maybe Verizon should stop having their techs travel by horseback, they might get it done faster.

  6. Verizon's Response by CanadianRealist · · Score: 5, Funny

    Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more

    Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."

    "Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."

    1. Re:Verizon's Response by idontgno · · Score: 5, Funny

      If this scenario made sense, you'd see Cisco routers with magazine-fed 10gb cards. Automatically eject a spent card and load the next.

      That may be a rare example of an expendable with a higher per-unit and per-use price than HP inkjet cartridges.

      --
      Welcome to the Panopticon. Used to be a prison, now it's your home.
  7. Re:Answer needed by doug · · Score: 5, Insightful

    Sure. The content streaming from Netflix has been requested by Verizion customers. They've paid for access to the internet, which includes Netflix. They are the ones being throttled. Basically Verizon is trying to double dip here - get money from regular customers plus shaking down more from content providers. If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?

  8. Re:Answer needed by bill_mcgonigle · · Score: 5, Insightful

    Got anything better?

    Remove the laws and regulations holding back community fiber projects.

    --
    My God, it's Full of Source!
    OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
  9. Re:Answer needed by AnontheDestroyer · · Score: 5, Insightful

    +1. Politicians at the state level have been paid off by cable companies and ISPs to squash competition from local municipalities. The cities who got in before the legislation are loving their services. It is absolutely insane what money can do in the political process.

  10. Re:I disagree by suutar · · Score: 5, Insightful

    Costs should be driven by the party responsible for the traffic being on the network. In the case of neflix traffic, that's _me_, the end recipient. And I've already ponied up to the cable company to cover their cost to transfer the bits to me. The cable co just wants to double dip.

  11. Re:I disagree by BronsCon · · Score: 5, Insightful

    Bingo. Well, almost; it's a little more nuanced than that. Costs should be driven by the party responsible for the traffic being on *your* network. For Verizon, that's Verizon's customer; for Level3, that's Netflix. And they both already pay their providers. Where Verizon and Level3 peer, it's a matter of recognizing that the imbalance of traffic across that link is caused by Verizon's customers requesting more traffic than they (can) return. Thus, Verizon caused the imbalance and should therefore pay for it. If Verizon primarily sold symmetrical access and allowed their users to run servers, there would likely be a balance, and if there was not, they'd have a leg to stand on here, but they don't sell symmetrical access to the end user and they don't have a leg to stand on in this debate; what they do have is a monopoly on Verizon customers, which they're attempting to abuse right now, which should warrant an anti-trust suit, if anything. No additional regulation needed.

    --
    APK quotes people (including myself) without context and should not be trusted. Just thought you should know.