Slashdot Mirror
Verizon's Accidental Mea Culpa
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?
https://www.eff.org/https-everywhere
Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.
Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.
Waiting for an amusing sig.
Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.
Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.
Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."
"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."
What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
They obviously didn't understand the presentation so they assumed no one else would either.
Customers.
You're paying for a service, and nowhere does it say that they will discriminate against a particular service, such as Netflix.
It's obstructive business, against your customer's best interests, for no particular reason. It will also violate any given "net neutrality" laws that are / may come into effect.
Those laws are the answers. The reason for their existence is this sort of unnecessary posturing. And governments make companies do a lot of things against a company's best interests - all the time. It would be in the company's best interest to not pay tax, screw over its customers, not ship goods that have been paid for, be monopolistic, collude with others to enforce market prices, etc. The laws are brought in to stop that shit in the PEOPLE'S best interest, not the company's.
Not saying it's anywhere near perfect, but your post seems to want to back a corporation screwing over its customers and then (falsely) blaming its competitors and random third-party companies for that.
Level 3 doesn't pay Comcast for bandwidth. Why should they? Comcast customers have already paid Comcast for the links to their house and they're the ones pulling data from Level 3. Level 3's customers pay Level 3 to deliver to the edge of their network. As the Level 3 post points out, the cost for Verizon to add more bandwidth between the Level 3 network and the Verizon network is minimal.
Sure. The content streaming from Netflix has been requested by Verizion customers. They've paid for access to the internet, which includes Netflix. They are the ones being throttled. Basically Verizon is trying to double dip here - get money from regular customers plus shaking down more from content providers. If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?
Sure, the governments should break up the ISP monopolies that restrict access to only one or two ISPs. They should also try to reduce any barriers to entry in the market for new competitors to spring up. This would increase competition and force verizon to better their service to retain their customers.
The issue with this is that its good for consumers but bad for investors, and we all know who our esteemed congress men actually represent.
Here is a copy of the text, just in case:
Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.
His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
lvltvzw
Verizon has
\m/
Part of the issue is that Verizon is a last-mile network, and does not sell symmetric bandwidth to its subscribers. So, the typical agreement between providers - where they each send about the same amount of traffic to each other and upgrade the interconnects to handle that traffic - will not work between Verizon and Level 3. Verizon (and the vast majority of other last-mile providers, including Comcast) will NEVER have a balanced interconnection with Level 3, because the home subscribers can all download far faster than they can upload.
Really, it's Verizon's customers who are causing all this bandwidth usage, so it should be Verizon ensuring that their interconnects can handle the requested bandwidth. If anything, Verizon (and Comcast) should be paying Level 3 for additional download capacity... but we all know that is never going to happen.
So when Netflix decided to pay Comcast, they were able to upgrade all of those remote trunks in ~24 hours, even though they cost of fortune?
Got anything better?
Remove the laws and regulations holding back community fiber projects.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
"I want it and my government friends have guns..." Is this the best we can do?
The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies, federal grants and subsidies, and the reason they even exist at all is because of a government approved corporate charter. Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?
If you want a vision of the future, imagine a youtube comments section scrolling - forever.
Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.
I really miss working with them.
Serious? Seriousness is well above my pay grade.
+1. Politicians at the state level have been paid off by cable companies and ISPs to squash competition from local municipalities. The cities who got in before the legislation are loving their services. It is absolutely insane what money can do in the political process.
Costs should be driven by the party responsible for the traffic being on the network. In the case of neflix traffic, that's _me_, the end recipient. And I've already ponied up to the cable company to cover their cost to transfer the bits to me. The cable co just wants to double dip.
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
And a government's responsibility is to take action against a company which is committing wholesale fraud against its customers by selling them Internet Service which promises bandwidth speeds which they are then purposefully not providing in order to shake down their customers and companies trying to provide services to those customers more money.
A government's responsibility is to ensure that companies that are given government licenses and franchise agreements which restrict competition in certain geographic areas are providing the service that the people of that area want and need at a fair price.
A government's responsibility is to ensure that companies which get too big, hold too much market share and are too horizontally or vertically integrated are broken up so that there can be real competition and a real free market.
Verizon to level3: "Our traffic from netflix moved over to Level3 last night... very strange, anyways we need to increase our capacity..."
Level3 to Verizon: "Ok, that will be $X"
Verizon to level3: "um... That's 300% higher than any other provider out there..."
Level3 to Verizon: "suck it... your monies are belong to us"
Except that this fictional exchange you've created, in which Level 3 is extorting Verizon for more, is easily refuted by using either blog post. For instance, from Verizon:
Netflix did not make arrangements to deliver this massive amount of traffic through connections that can handle it.
[...] Netflix is responsible for either using connections that can carry the volume of traffic it is sending, or working out arrangements with its suppliers so they can handle the volumes. As we’ve made clear before, we regularly negotiate reasonable commercial arrangements with transit providers or content providers to ensure a level of capacity that accommodates their volume of traffic.
Which is a nice way of saying, "Level 3 is refusing to negotiate rates for more capacity with us, so we've refused to give them more." Level 3's blog post also affirms that the issue is Verizon's refusal to act:
Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused.
Even without the blog posts, it should be obvious your notion makes little business sense. Level 3 is in no business position to play hardball like you've suggested. If they sacrificed on performance as a ploy to double-dip (i.e. get both Netflix and a lower-tier ISP* like Verizon to pay), Netflix would simply take its traffic to a different Tier 1 ISP that doesn't play those sorts of games, since the double-dipping would be hurting their bottom line. Or, at the very least, they'd be calling out their own ISP, rather than calling out the customer's ISP.
On the other hand, as a lower-tier ISP, Verizon has a monopoly on its own end users: if you want to reach them, you MUST go through them. If Verizon tries to double-dip by getting money out of both the higher-tier ISP and its end users, the end users won't understand what's going on, and in many cases they lack any viable alternatives anyway. Meanwhile, the higher-tier ISP can't switch out for a different peer, since Verizon is the only way to get to those end users.
Besides which, it's not like Netflix's switch from Akamai to Level 3 took Verizon by surprise, as you suggest, since it happened way back in 2010 and has been working fine for most of that time. If there was a problem resulting from the switch, it would have come up before now. Which is to say, this isn't a "Wow! Level 3's traffic is suddenly skyrocketing and we can't keep up!" situation. Rather, it's almost certainly a, "Hey, that Comcast company had a good idea to try getting money out of both sides...let's see if we can do it too!" situation, given the timing of it all.
* A quick aside: I'm well aware that Verizon also maintains a Tier 1 network, but Tier 1 networks rarely connect directly to end users. That's what lower-tier networks do. Moreover, the defining characteristic of a Tier 1 network is that it enjoys free peering with other Tier 1 networks. As such, the Verizon network being discussed here is clearly not their Tier 1 network, but rather a lower-tier one they control (e.g. a Tier 2 or 3 network) that has direct access to their end customers.
"it's Verizon's customers who are causing all this bandwidth usage"
Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.
And then Verizon passive aggressively acts like the problem is not of their own making.
Sorry that scenario doesn't fly:
http://www.cidr-report.org/cgi...
Netflix has peerings with:
AS2828 XO-AS15 - XO Communications,US (Tier 1)
AS55095 AS-NFLXCORP - Netflix Inc,US
AS3257 TINET-BACKBONE Tinet SpA,DE (Tier 1)
AS4436 AS-GTT-4436 - nLayer Communications, Inc.,US
AS3356 LEVEL3 - Level 3 Communications, Inc.,US (Tier 1)
AS16397 ALOG SOLUCOES DE TECNOLOGIA EM INFORMATICA S.A.,BR
AS26592 ALOG SOLUCOES DE TECNOLOGIA EM INFORMATICA S.A.,BR (Tier 2 - Has large footprint in latin america).
AS1299 TELIANET TeliaSonera International Carrier,SE (Tier 2 - Apparently the largest fiber providers in Europe).
AS174 COGENT-174 - Cogent Communications,US (Tier 1)
So no this isn't a case of exclusive peering, Level 3 being such a large provider just happens to be the best connection between Verizon and Netflix.
Secondly, that whole thing of 'Level3 to Verizon: "Ok, that will be $X"' has no bearing on a peering agreement, the statement would have been more like "The link between us is congested, want to upgrade the link?" each side upgrades their switch (if neccesary) and they connect the cable / fiber (given that they are in the same location we are talking about a multimode fiber patch at the high end).
If you are paying for 75/25 or 50/25 and they are throttling it at the borders of their network, then you aren't getting the bandwidth you are paying for... downgrade your service. That $10, $20 or more per month they aren't getting from you because of their throttling practices should get their attention.
Bingo. Well, almost; it's a little more nuanced than that. Costs should be driven by the party responsible for the traffic being on *your* network. For Verizon, that's Verizon's customer; for Level3, that's Netflix. And they both already pay their providers. Where Verizon and Level3 peer, it's a matter of recognizing that the imbalance of traffic across that link is caused by Verizon's customers requesting more traffic than they (can) return. Thus, Verizon caused the imbalance and should therefore pay for it. If Verizon primarily sold symmetrical access and allowed their users to run servers, there would likely be a balance, and if there was not, they'd have a leg to stand on here, but they don't sell symmetrical access to the end user and they don't have a leg to stand on in this debate; what they do have is a monopoly on Verizon customers, which they're attempting to abuse right now, which should warrant an anti-trust suit, if anything. No additional regulation needed.
APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
Ahh... I thought of this as soon as I clicked the button... If Netflix *really* wants to drive the point home, they can simply start peering or buying transit (more likely) from the providers Verizon has non-congested links with and stop routing to Verizon through L3 and Cogent. When Verizon refuses to upgrade *those* links, Netflix will be able to say "Either Verizon is refusing to upgrade their links, as we've been saying, or they only use providers which, as they claimed of Level3 and Cogent, can't handle the throughput their customers are requesting. In either case, this should be a wake-up call for Verizon customers to stop giving them money for a service they aren't, for whatever reason, delivering." And they shouldn't stop there; after that statement, they should re-enable routing to Verizon over all available links and watch the congestion continue; regardless of Verizon's response (which will likely be something along the lines of "Netflix performance continues to be slow because they have disabled routing to our network over multiple providers"), Netflix can stand up and say "We are currently routing to the Verizon network through every provider Verizon also uses, and make our routing decisions based on performance metrics, including packet loss and ping time to each user, to ensure that our users get the best possible experience we can provide. Unfortunately, as every link Verizon maintains appears to be congested, packet loss and ping times are high in all cases; the only solutions that exist are for Verizon to upgrade their links or peer with us, or for Verizon customers to find an alternate provider."
I don't think Netflix *wants* to fight dirty, or they would have done this already (and with Comcast, as well).
APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
Individuals, and individual rights, are like single atoms. They only exist in the abstract sense. The real world is entirely dominated by groups and collective actions.
You're a confused anarchist. The problem with non-coercive government is that all government is coercive. Government is primarily a set of restrictions on the use of force, or alternately the monopoly on that use of force. Getting rid of a government, or disarming it, merely allows anyone with a larger arsenal to set up their own government -- anarchy is an unstable system. We all have a right to violence, because it cannot be taken from us except in extreme situations. Remember, the Code of Hammurabi was instituted, "...so that the strong might not harm the weak." Coercive government is a necessary evil, and it will remain necessary so long as men are capable of harming their fellows, for that is its justification and primary purpose.
Rights are not inherent, except in some abstract sense. In the real world, your rights are what the men with guns say they are. You may feel fortunate that the world has had a long, bloody time to work out semi-cooperative frameworks to restrain our darker impulses. Individual rights are an important conceptual counterbalance to the overwhelming powers of the collective, but they are no justification for anarchy, economic or otherwise. The "free market" is an ideal, even a good one, but in most cases removing government interference makes markets less free, more subject to collusion and fraud. In some cases, where the service is required to be universal, or when the barriers to entry would be insurmountable, it makes sense for the government to assume these functions directly. Govenment can also be thought of as the natural monopoly of natural monopolies, in that sense.
Slavery is a word that has a specific meaning; your definition is specious. You just fundamentally don't like being told what to do. To some degree this idealism is admirable. For the true individualist, I can recommend (from long personal experience) the Alaskan wilderness; you can get land for free still up there, provided you build upon it. Whatever romantic images your mind conjures upon thinking of Alaska are all true; I can't stand the weather, personally, but it's as close to a pure state of nature as you will ever find. If you'd like to enjoy the benefits of society, however, you have to play by the rules. "Slavery" isn't an option -- it's mandatory.
Those who advocate genocide deserve every protection afforded by law, and none afforded by common human decency.