Slashdot Mirror
Verizon's Accidental Mea Culpa
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?
https://www.eff.org/https-everywhere
Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.
Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.
Waiting for an amusing sig.
Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.
Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.
Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.
Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."
"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."
I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.
Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.
Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.
I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)
I came, I conquered, I coredumped
What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
They obviously didn't understand the presentation so they assumed no one else would either.
Verizon’s Accidental Mea Culpa Mark Taylor / 17 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.
His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion
Customers.
You're paying for a service, and nowhere does it say that they will discriminate against a particular service, such as Netflix.
It's obstructive business, against your customer's best interests, for no particular reason. It will also violate any given "net neutrality" laws that are / may come into effect.
Those laws are the answers. The reason for their existence is this sort of unnecessary posturing. And governments make companies do a lot of things against a company's best interests - all the time. It would be in the company's best interest to not pay tax, screw over its customers, not ship goods that have been paid for, be monopolistic, collude with others to enforce market prices, etc. The laws are brought in to stop that shit in the PEOPLE'S best interest, not the company's.
Not saying it's anywhere near perfect, but your post seems to want to back a corporation screwing over its customers and then (falsely) blaming its competitors and random third-party companies for that.
Sure. The content streaming from Netflix has been requested by Verizion customers. They've paid for access to the internet, which includes Netflix. They are the ones being throttled. Basically Verizon is trying to double dip here - get money from regular customers plus shaking down more from content providers. If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?
...because maybe Verizon's paying customers want it and expect to get what they pay for?
I know. Serving your customers. Totally alien concept.
Sure, the governments should break up the ISP monopolies that restrict access to only one or two ISPs. They should also try to reduce any barriers to entry in the market for new competitors to spring up. This would increase competition and force verizon to better their service to retain their customers.
The issue with this is that its good for consumers but bad for investors, and we all know who our esteemed congress men actually represent.
Here is a copy of the text, just in case:
Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.
His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
lvltvzw
Verizon has
\m/
Cached, so you can see the image: http://webcache.googleusercont...
A black cat crossing your path signifies that the animal is going somewhere. -- Groucho Marx
Its L3's blog, not Verizon's.
So when Netflix decided to pay Comcast, they were able to upgrade all of those remote trunks in ~24 hours, even though they cost of fortune?
Got anything better?
Remove the laws and regulations holding back community fiber projects.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
Perhaps a rule where cable or satellite TV providers are prohibited from operating centralized peering points. If Verizon had to buy their bandwidth from upstream providers, they wouldn't be able to choke L3. And L3 would have to bid against Verizon's upstream providers to get Netflix's business.
Essentially, less economic centralization in the network infrastructure would provide for more opportunities for competitive bidding all along the chain. Everyone would end up with more customer-focused incentives.
"I want it and my government friends have guns..." Is this the best we can do?
The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies, federal grants and subsidies, and the reason they even exist at all is because of a government approved corporate charter. Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?
If you want a vision of the future, imagine a youtube comments section scrolling - forever.
Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.
I really miss working with them.
Serious? Seriousness is well above my pay grade.
Laws are one of the tools we use to align the interests of groups that are potentially in conflict. Why aren't they a valid option here?
+1. Politicians at the state level have been paid off by cable companies and ISPs to squash competition from local municipalities. The cities who got in before the legislation are loving their services. It is absolutely insane what money can do in the political process.
Even if that were the case, Verizon didn't sell its customers a 50Mbps network connection that only gets 5Mbps to Netflix during peak hours. They marketed and sold their service at the full speed and have been completely happy to take our money despite the problems. The bottom line is that they need to fix their shit. Yes it may cost a lot of money but the last time I checked, Verizon was doing just fine and can definitely afford it.
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
And a government's responsibility is to take action against a company which is committing wholesale fraud against its customers by selling them Internet Service which promises bandwidth speeds which they are then purposefully not providing in order to shake down their customers and companies trying to provide services to those customers more money.
A government's responsibility is to ensure that companies that are given government licenses and franchise agreements which restrict competition in certain geographic areas are providing the service that the people of that area want and need at a fair price.
A government's responsibility is to ensure that companies which get too big, hold too much market share and are too horizontally or vertically integrated are broken up so that there can be real competition and a real free market.
"it's Verizon's customers who are causing all this bandwidth usage"
Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.
And then Verizon passive aggressively acts like the problem is not of their own making.
And I'm tired of people who claim that anything but greed is a problem. The graph in the blog linked here clearly shows that there are no internal network problems at Verizon, everything is below 65% peak utilization - and those are only the metro and backbone routers, which are few and far between. Everything else is at ~50% peak utilization. No immediate upgrades are required, except the interconnect to L3, where both L3 and Verizon have spare capacity, and L3 has even offered to pay for the upgrade.
If you are paying for 75/25 or 50/25 and they are throttling it at the borders of their network, then you aren't getting the bandwidth you are paying for... downgrade your service. That $10, $20 or more per month they aren't getting from you because of their throttling practices should get their attention.
So why does running a VPN fix this? A To a CO there is no difference between Netflix or a VPN connection.
I wish it was that simple, but I'm on board with the general idea. I wouldn't publicly cut the cables. That's too extreme. I would, however, like to see Level 3 turn the tables and publicly (as noisily as possible) accuse Verizon of using up all their bandwidth and that if Verizon doesn't help them pay for the costs of upgrading, their customers just won't be able to watch Netflix anymore. "I don't see why Verizon is taking issue with this. It's standard practice. People download a lot of video from us. Remember folks, we're not downloading from them! Verizon's users are downloading from us! They're the ones using all the traffic!" (I know that's a completely inaccurate and misleading explanation of the situation - an outright lie, if you will - but that's the point.)
If Verizon customer's thought "Who's Level 3? Netflix is paying them so I can watch Netflix, and now they want to charge Verizon money so I can watch Netflix? So basically I'm getting charged twice to watch Netflix? This will not stand!"
Then when everybody has turned their heads in their direction, Level 3 would say, "Just kidding! Here's what's really going on..." and tell people a simplified version of what they said in TFA. Then maybe people would start to care about net neutrality. (If you were to replace Level 3 with Netflix themselves, it would oh, so much better!)
Of course, it's a good thing I'm not in charge of either company because I'd have just lost an unprecedented amount of business and ruined the company's reputation in the process, and unfortunately, the average person isn't going to respond a calm, well-reasoned, fact-based argument like TFA. Most people just aren't going to care as long as there's something playing on TV. When you get in the way of that, people will definitely notice, but anybody even remotely involved in such an affair will be ruined for life.
Individuals, and individual rights, are like single atoms. They only exist in the abstract sense. The real world is entirely dominated by groups and collective actions.
You're a confused anarchist. The problem with non-coercive government is that all government is coercive. Government is primarily a set of restrictions on the use of force, or alternately the monopoly on that use of force. Getting rid of a government, or disarming it, merely allows anyone with a larger arsenal to set up their own government -- anarchy is an unstable system. We all have a right to violence, because it cannot be taken from us except in extreme situations. Remember, the Code of Hammurabi was instituted, "...so that the strong might not harm the weak." Coercive government is a necessary evil, and it will remain necessary so long as men are capable of harming their fellows, for that is its justification and primary purpose.
Rights are not inherent, except in some abstract sense. In the real world, your rights are what the men with guns say they are. You may feel fortunate that the world has had a long, bloody time to work out semi-cooperative frameworks to restrain our darker impulses. Individual rights are an important conceptual counterbalance to the overwhelming powers of the collective, but they are no justification for anarchy, economic or otherwise. The "free market" is an ideal, even a good one, but in most cases removing government interference makes markets less free, more subject to collusion and fraud. In some cases, where the service is required to be universal, or when the barriers to entry would be insurmountable, it makes sense for the government to assume these functions directly. Govenment can also be thought of as the natural monopoly of natural monopolies, in that sense.
Slavery is a word that has a specific meaning; your definition is specious. You just fundamentally don't like being told what to do. To some degree this idealism is admirable. For the true individualist, I can recommend (from long personal experience) the Alaskan wilderness; you can get land for free still up there, provided you build upon it. Whatever romantic images your mind conjures upon thinking of Alaska are all true; I can't stand the weather, personally, but it's as close to a pure state of nature as you will ever find. If you'd like to enjoy the benefits of society, however, you have to play by the rules. "Slavery" isn't an option -- it's mandatory.
Those who advocate genocide deserve every protection afforded by law, and none afforded by common human decency.