Slashdot Mirror
Verizon's Accidental Mea Culpa
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?
https://www.eff.org/https-everywhere
Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.
Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.
Waiting for an amusing sig.
Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.
Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.
L3's blog still has a summary blurb, but the link to the actual post gets a 404 - did they take it down or did they just link it wrong? Anybody have a cached copy?
Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.
Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."
"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."
I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.
Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.
Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.
I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)
I came, I conquered, I coredumped
What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
They obviously didn't understand the presentation so they assumed no one else would either.
How about because Netflix paid them to do it?
Stop whining about things we have a legitimate reason to bitch about!
Verizon’s Accidental Mea Culpa Mark Taylor / 17 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.
His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion
My company has Verizon and Cogent (and others). Our cogent offices have an absolute terrible time maintaining any type of stable throughput to our midwest data center running on Verizon. Often times we get speeds of several KB/sec for hours on end between them over our 100/200mbit connections. I've called Cogent and Verizon and complained for years. The bottleneck is the handoff from Cogent to Verizon. Verizon will not budge and claims no responsibility.
Interesting tidbit is the average speed for the last year shows Verizon --> Cogent is about 2x-10x the speed compared to Cogent --> Verizon on the same circuits. Verizon must not like to take on incoming Cogent data.
So maybe someone can explain this to me because I don't entirely get it.
Right now Level 3 doesn't pay Verizon any additional money for the data being sent its way (yes, requested by Verizon customers, but transport is usually paid by the shipper -- when I order a physical product I pay for shipping to the vendor, who pays the transporter).
The reason Level 3 doesn't pay any more is because they are using settlement-free links established to provide basic bi-directional communication between the two networks. Because of the way they are using them, these links (which are set up to provide balanced access) are saturated in one direction while only 30-60% utilized in the other direction.
The point made by both companies is that fixing the congestion is a simple matter of hooking up a couple ports (which would increase the utilization of Verizon's network).
Level 3 wants Verizon to agree to expand the settlement free ports to allow for the imbalance of traffic. Verizon says "our settlement free ports are sufficient for normal traffic, and if you want to avoid congestion for the additional traffic you are charging Netflix to carry then you're going to need to purchase additional ports and pay for that traffic."
Neither wants to budge and so they fight a PR war about it. Level 3 says "It's just a couple ports and a little cable" while disregarding the downstream impact on Verizon's network. Verizon says "Level 3 is taking undue advantage of our mutually beneficial arrangement and wants us to help them do it for free."
Is this accurate?
Customers.
You're paying for a service, and nowhere does it say that they will discriminate against a particular service, such as Netflix.
It's obstructive business, against your customer's best interests, for no particular reason. It will also violate any given "net neutrality" laws that are / may come into effect.
Those laws are the answers. The reason for their existence is this sort of unnecessary posturing. And governments make companies do a lot of things against a company's best interests - all the time. It would be in the company's best interest to not pay tax, screw over its customers, not ship goods that have been paid for, be monopolistic, collude with others to enforce market prices, etc. The laws are brought in to stop that shit in the PEOPLE'S best interest, not the company's.
Not saying it's anywhere near perfect, but your post seems to want to back a corporation screwing over its customers and then (falsely) blaming its competitors and random third-party companies for that.
Is this a resurrection of the Slashdot effect? Or did someone higher up at Level(3) notice this and pull the blog post?
Sure. The content streaming from Netflix has been requested by Verizion customers. They've paid for access to the internet, which includes Netflix. They are the ones being throttled. Basically Verizon is trying to double dip here - get money from regular customers plus shaking down more from content providers. If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?
Verizon seems to be saying that L3 needs to pay more because the bandwidth flowing into Verizon's network far outweighs the bandwidth flowing into L3's. While this is fundamentally idiotic for a last-mile ISP for any reason than money grabbing due to the asymmetric up/down bandwidth offered to last-mile customers, let's play their game. It sounds like all we need to do is send an equivalent amount of traffic back to Netflix, and everything will be right in the world. Even better, if we send more traffic to Netflix than we receive, L3 ought to be able to get money from Verizon because that traffic is out of balance in the other direction now.
...because maybe Verizon's paying customers want it and expect to get what they pay for?
I know. Serving your customers. Totally alien concept.
Sure, the governments should break up the ISP monopolies that restrict access to only one or two ISPs. They should also try to reduce any barriers to entry in the market for new competitors to spring up. This would increase competition and force verizon to better their service to retain their customers.
The issue with this is that its good for consumers but bad for investors, and we all know who our esteemed congress men actually represent.
because Verizon is being paid by their customers to access this data.
Why should verizon have any peering agreements with anyone?
Why not just operate their own network that doesn't peer out to the internet as a whole?
As a customer of an ISP, I'm not paying for the ISP to deliver me to their own internal network. I'm paying them to deliver my data to the peered connections. ALWAYS.
If my ISP has one peered connection that's beyond capacity, I expect them to upgrade it, as it's the cost of doing business.
And especially when it's cheap. This is chump change to them, and Level3 even offered to pay for it..... why is verizon saying no?
Just tried going to the link, it's been 404'ed.
Here is a copy of the text, just in case:
Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.
His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
lvltvzw
Verizon has
\m/
How about because customers are paying them for Internet service, and going to Netflix is where they want to go? People don't want to pay for bad service do they? The real problem here is Verizon is a competitor to Netflix, and Verizon is not only being allowed to be anti-competitive but also hoping to get paid for it.
I get tired of being the only person on slashdot that understands this...
The problem isn't the interconnect. The problem is between the local remote that feeds your house and the Central Office. When that much data comes from Netflix all at the same time, the remotes trunks can't handle it. Upgrading THOSE trunks costs a fortune. Throttling netflix at the peer reduces load on those trunks without affecting other services. That's what's going on and why Verizon (and others) are throttling Netflix. They have no other way of targeting netflix traffic directly without sending the FCC into a tizzy.
Netflix-------> Level3-------> Verizon core network-------> Verizon local CO----(the problem is here)---> remote-------> your house
You can argue that Verizon should fix that trunking themselves, we'd have a different argument then... but what this "Story" is about isn't even what's wrong. You can't just look at one section of the cheapest part of Verizons network and claim how easy it would be for them to fix. They've got a huge multi-billion dollar network to maintain and the front door to that network is the least of their concerns.
Customers.
They seem to have very limited interest in what their customers want for Netflix streaming quality. What is their incentive to care?
Those laws are the answers.
I covered that with "the government should threaten Verizon and force them to operate the network contrary to Verizon's best interests".
"I want it and my government friends have guns..." Is this the best we can do?
to make their customers happy. that is it. The customers are purchasing something, then finding out that they cant get what they thought they paid for. When customers find out that they can get some things at the speeds that they paid for , but not others, then it should be their ISPs job to provide that service. If there was true competition then this would happen quickly. Verizon doesnt really compete, so they can play these bullshit games.
Lets reverse your question. Why should people not be entitled to better netflix quality? Netflix provides enough infrastructure. Verizon claims to not have congestion issues WITHIN their network. People have PAID for a level of delivery that they arent getting. Your word entitled says a lot. this isnt a favor, this is what has been paid for.
Now lets play stupid analogy. If the only way to deliver a package was UPS, and you were forced to pay for 2 day shipping on everything, but it took a month to get the delivery complete, wouldnt you be pissed? I would. I paid for a service, and got a fraction of it, I was ENTITLED to the service, as our contract doesnt say 2 day delivery unless the delivery company just isnt in the mood, and in that case fuck you.
This is a different answer. Thanks.
Actually it's more like a company paying for x telephone circuits to their PBX, only Verizon made most of them 'party lines', so even though you can connect you're now having to talk and listen over both your fellow employees as well as their customer's replies in order to get anything done. However if you do a 'long distance' callout via calling card then call from there to your customer you get a perfectly free and clear line instead of a 'party line' :)
I hope this makes for a relatively accurate, concise, and entertaining telephone analogy.
The government should threaten them because they are not providing the service their customers have paid for and they advertised. The Internet works because tier 1's all connect to each other and upgrade ports as they get saturated (good ISP's are proactive, ok ones last minute and crappy ones only after the link is saturated). While this is related to net neutrality it's primarily a false advertising issue, if they change their wording to correct that it's a DPUC (or similar) issue as they are no longer providing internet access per their monopoly contract.
No sir I dont like it.
Dear Verizon Customers,
We were recently told we needed ~$100,000 worth of equipment. In order to pay for this we're going to need to add an interconnect service fee of $5/month to your bill. Sorry. Try a competitor? Hah, suck it.
Thanks,
Verizon CEO
PS We have also sued the company who told us this for saying it publicly, and we will most likely charge them more money, too.
The link to the LEVEL3 Blog post has been blocked...but modifying the link a bit pulls it back up again. Blog post reproduced below (with relevant image here)
VerizonÃ(TM)s Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns NetflixÃ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by VerizonÃ(TM)s own paying broadband consumers.
His explanation for NetflixÃ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know itÃ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But hereÃ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as VerizonÃ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatÃ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network Ã" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3Ã(TM)s network interconnects with VerizonÃ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So letÃ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of th
Got anything better?
Remove the laws and regulations holding back community fiber projects.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)
How about to make their fucking customers happy. I pay Verizon (because my only other choice is Comcast, and I hate them more). I request a service, I expect my provider to give me access to this service. Netflix pays L3, L3 is their service provider. Service providers peer, that is the way the internet has always worked.
I came, I conquered, I coredumped
It seems more like:
L3 provides bulk backbone data haulage to its customers, which they pay for. It provides the resources to deliver them to their destination. It will hand over packets to whichever ISP provides service to the packets' destination. All this, it does.
Verizon is supposed to provide a certain bandwidth to its customers, which they pay for. These are the end users. It does provide a link meeting bandwidth requirements between the customer and Verizon's internal network, which includes some services like Verizon's pay TV. However by failing to upgrade an at-capacity interconnect for which it is responsible it fails to provide the full bandwidth to their subscribers when they visit certain sites beyond Verizon's internal network, such as Netflix.
L3 appear to be doing all they are contracted to - delivering data to its destination ISP - and at a high bandwidth.
Verizon do appear to be falling short of their obligations as they have contracted to provide consumers a package based on provision of a certain bandwidth, but are failing to meet the terms of that contract at least for data from some parts of the internet by failing to upgrade parts of their infrastructure. The fact that this would increase Verizon's network utilisation is neither here nor there - if that is the cost of providing their clients with their contracted bandwidth then that is what they are bound to do.
The traffic isn't uniform in both directions. When it is, its an easy agreement. When its not, someone usually pays. Level3 offered to pay for the hardware to make additional connections.
But thats kind of like me telling verision that I'll pay for the router and won't charge them for using their network and expecting them to provide me with free service. There isn't enough of an incentive to allow them to agree to my free internet for Bill proposal. Same thing for most networks with unequal traffic patterns. Netflix is so big that any network they buy transit from, will cause their data flows to become unbalanced.
Well.. maybe. Or Maybe not. But Definitely not sort of.
Perhaps a rule where cable or satellite TV providers are prohibited from operating centralized peering points. If Verizon had to buy their bandwidth from upstream providers, they wouldn't be able to choke L3. And L3 would have to bid against Verizon's upstream providers to get Netflix's business.
Essentially, less economic centralization in the network infrastructure would provide for more opportunities for competitive bidding all along the chain. Everyone would end up with more customer-focused incentives.
"I want it and my government friends have guns..." Is this the best we can do?
The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies, federal grants and subsidies, and the reason they even exist at all is because of a government approved corporate charter. Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?
If you want a vision of the future, imagine a youtube comments section scrolling - forever.
the fcc will soon rule in favor of big multi-service providers, such as verizon and comcast, and allow the decimation of the hope of network neutrality so that they can stifle online-based competition in favor of their own video services.
Thou shall not pass
I am a bit confused as to how this all works, perhaps someone could clear things up for me.
Why should Level3 pay Verizon for access to their networks, and not Verizon pay Level3 for access?
In this case, Level3 is the supplier of the goods (Netflix data), Verizon and its customers are the consumers of the goods.
The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies...
I know. So a different answer might be to break up the monopolies and tell local governments that they can't make long term monopoly deals any more.
Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?
It's not good in either case. We should head in the other direction.
Verizon can afford more government friends than you can. Do you honestly foresee a time when they won't? If not, maybe you shouldn't want things to be decided based on who has more government friends?
[why should Verison care]
Customers.
They seem to have very limited interest in what their customers want for Netflix streaming quality. What is their incentive to care?
Are you fucking with me?
Are you asking why a company should care what it's customer's want?
Ok ok ok.... just to play along: Because the customers pay Verizon for them to connect Netflix to their home. They should upgrade their network to make their customers happy. Because that's who pays them. The customers. Who want to watch Netflix. Over Verizon's lines.
Are you dense? Is this some sort of painfully slow and round-about devil advocate? Are you really that sort of disconnected-from-reality libertarian?
Netflix paid for more direct access which means directly between Netflix and Verizon rather than using an intermediary like L3. Upgrading capacity between L3 and Verizon would not satisfy such an agreement.
"Lack of speed can be overcome. In the worst case by patience." --Znork
Your UPS analogy is slightly off... ... You buy a house near a bridge
The better analogy is
The bridge has 2 lanes
When you see the bridge at 4am there is nobody
But at 7am it's packed with delays towards your office and the other lane empty (empty lane has a toll booth)
You bought the house based on thinking it was going to be a simple ride to the office and free 1 way
What you discovered is huge delays.
Now you're asked to build another bridge lane, but the town won't help foot the bill unless it get's to build a toll booth and raise
the rates. Taxpayers don't want to foot the bill.
if you see me, smile and say hello.
Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.
I really miss working with them.
Serious? Seriousness is well above my pay grade.
The reality is: they don't seem to care. If you want them to care, shouldn't you try to understand why they don't?
With Comcast, you get some sort of boost... But they are rather correct on the service contract. You get internet access.
Here is the catch
The other side (in this case net flicks) has flooded the lines and won't pay to carry the transmission.
The consumer still gets the internet access and the transmission, but not at the fault of the Comcast.
if you see me, smile and say hello.
Laws are one of the tools we use to align the interests of groups that are potentially in conflict. Why aren't they a valid option here?
Yes, he is, he's trying to get people to say "it's the government's fault" but then will start to whine and cry if they say "it's the government's fault for not forcing Verizon to do this". It's not clear to me why the libertarian loons think that if the government just vanished completely tomorrow that Verizon would just suddenly up and say "gee, I guess we better get started on not sucking" but maybe they're just waiting for all the scapegoats to die before accepting the blame for their own actions.
+1. Politicians at the state level have been paid off by cable companies and ISPs to squash competition from local municipalities. The cities who got in before the legislation are loving their services. It is absolutely insane what money can do in the political process.
That's not an answer though. It's a description of the problem.
Mark Taylor of L3: “I appreciate that traffic ratios were used as a proxy for cost equality between backbone network peers historically [ellipsis] Enforcing balance in these circumstances is, in our view, a way of arbitrarily raising a toll.”
In other words, you want to change the way interconnect business arrangements have been done in the past because it disadvantages you because you are the primary source network for probably the biggest sender of data in the history of the world. Not to mention that Netflix is paying you a lot for all that access bandwidth to your network.
In this case, Verizon is quite explicit in that post that they either want you to pay them for the traffic imbalance or have Netflix buy access directly to their network.
This seems reasonable to me.
Verizon can afford more lawmakers than you.
We, the customers, are paying exactly for this kind of one-sided agreement though. Pretty much every residential internet connection is heavily biased towards download speeds vs. upload speeds. That means that the ISP (Verizon) knows that traffic is very one-sided, and the cost of making sure the 'pipes flow smoohtly' is just part of how they should be running their business.
Unfortunately the problem is that PR isn't done for the people who can see through the BS, will investigate the BS or will challenge the BS. it's for every day consumers who are not experts and are not interested in the details. They will remain completely ignorant or they'll only hear fragments of the whole drama. As long as enough of them hear the bits that Verizon has plenty of bandwidth available, so it must be Netflix, they're fine. And as long as all other mega-ISP's are playing the same game, they're completely safe from any sizable backlash. We're just malcontents that can be safely ignored.
Perhaps it is one or the other (or both) parties' intentions, but I'm now totally confused as to how the business arrangements are set up relative to the network architecture... I pay my ISP (happens to be Verizon but that's irrelevant for the purpose of this thought experiment) a certain amount per month to deliver content at a certain bitrate. Now I realize that's not a "guaranteed" rate, especially for peak times, but that's another issue (IMO, Verizon should not be overselling their bandwidth to such an extent that they can't deliver a certain minimum threshold of performance during peak usage, or minimum likelihood of achieving published bandwidth during peak usage). Do the content providers (and by proxy or extension their network hardware and infrastructure subs) - especially the high-bandwidth content providers like Netflix - *not* pay for a certain network capacity? If, for example, as a content provider, I expect to need to deliver 50 Gbps based on the size of my customer base and peak usage rates, then I would build out my network infrastructure to supply that, and contract with the service providers to provide appropriately sized interconnects and deliver that to subscribers. That sounds like a pretty simple arrangement to me - simple enough that it should be relatively easy to identify who's not holding up their end of the bargain. But it must not be the case, becase that doesn't seem to be happening. Verizon is talking about balanced vs. imbalanced arrangements, and Level 3 is talking about proportionate mileage costs etc. Perhaps one of you telecom experts - hopefully someone who is independent of either type of provider - can illuminate the situation for "lay" people.
How about so that Verizon customers can access Internet resources at something near the speed advertised and paid for.
I think Verizon customers have a case to sue Verizon for non-performance.
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
And a government's responsibility is to take action against a company which is committing wholesale fraud against its customers by selling them Internet Service which promises bandwidth speeds which they are then purposefully not providing in order to shake down their customers and companies trying to provide services to those customers more money.
A government's responsibility is to ensure that companies that are given government licenses and franchise agreements which restrict competition in certain geographic areas are providing the service that the people of that area want and need at a fair price.
A government's responsibility is to ensure that companies which get too big, hold too much market share and are too horizontally or vertically integrated are broken up so that there can be real competition and a real free market.
"it's Verizon's customers who are causing all this bandwidth usage"
Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.
And then Verizon passive aggressively acts like the problem is not of their own making.
Throttling netflix at the peer reduces load on those trunks without affecting other services.
I understood that, or at least guessed that this was the reason. However when you are restricting the connection like that yourself you cannot go and claim that it is the fault of the other party. Either you need to admit that the limiting factor is your own internal network or you need to spend some of the large amount of cash that is flowing in from your subscribers to upgrade that network to handle it instead of using it to see whether you can break the record for executive bonuses.
Not all the cities. Some municipal fiber projects failed.
But some have succeeded.
At what point do I (a FIOS customer) have recourse to sue FIOS? They sold me unlimited access to the internet: all of the internet, not just the parts which pay them to access me.
If you are paying for 75/25 or 50/25 and they are throttling it at the borders of their network, then you aren't getting the bandwidth you are paying for... downgrade your service. That $10, $20 or more per month they aren't getting from you because of their throttling practices should get their attention.
Congestion doesn't even matter much. If it matters, you're probably doing something wrong. Download your video at 3am when it's not congested. Or start your download at 7pm and if, for whatever reason, your 60 minute video doesn't finish until midnight: fine, whatever. Who cares?
Problem solved. Playback should be from in-house disk.
Netflix's customers knew they were getting into something technologically stupid, the moment they signed up. So I don't see what they're complaining about. If you want Verizon to make your dumb tech work better, then yeah, pay 'em more. I wouldn't do that, but then I'm not the princess with the high-bandwidth-and-must-be-realtime fetish. Do you think a person who takes baths in champagne complains about their champagne expenses? No, because if you're the kind of person who would object to spending $5000 on your daily bath, then you're the kind of person who uses water.
Thanks for creating the scaling demand, though. Your luddite problems are making my relatively high-tech solution (download file, click it) work better. It reminds me of all the Windows users in the 1990s, making RAM and disk drives cheaper for my non-Windows computers. You people got me my first GigaHertz CPU! There's no way that would have happened without you, and for that, I'm grateful. Why you used 90% of your CPU to run Norton Anti-Virus, I never really understood, but to each, his own.
The idiocy we are seeing now is the result of poor regulation by the FCC, the states, and localities. We have been predicting fragmentation of the Internet because of the failures of our government to properly regulate and now we are seeing it. We are already seeing some Internet content available to customers of one company but not another.
Verizon has four truck loading docs, Level3 has eight, Level3 sends and receives as much as Verizon's 4 loading docs can handle.
Verizon then indicates that there's no congestion on their side, because every dock is being used at maximum capacity, so the problem must be Level3 is sending as many trucks as possible. To prove this claim, they look at the workers in the warehouse and notice that about half of them are idle.
Meanwhile, Level3 has been very upset because if only Verizon would open up the additional 4 loading bays that are not used (which will incur a small one time cost, to cut chains and move the stored items out of the way), then Level3 could send data using all eight of it's bays.
Youtube is shipping packages, they have received so many customer complaints that they are actually getting into the nasty details of how the shipping system works, even to the detail where they know enough about the problem that they are willing to relay out the warehouse, cut the locks off the extra shipping bays, clean and repair the door lifts, and generally get the warehouse working at a higher capacity.
Verizon's been badmouthing Level3, so Level3 put all the data on the table to prove it wasn't them. Verizon's been griping about the cost of supporting YouTube, and now YouTube's been offering to retrofit the warehouse for free.
What Verizon really wants, and what they've been asking for is to paint the additional 4 bays with YouTube logos, and only have YouTube trucks from Level3 hit those bays at an additional cost to YouTube. So basically when getting something shipped from YouTube, as a consumer, you pay Verizon for the shipping, but for you to actually get it delivered before Christmas, YouTube pays Verizon too (you know, so the package doesn't get "lost", it's a special handling fee, you see. Nudge, nudge, wink, wink!).
There was a car analogy in there somewhere.
and in trucks ar
Netflix has never "flooded" any lines, Comcast refused to upgrade their circuits with L3 meaning they did not provided the advertised speeds to the Internet through there own negligence. L3 was very clear they were happy to increase port count/speed Comcast refused to do so. L3 is a tier 1 Comcast is not yet L3 was providing them with free peering in the interest of making the internet work.
No sir I dont like it.
The EULA for an ISP never guarantees service on residential links. All speeds are listed as "up to".
I read the internet for the articles.
L3 should just de-peer Verizon on a Friday evening and dump that Netflix traffic to the provider where Verizon buys transit. Then for a whole weekend Verizon customers will notice the shitty Internet speed for *all* services, and Verizon will pop their usual 95th-percentile bandwidth at their transit and have a huge bill for that month.
Then see whether they are willing to operate that peering responsibly. I mean, they can have it their way, if they want.
Has anyone started a class action law suit on behalf of the paying Verizon customers that are getting screwed in order for Verizon to extort money out of Netflix?
Too well-bribed to regulate....
Not on your life.
You're trying to replace a direct cont(r)act with a single company (the UPS) with vague, at best indirect "contract" with a gouverment entity.
The road is not yours, nor is there a "breach of contract" when you can't use it ("best effort" and all that). In the case of UPS not delivering the package it is a breach of contract, and maybe even theft too (wire fraud ?).
.
.
But lets continue on your road (literally and figurativily): Just imagine that you next to that bridge and others in the town just a few miles down the road agree to pay more tax to get a four-lane road.
Than the road is widened, but not fully upto the interstate road the origional road was/is connected to: somewhere the four-lane road simply ends and becomes the old two-lane road again.
Do you think that the gouverment that took your taxes in the promiss it would give you a four-lane road came true to what it offered and was payed to do ?
Yes, they did put a nice stretch of 4-lane road down. But can you actually use it as intended ?
"If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?"
Yeah, I'm sure they're working on that, too.
So, you're saying the checks Netflix writes to Cogent and Level3 every month are donations? No, Netflix pays for the bandwidth they consume,
APK quotes people (including myself) without context and should not be trusted. Just thought you should know.
But how are Verizon going to keep investors happy if they start using their excess capital for reinvestments instead of paying out dividends?
Won't you think of the investors?!?!
From their time table, I am *guessing* they're traveling on foot. It would take but a few months if they're traveling on horse back.
ELOI, ELOI, LAMA SABACHTHANI!?
People usually want to pay the least for the service they find acceptable. So yes, if they cut my bill in half and said netflix would never work on my connection than I would jump at it. I really don't want to be subsidizing you.
Charging Netflix money just to connect is anti-competitive, because both companies sell the same service, only one is the gatekeeper to the customer. If anyone gets charged more for my bandwidth it should be me, and only me. I then get to choose who I use to connect to the Internet. If Verizon raises my rate I have the right to go shopping. If they rise the rate for Netflix that is extortion, and Netflix doesn't get to go shopping for a better source of customers. it doesn't work that way, and there are laws against that sort of thing. Why Congress doesn't get it is a mystery to me.
I wish it was that simple, but I'm on board with the general idea. I wouldn't publicly cut the cables. That's too extreme. I would, however, like to see Level 3 turn the tables and publicly (as noisily as possible) accuse Verizon of using up all their bandwidth and that if Verizon doesn't help them pay for the costs of upgrading, their customers just won't be able to watch Netflix anymore. "I don't see why Verizon is taking issue with this. It's standard practice. People download a lot of video from us. Remember folks, we're not downloading from them! Verizon's users are downloading from us! They're the ones using all the traffic!" (I know that's a completely inaccurate and misleading explanation of the situation - an outright lie, if you will - but that's the point.)
If Verizon customer's thought "Who's Level 3? Netflix is paying them so I can watch Netflix, and now they want to charge Verizon money so I can watch Netflix? So basically I'm getting charged twice to watch Netflix? This will not stand!"
Then when everybody has turned their heads in their direction, Level 3 would say, "Just kidding! Here's what's really going on..." and tell people a simplified version of what they said in TFA. Then maybe people would start to care about net neutrality. (If you were to replace Level 3 with Netflix themselves, it would oh, so much better!)
Of course, it's a good thing I'm not in charge of either company because I'd have just lost an unprecedented amount of business and ruined the company's reputation in the process, and unfortunately, the average person isn't going to respond a calm, well-reasoned, fact-based argument like TFA. Most people just aren't going to care as long as there's something playing on TV. When you get in the way of that, people will definitely notice, but anybody even remotely involved in such an affair will be ruined for life.
because socialism that helps businesses is AOK, but socialism that helps the average shmuck is an affront to everything this country was founded on.
something something welfare queens, something something defund obamacare.
that's already built into fiber points directly. Verizon can't so no to this since they are a direct arm of the NSA/CIA
Using your logic, anyone that has Comcast or Verizon for their internet connection is screwed unless the place they are getting bandwidth from is also a Verizon or Comcast customer. Wow, wouldn't that be an awesome setup for Verizon and Comcast.
This is what Comcast and Verizon want.
Bad boys rape our young girls but Violet gives willingly.
While I'm not doubting the actually connection point could just use a couple network cards to get all that additional traffic onto Verizon's network, what does that do to the rest of Verizon's network?
If allowing all of Level 3's traffic pushes Verizon's bandwidth usage at any point in their system to unacceptable levels, then that part needs to be upgraded as well... which may cost a lot more than a couple network cards.
So, how much traffic is Level 3 pushing? What are the costs to upgrade all of Verizon's weak-points to accommodate the additional traffic? How much do Verizon's customers pay towards that? How much do the peering contracts pay towards that?
Hating Verizon is not enough to answer those questions.
The verizon customers need to help their neighboring verizon customers and ditch them completely.
Maybe they can get verizon down to the non-netflix and non-youtube customers. Or with any luck zero customers.
That and the insane cost to enter the ISP market, sure. Are you the dumbass who thinks that natural monopolies don't exist, or the dumbass who thinks that non-coercive governments can exist, or do you combine them to be a kind of super-dumbass?
If it were all about "even" traffic flows, then netflix could have their clients send garbage data back to balance out the flows. This would result in *more* traffic on the network overall, but hey it'd at least be balanced!
No...the traffic is there because it was *requested by verizon's subscribers*. There is no logical reason why cogent/level3 should pay extra for traffic requested by verizon. I know this is how it was done in the past, but that was under the assumption that the types of flow is more or less similar. In the case of verizon, it's mostly consuming data rather than sending it, so it shouldn't be treated as a regular peer.
Booooo Yaaaahhhhhh!!!
Netflix already pays verizon for DIRECT connection to its customers Netflix did not pay verizon for its interconnect via Level3.
Verizon is basicaly saying that Netflix routes alot of traffic through Level3 (probably because they're cheaper) instead of via their direct verizon line.
Level 3 is saying that the pipe between Verizon backbone and L3 backbone is where the bandwidth gets clogged.
take out verizon and insert (I'm fairly positive) every single US ISP, and I'm pretty sure you'll hear teh exact same story. L3 will probably blame all the ISPs for their lack of bandwidth interconnecting them. And all the other US isps will say, they're not the issue because their internal networks aren't close to full saturation.
So the real question at hand is, who really should pay for upgrading the capacity between the tier 1 interconnects.
Sounds like there was a handshake agreement between the companies originally. This handshake probably didn't include a late comer like level3.
I would argue that if l3 approached these companies and said, we'll install the cards AND maintain these connections on L3's bill, everything will get resolved.
Ie. L3 is pretty much saying, sure we're tryign to compete with you big ISPs, and we're gaining customers to your detriment on the Inet tier1 space, but we can't seem to maintain our client base if you guys at the last mile/end point don't increase your bandwidth with us.
The 6 other major ISPs are saying to the content providers, why bother using L3, sign up with our own t1 bandwidth and we can provide to your customers directly a great service.
Individuals, and individual rights, are like single atoms. They only exist in the abstract sense. The real world is entirely dominated by groups and collective actions.
You're a confused anarchist. The problem with non-coercive government is that all government is coercive. Government is primarily a set of restrictions on the use of force, or alternately the monopoly on that use of force. Getting rid of a government, or disarming it, merely allows anyone with a larger arsenal to set up their own government -- anarchy is an unstable system. We all have a right to violence, because it cannot be taken from us except in extreme situations. Remember, the Code of Hammurabi was instituted, "...so that the strong might not harm the weak." Coercive government is a necessary evil, and it will remain necessary so long as men are capable of harming their fellows, for that is its justification and primary purpose.
Rights are not inherent, except in some abstract sense. In the real world, your rights are what the men with guns say they are. You may feel fortunate that the world has had a long, bloody time to work out semi-cooperative frameworks to restrain our darker impulses. Individual rights are an important conceptual counterbalance to the overwhelming powers of the collective, but they are no justification for anarchy, economic or otherwise. The "free market" is an ideal, even a good one, but in most cases removing government interference makes markets less free, more subject to collusion and fraud. In some cases, where the service is required to be universal, or when the barriers to entry would be insurmountable, it makes sense for the government to assume these functions directly. Govenment can also be thought of as the natural monopoly of natural monopolies, in that sense.
Slavery is a word that has a specific meaning; your definition is specious. You just fundamentally don't like being told what to do. To some degree this idealism is admirable. For the true individualist, I can recommend (from long personal experience) the Alaskan wilderness; you can get land for free still up there, provided you build upon it. Whatever romantic images your mind conjures upon thinking of Alaska are all true; I can't stand the weather, personally, but it's as close to a pure state of nature as you will ever find. If you'd like to enjoy the benefits of society, however, you have to play by the rules. "Slavery" isn't an option -- it's mandatory.
Those who advocate genocide deserve every protection afforded by law, and none afforded by common human decency.
So... you really can't think of a solution that works around those constraints? The world is both simpler and more complex than your narrow and politically motivated mindset can grasp.
Level3 schools Verizon.
That would make too much sense. You can also look at it as if Verizon is supplying the eyeballs (consumers) and Netflix is consuming them. Netflix (and therefore L3), can't make money if they can't get their content to the subscribers, so Verizon can hold them hostage. It also doesn't hurt that Verizon has its own competing product and if there are enough problems with Netflix, some of their subscribers might convert, making them even more money. Oh and, of course, you can't change providers, that would be something dangerous, like competition.
Then they can throttle proportionally to how you downgraded. You cut your service in half, and they cut your throughput allowance to a particular site in half on the throttling server.
Deregulate!! Deregulate!! Deregulate!!
Its a good thing to be known as Verizon management.
Brutal management trolls throttling their own kids and family members for money every day of their living life.
Allow more competitors in for every square mile of customers
and they will fsck off with all the good customers!!!!!!!!!!!!!!!!!
So in other words, an intentionally misbehaving caching proxy. Good luck getting subscribers to install the root certificate that lets the ISP forge such responses to HTTPS requests.
Well, NetFlix could also enter into agreements with ever backbone provider, thereby forcing Verizon to either do the same to everyone or start upgrading
This brings to the question of why Netflix has chosen to deal with Verizon instead of with Level3 directly in the first place ?
Even if Netflix didn't know of the existence of Level3 (which I find too ludicrous to be possible) that they had signed up with Verizon, they could have changed the situation right now by dealing directly with Level3, and why wait anymore ?
Muchas Gracias, Señor Edward Snowden !
Well only a handful of libertarians want the govt to disappear. But lets grant that. If they did who would enforce the last mile monopoly laws and right of ways? No one. Guess what would happen VZ would have to start paying people for those right of way accesses and guess what other providers could use those acccesses as well. As I have said forget Net Neutrality fix the last mile monopoly franchises and CRIMINAL laws prohibiting addt. providers will cover a lot of the country. Also no FCC who just took 3.65 GHz away from small WISPs and gave to big telcos. You know those WISPs that provide a lot of rural US internet.
Govt is the one creating these monopolies through regulation mostly because lobbying and regulatory capture. I am not suggesting end govt. but fix the barriers to last mile competition issue and this crap will go away.
http://blog.level3.com/global-...
"A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.
That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.
Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers."
We don't have all the information here. The Level 3 post only talks about how much the hardware costs.
Level 3 very explicitly avoided mentioning services charges which Level 3 may (or may not) have been trying to get from Verizon. Note that Level 3 didn't explicitly say they would provide the extra connection bandwidth free of charge, only the hardware. I think there is more going on here than we know...
Of course, it could also be that Verizon is trying to get money from Level 3 because Level 3 was sending so much more traffic than Verizon. I am a very experienced network software designer and it is more expensive to receive IP traffic than to transmit it -- especially non-TCP traffic, because there is no flow control -- you need a lot of extra hardware capacity to handle bursts (buffering and CPU). From a technical standpoint, it makes a lot more sense for Level 3 to pay Verizon to handle the extra traffic.
BUT, compared to the cost of the entire network, the real cost of the peering is probably pretty insignificant for both parties. The only conclusion here is that they are probably both spinning the message by leaving out the unflattering information.
An engineer who ran for Congress. http://herbrobinson.us
Got anything better?
Remove the laws and regulations holding back community fiber projects.
If you truly believe this is the problem, then you clearly have never tried to run a business in that market.
a handful of selfish greedy people are no match for millions of selfish, greedy people -u4ya
Well, NetFlix could also enter into agreements with ever backbone provider, thereby forcing Verizon to either do the same to everyone or start upgrading
This brings to the question of why Netflix has chosen to deal with Verizon instead of with Level3 directly in the first place ?
Even if Netflix didn't know of the existence of Level3 (which I find too ludicrous to be possible) that they had signed up with Verizon, they could have changed the situation right now by dealing directly with Level3, and why wait anymore ?
NetFlix has contracts with lots of folks - Level3 included. These are with respect to pushing content from NetFlix over the backbone into various networks.
NetFlix also promotes having a CDN end-point within an ISP's network to alleviate the need for as much peering; which is what I believe the NetFlix-Verizon deal was about, which Verizon may have (or may not have, we don't really know) charged NetFlix for installing in the datacenters/hubs/central-offices.
However, the fact that NetFlix has done that, which should IMPROVE speed on Verizon's network, and there are still major issues shows that there is something else wrong with Verizon's network. Of course, they might rely on the back bone having sufficient capacity to pull down the information over the CDN too; or it might be that NetFlix installs a direct pipe for the CDNs, we don't know the details. Most likely NetFlix has a contracted pipe with a Level3 interconnect to these, and that is why we're hearing all about it between Level3 and Verizon as Verizon doesn't want to increase their interconnects with Level3 over which those CDN systems are suppose to operate.
But that's just a bit of (educated) guess work as I don't know the details of the Netflix-Verizon arrangement or the network layouts or the NetFlix CDN provisions.
Truth is like the sun. You can shut it out for a time, but it ain't goin' away. - Elvis Presley (source: imdb.com)
If you truly believe this is the problem, then you clearly have never tried to run a business in that market.
Incorrect assumption. Been there, done that, got the business cards of half the executive branch.
My God, it's Full of Source!
OUTSIDE_IP=$(dig +short my.ip @outsideip.net)