Slashdot Mirror
Verizon's Accidental Mea Culpa
Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).
If people don't think bandwidth is a scarce commodity, how will we get them to pay through the nose for it?!?
https://www.eff.org/https-everywhere
Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.
Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this. You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this" All too predictable, and the same techs are probably still being yelled at.
Waiting for an amusing sig.
Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.
Netflix has *yet* to pull up a dump-truck full of money to Verizon HQ.
SJW's don't eliminate discrimination. They just expropriate it for themselves.
We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.
L3's blog still has a summary blurb, but the link to the actual post gets a 404 - did they take it down or did they just link it wrong? Anybody have a cached copy?
Why should Verizon do the upgrade? Why would they want to? To make Level 3 happy? To make Netflix happy? What is their incentive?
The only answers I've seen for this on Slashdot are:
- the government should threaten Verizon and force them to operate the network contrary to Verizon's best interests,
- the government should seize Verizon's network
- no answer, just crying about how you're entitled to better Netflix video quality
Got anything better?
Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.
Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."
"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."
I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.
Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.
Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.
I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)
I came, I conquered, I coredumped
What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
They obviously didn't understand the presentation so they assumed no one else would either.
First, this is a violation of the Telecommunications Act of 1996.
Secondly, we are seeing the rip off of the US taxpayer. And I think Verizon should pay ALL of the taxes they should have paid:
Over the decade from 1994-2004 the major telephone companies profited from higher phone rates paid by all of us, accelerated depreciation on their networks, and direct tax credits an average of $2,000 per subscriber for which the companies delivered precisely nothing in terms of service to customers. That's $200 billion with nothing to be shown for it.
And thanks to the Hobby Lobby ruling (pierced the corporate veil), it looks like that the CEO should go to jail for tax evasion.
Verizon’s Accidental Mea Culpa Mark Taylor / 17 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s own paying broadband consumers.
His explanation for Netflix’s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it’s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here’s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon’s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that’s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3’s network interconnects with Verizon’s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let’s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion
My company has Verizon and Cogent (and others). Our cogent offices have an absolute terrible time maintaining any type of stable throughput to our midwest data center running on Verizon. Often times we get speeds of several KB/sec for hours on end between them over our 100/200mbit connections. I've called Cogent and Verizon and complained for years. The bottleneck is the handoff from Cogent to Verizon. Verizon will not budge and claims no responsibility.
Interesting tidbit is the average speed for the last year shows Verizon --> Cogent is about 2x-10x the speed compared to Cogent --> Verizon on the same circuits. Verizon must not like to take on incoming Cogent data.
So maybe someone can explain this to me because I don't entirely get it.
Right now Level 3 doesn't pay Verizon any additional money for the data being sent its way (yes, requested by Verizon customers, but transport is usually paid by the shipper -- when I order a physical product I pay for shipping to the vendor, who pays the transporter).
The reason Level 3 doesn't pay any more is because they are using settlement-free links established to provide basic bi-directional communication between the two networks. Because of the way they are using them, these links (which are set up to provide balanced access) are saturated in one direction while only 30-60% utilized in the other direction.
The point made by both companies is that fixing the congestion is a simple matter of hooking up a couple ports (which would increase the utilization of Verizon's network).
Level 3 wants Verizon to agree to expand the settlement free ports to allow for the imbalance of traffic. Verizon says "our settlement free ports are sufficient for normal traffic, and if you want to avoid congestion for the additional traffic you are charging Netflix to carry then you're going to need to purchase additional ports and pay for that traffic."
Neither wants to budge and so they fight a PR war about it. Level 3 says "It's just a couple ports and a little cable" while disregarding the downstream impact on Verizon's network. Verizon says "Level 3 is taking undue advantage of our mutually beneficial arrangement and wants us to help them do it for free."
Is this accurate?
Is this a resurrection of the Slashdot effect? Or did someone higher up at Level(3) notice this and pull the blog post?
Verizon seems to be saying that L3 needs to pay more because the bandwidth flowing into Verizon's network far outweighs the bandwidth flowing into L3's. While this is fundamentally idiotic for a last-mile ISP for any reason than money grabbing due to the asymmetric up/down bandwidth offered to last-mile customers, let's play their game. It sounds like all we need to do is send an equivalent amount of traffic back to Netflix, and everything will be right in the world. Even better, if we send more traffic to Netflix than we receive, L3 ought to be able to get money from Verizon because that traffic is out of balance in the other direction now.
Just tried going to the link, it's been 404'ed.
Here is a copy of the text, just in case:
Verizon's Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.
His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.
lvltvzw
Verizon has
\m/
I get tired of being the only person on slashdot that understands this...
The problem isn't the interconnect. The problem is between the local remote that feeds your house and the Central Office. When that much data comes from Netflix all at the same time, the remotes trunks can't handle it. Upgrading THOSE trunks costs a fortune. Throttling netflix at the peer reduces load on those trunks without affecting other services. That's what's going on and why Verizon (and others) are throttling Netflix. They have no other way of targeting netflix traffic directly without sending the FCC into a tizzy.
Netflix-------> Level3-------> Verizon core network-------> Verizon local CO----(the problem is here)---> remote-------> your house
You can argue that Verizon should fix that trunking themselves, we'd have a different argument then... but what this "Story" is about isn't even what's wrong. You can't just look at one section of the cheapest part of Verizons network and claim how easy it would be for them to fix. They've got a huge multi-billion dollar network to maintain and the front door to that network is the least of their concerns.
Actually it's more like a company paying for x telephone circuits to their PBX, only Verizon made most of them 'party lines', so even though you can connect you're now having to talk and listen over both your fellow employees as well as their customer's replies in order to get anything done. However if you do a 'long distance' callout via calling card then call from there to your customer you get a perfectly free and clear line instead of a 'party line' :)
I hope this makes for a relatively accurate, concise, and entertaining telephone analogy.
Dear Verizon Customers,
We were recently told we needed ~$100,000 worth of equipment. In order to pay for this we're going to need to add an interconnect service fee of $5/month to your bill. Sorry. Try a competitor? Hah, suck it.
Thanks,
Verizon CEO
PS We have also sued the company who told us this for saying it publicly, and we will most likely charge them more money, too.
The link to the LEVEL3 Blog post has been blocked...but modifying the link a bit pulls it back up again. Blog post reproduced below (with relevant image here)
VerizonÃ(TM)s Accidental Mea Culpa
Mark Taylor / 23 hours ago
David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns NetflixÃ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by VerizonÃ(TM)s own paying broadband consumers.
His explanation for NetflixÃ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.
The diagram then shows this one little bar, suggestively color-coded in red so you know itÃ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.
But hereÃ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as VerizonÃ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatÃ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network Ã" IN BOTH DIRECTIONS.
So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?
Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3Ã(TM)s network interconnects with VerizonÃ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.
So letÃ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of th
It seems more like:
L3 provides bulk backbone data haulage to its customers, which they pay for. It provides the resources to deliver them to their destination. It will hand over packets to whichever ISP provides service to the packets' destination. All this, it does.
Verizon is supposed to provide a certain bandwidth to its customers, which they pay for. These are the end users. It does provide a link meeting bandwidth requirements between the customer and Verizon's internal network, which includes some services like Verizon's pay TV. However by failing to upgrade an at-capacity interconnect for which it is responsible it fails to provide the full bandwidth to their subscribers when they visit certain sites beyond Verizon's internal network, such as Netflix.
L3 appear to be doing all they are contracted to - delivering data to its destination ISP - and at a high bandwidth.
Verizon do appear to be falling short of their obligations as they have contracted to provide consumers a package based on provision of a certain bandwidth, but are failing to meet the terms of that contract at least for data from some parts of the internet by failing to upgrade parts of their infrastructure. The fact that this would increase Verizon's network utilisation is neither here nor there - if that is the cost of providing their clients with their contracted bandwidth then that is what they are bound to do.
the fcc will soon rule in favor of big multi-service providers, such as verizon and comcast, and allow the decimation of the hope of network neutrality so that they can stifle online-based competition in favor of their own video services.
Thou shall not pass
I am a bit confused as to how this all works, perhaps someone could clear things up for me.
Why should Level3 pay Verizon for access to their networks, and not Verizon pay Level3 for access?
In this case, Level3 is the supplier of the goods (Netflix data), Verizon and its customers are the consumers of the goods.
Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.
I really miss working with them.
Serious? Seriousness is well above my pay grade.
Mark Taylor of L3: “I appreciate that traffic ratios were used as a proxy for cost equality between backbone network peers historically [ellipsis] Enforcing balance in these circumstances is, in our view, a way of arbitrarily raising a toll.”
In other words, you want to change the way interconnect business arrangements have been done in the past because it disadvantages you because you are the primary source network for probably the biggest sender of data in the history of the world. Not to mention that Netflix is paying you a lot for all that access bandwidth to your network.
In this case, Verizon is quite explicit in that post that they either want you to pay them for the traffic imbalance or have Netflix buy access directly to their network.
This seems reasonable to me.
Unfortunately the problem is that PR isn't done for the people who can see through the BS, will investigate the BS or will challenge the BS. it's for every day consumers who are not experts and are not interested in the details. They will remain completely ignorant or they'll only hear fragments of the whole drama. As long as enough of them hear the bits that Verizon has plenty of bandwidth available, so it must be Netflix, they're fine. And as long as all other mega-ISP's are playing the same game, they're completely safe from any sizable backlash. We're just malcontents that can be safely ignored.
Perhaps it is one or the other (or both) parties' intentions, but I'm now totally confused as to how the business arrangements are set up relative to the network architecture... I pay my ISP (happens to be Verizon but that's irrelevant for the purpose of this thought experiment) a certain amount per month to deliver content at a certain bitrate. Now I realize that's not a "guaranteed" rate, especially for peak times, but that's another issue (IMO, Verizon should not be overselling their bandwidth to such an extent that they can't deliver a certain minimum threshold of performance during peak usage, or minimum likelihood of achieving published bandwidth during peak usage). Do the content providers (and by proxy or extension their network hardware and infrastructure subs) - especially the high-bandwidth content providers like Netflix - *not* pay for a certain network capacity? If, for example, as a content provider, I expect to need to deliver 50 Gbps based on the size of my customer base and peak usage rates, then I would build out my network infrastructure to supply that, and contract with the service providers to provide appropriately sized interconnects and deliver that to subscribers. That sounds like a pretty simple arrangement to me - simple enough that it should be relatively easy to identify who's not holding up their end of the bargain. But it must not be the case, becase that doesn't seem to be happening. Verizon is talking about balanced vs. imbalanced arrangements, and Level 3 is talking about proportionate mileage costs etc. Perhaps one of you telecom experts - hopefully someone who is independent of either type of provider - can illuminate the situation for "lay" people.
Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)
And a government's responsibility is to take action against a company which is committing wholesale fraud against its customers by selling them Internet Service which promises bandwidth speeds which they are then purposefully not providing in order to shake down their customers and companies trying to provide services to those customers more money.
A government's responsibility is to ensure that companies that are given government licenses and franchise agreements which restrict competition in certain geographic areas are providing the service that the people of that area want and need at a fair price.
A government's responsibility is to ensure that companies which get too big, hold too much market share and are too horizontally or vertically integrated are broken up so that there can be real competition and a real free market.
"it's Verizon's customers who are causing all this bandwidth usage"
Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.
And then Verizon passive aggressively acts like the problem is not of their own making.
Throttling netflix at the peer reduces load on those trunks without affecting other services.
I understood that, or at least guessed that this was the reason. However when you are restricting the connection like that yourself you cannot go and claim that it is the fault of the other party. Either you need to admit that the limiting factor is your own internal network or you need to spend some of the large amount of cash that is flowing in from your subscribers to upgrade that network to handle it instead of using it to see whether you can break the record for executive bonuses.
At what point do I (a FIOS customer) have recourse to sue FIOS? They sold me unlimited access to the internet: all of the internet, not just the parts which pay them to access me.
If you are paying for 75/25 or 50/25 and they are throttling it at the borders of their network, then you aren't getting the bandwidth you are paying for... downgrade your service. That $10, $20 or more per month they aren't getting from you because of their throttling practices should get their attention.
Congestion doesn't even matter much. If it matters, you're probably doing something wrong. Download your video at 3am when it's not congested. Or start your download at 7pm and if, for whatever reason, your 60 minute video doesn't finish until midnight: fine, whatever. Who cares?
Problem solved. Playback should be from in-house disk.
Netflix's customers knew they were getting into something technologically stupid, the moment they signed up. So I don't see what they're complaining about. If you want Verizon to make your dumb tech work better, then yeah, pay 'em more. I wouldn't do that, but then I'm not the princess with the high-bandwidth-and-must-be-realtime fetish. Do you think a person who takes baths in champagne complains about their champagne expenses? No, because if you're the kind of person who would object to spending $5000 on your daily bath, then you're the kind of person who uses water.
Thanks for creating the scaling demand, though. Your luddite problems are making my relatively high-tech solution (download file, click it) work better. It reminds me of all the Windows users in the 1990s, making RAM and disk drives cheaper for my non-Windows computers. You people got me my first GigaHertz CPU! There's no way that would have happened without you, and for that, I'm grateful. Why you used 90% of your CPU to run Norton Anti-Virus, I never really understood, but to each, his own.
The idiocy we are seeing now is the result of poor regulation by the FCC, the states, and localities. We have been predicting fragmentation of the Internet because of the failures of our government to properly regulate and now we are seeing it. We are already seeing some Internet content available to customers of one company but not another.
Verizon has four truck loading docs, Level3 has eight, Level3 sends and receives as much as Verizon's 4 loading docs can handle.
Verizon then indicates that there's no congestion on their side, because every dock is being used at maximum capacity, so the problem must be Level3 is sending as many trucks as possible. To prove this claim, they look at the workers in the warehouse and notice that about half of them are idle.
Meanwhile, Level3 has been very upset because if only Verizon would open up the additional 4 loading bays that are not used (which will incur a small one time cost, to cut chains and move the stored items out of the way), then Level3 could send data using all eight of it's bays.
Youtube is shipping packages, they have received so many customer complaints that they are actually getting into the nasty details of how the shipping system works, even to the detail where they know enough about the problem that they are willing to relay out the warehouse, cut the locks off the extra shipping bays, clean and repair the door lifts, and generally get the warehouse working at a higher capacity.
Verizon's been badmouthing Level3, so Level3 put all the data on the table to prove it wasn't them. Verizon's been griping about the cost of supporting YouTube, and now YouTube's been offering to retrofit the warehouse for free.
What Verizon really wants, and what they've been asking for is to paint the additional 4 bays with YouTube logos, and only have YouTube trucks from Level3 hit those bays at an additional cost to YouTube. So basically when getting something shipped from YouTube, as a consumer, you pay Verizon for the shipping, but for you to actually get it delivered before Christmas, YouTube pays Verizon too (you know, so the package doesn't get "lost", it's a special handling fee, you see. Nudge, nudge, wink, wink!).
There was a car analogy in there somewhere.
and in trucks ar
L3 should just de-peer Verizon on a Friday evening and dump that Netflix traffic to the provider where Verizon buys transit. Then for a whole weekend Verizon customers will notice the shitty Internet speed for *all* services, and Verizon will pop their usual 95th-percentile bandwidth at their transit and have a huge bill for that month.
Then see whether they are willing to operate that peering responsibly. I mean, they can have it their way, if they want.
Has anyone started a class action law suit on behalf of the paying Verizon customers that are getting screwed in order for Verizon to extort money out of Netflix?
Too well-bribed to regulate....
From their time table, I am *guessing* they're traveling on foot. It would take but a few months if they're traveling on horse back.
ELOI, ELOI, LAMA SABACHTHANI!?
I wish it was that simple, but I'm on board with the general idea. I wouldn't publicly cut the cables. That's too extreme. I would, however, like to see Level 3 turn the tables and publicly (as noisily as possible) accuse Verizon of using up all their bandwidth and that if Verizon doesn't help them pay for the costs of upgrading, their customers just won't be able to watch Netflix anymore. "I don't see why Verizon is taking issue with this. It's standard practice. People download a lot of video from us. Remember folks, we're not downloading from them! Verizon's users are downloading from us! They're the ones using all the traffic!" (I know that's a completely inaccurate and misleading explanation of the situation - an outright lie, if you will - but that's the point.)
If Verizon customer's thought "Who's Level 3? Netflix is paying them so I can watch Netflix, and now they want to charge Verizon money so I can watch Netflix? So basically I'm getting charged twice to watch Netflix? This will not stand!"
Then when everybody has turned their heads in their direction, Level 3 would say, "Just kidding! Here's what's really going on..." and tell people a simplified version of what they said in TFA. Then maybe people would start to care about net neutrality. (If you were to replace Level 3 with Netflix themselves, it would oh, so much better!)
Of course, it's a good thing I'm not in charge of either company because I'd have just lost an unprecedented amount of business and ruined the company's reputation in the process, and unfortunately, the average person isn't going to respond a calm, well-reasoned, fact-based argument like TFA. Most people just aren't going to care as long as there's something playing on TV. When you get in the way of that, people will definitely notice, but anybody even remotely involved in such an affair will be ruined for life.
that's already built into fiber points directly. Verizon can't so no to this since they are a direct arm of the NSA/CIA
While I'm not doubting the actually connection point could just use a couple network cards to get all that additional traffic onto Verizon's network, what does that do to the rest of Verizon's network?
If allowing all of Level 3's traffic pushes Verizon's bandwidth usage at any point in their system to unacceptable levels, then that part needs to be upgraded as well... which may cost a lot more than a couple network cards.
So, how much traffic is Level 3 pushing? What are the costs to upgrade all of Verizon's weak-points to accommodate the additional traffic? How much do Verizon's customers pay towards that? How much do the peering contracts pay towards that?
Hating Verizon is not enough to answer those questions.
The verizon customers need to help their neighboring verizon customers and ditch them completely.
Maybe they can get verizon down to the non-netflix and non-youtube customers. Or with any luck zero customers.
If it were all about "even" traffic flows, then netflix could have their clients send garbage data back to balance out the flows. This would result in *more* traffic on the network overall, but hey it'd at least be balanced!
No...the traffic is there because it was *requested by verizon's subscribers*. There is no logical reason why cogent/level3 should pay extra for traffic requested by verizon. I know this is how it was done in the past, but that was under the assumption that the types of flow is more or less similar. In the case of verizon, it's mostly consuming data rather than sending it, so it shouldn't be treated as a regular peer.
Booooo Yaaaahhhhhh!!!
Netflix already pays verizon for DIRECT connection to its customers Netflix did not pay verizon for its interconnect via Level3.
Verizon is basicaly saying that Netflix routes alot of traffic through Level3 (probably because they're cheaper) instead of via their direct verizon line.
Level 3 is saying that the pipe between Verizon backbone and L3 backbone is where the bandwidth gets clogged.
take out verizon and insert (I'm fairly positive) every single US ISP, and I'm pretty sure you'll hear teh exact same story. L3 will probably blame all the ISPs for their lack of bandwidth interconnecting them. And all the other US isps will say, they're not the issue because their internal networks aren't close to full saturation.
So the real question at hand is, who really should pay for upgrading the capacity between the tier 1 interconnects.
Sounds like there was a handshake agreement between the companies originally. This handshake probably didn't include a late comer like level3.
I would argue that if l3 approached these companies and said, we'll install the cards AND maintain these connections on L3's bill, everything will get resolved.
Ie. L3 is pretty much saying, sure we're tryign to compete with you big ISPs, and we're gaining customers to your detriment on the Inet tier1 space, but we can't seem to maintain our client base if you guys at the last mile/end point don't increase your bandwidth with us.
The 6 other major ISPs are saying to the content providers, why bother using L3, sign up with our own t1 bandwidth and we can provide to your customers directly a great service.
Individuals, and individual rights, are like single atoms. They only exist in the abstract sense. The real world is entirely dominated by groups and collective actions.
You're a confused anarchist. The problem with non-coercive government is that all government is coercive. Government is primarily a set of restrictions on the use of force, or alternately the monopoly on that use of force. Getting rid of a government, or disarming it, merely allows anyone with a larger arsenal to set up their own government -- anarchy is an unstable system. We all have a right to violence, because it cannot be taken from us except in extreme situations. Remember, the Code of Hammurabi was instituted, "...so that the strong might not harm the weak." Coercive government is a necessary evil, and it will remain necessary so long as men are capable of harming their fellows, for that is its justification and primary purpose.
Rights are not inherent, except in some abstract sense. In the real world, your rights are what the men with guns say they are. You may feel fortunate that the world has had a long, bloody time to work out semi-cooperative frameworks to restrain our darker impulses. Individual rights are an important conceptual counterbalance to the overwhelming powers of the collective, but they are no justification for anarchy, economic or otherwise. The "free market" is an ideal, even a good one, but in most cases removing government interference makes markets less free, more subject to collusion and fraud. In some cases, where the service is required to be universal, or when the barriers to entry would be insurmountable, it makes sense for the government to assume these functions directly. Govenment can also be thought of as the natural monopoly of natural monopolies, in that sense.
Slavery is a word that has a specific meaning; your definition is specious. You just fundamentally don't like being told what to do. To some degree this idealism is admirable. For the true individualist, I can recommend (from long personal experience) the Alaskan wilderness; you can get land for free still up there, provided you build upon it. Whatever romantic images your mind conjures upon thinking of Alaska are all true; I can't stand the weather, personally, but it's as close to a pure state of nature as you will ever find. If you'd like to enjoy the benefits of society, however, you have to play by the rules. "Slavery" isn't an option -- it's mandatory.
Those who advocate genocide deserve every protection afforded by law, and none afforded by common human decency.
Level3 schools Verizon.
That would make too much sense. You can also look at it as if Verizon is supplying the eyeballs (consumers) and Netflix is consuming them. Netflix (and therefore L3), can't make money if they can't get their content to the subscribers, so Verizon can hold them hostage. It also doesn't hurt that Verizon has its own competing product and if there are enough problems with Netflix, some of their subscribers might convert, making them even more money. Oh and, of course, you can't change providers, that would be something dangerous, like competition.
Then they can throttle proportionally to how you downgraded. You cut your service in half, and they cut your throughput allowance to a particular site in half on the throttling server.
Deregulate!! Deregulate!! Deregulate!!
Its a good thing to be known as Verizon management.
Brutal management trolls throttling their own kids and family members for money every day of their living life.
Allow more competitors in for every square mile of customers
and they will fsck off with all the good customers!!!!!!!!!!!!!!!!!
So in other words, an intentionally misbehaving caching proxy. Good luck getting subscribers to install the root certificate that lets the ISP forge such responses to HTTPS requests.
Well, NetFlix could also enter into agreements with ever backbone provider, thereby forcing Verizon to either do the same to everyone or start upgrading
This brings to the question of why Netflix has chosen to deal with Verizon instead of with Level3 directly in the first place ?
Even if Netflix didn't know of the existence of Level3 (which I find too ludicrous to be possible) that they had signed up with Verizon, they could have changed the situation right now by dealing directly with Level3, and why wait anymore ?
Muchas Gracias, Señor Edward Snowden !
http://blog.level3.com/global-...
"A port that is on average utilised at 90 percent will be saturated, dropping packets, for several hours a day. We have congested ports saturated to those levels with 12 of our 51 peers. Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers.
That leaves the remaining six peers with congestion on almost all of the interconnect ports between us. Congestion that is permanent, has been in place for well over a year and where our peer refuses to augment capacity. They are deliberately harming the service they deliver to their paying customers. They are not allowing us to fulfil the requests their customers make for content.
Five of those congested peers are in the United States and one is in Europe. There are none in any other part of the world. All six are large Broadband consumer networks with a dominant or exclusive market share in their local market. In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers."
We don't have all the information here. The Level 3 post only talks about how much the hardware costs.
Level 3 very explicitly avoided mentioning services charges which Level 3 may (or may not) have been trying to get from Verizon. Note that Level 3 didn't explicitly say they would provide the extra connection bandwidth free of charge, only the hardware. I think there is more going on here than we know...
Of course, it could also be that Verizon is trying to get money from Level 3 because Level 3 was sending so much more traffic than Verizon. I am a very experienced network software designer and it is more expensive to receive IP traffic than to transmit it -- especially non-TCP traffic, because there is no flow control -- you need a lot of extra hardware capacity to handle bursts (buffering and CPU). From a technical standpoint, it makes a lot more sense for Level 3 to pay Verizon to handle the extra traffic.
BUT, compared to the cost of the entire network, the real cost of the peering is probably pretty insignificant for both parties. The only conclusion here is that they are probably both spinning the message by leaving out the unflattering information.
An engineer who ran for Congress. http://herbrobinson.us
Well, NetFlix could also enter into agreements with ever backbone provider, thereby forcing Verizon to either do the same to everyone or start upgrading
This brings to the question of why Netflix has chosen to deal with Verizon instead of with Level3 directly in the first place ?
Even if Netflix didn't know of the existence of Level3 (which I find too ludicrous to be possible) that they had signed up with Verizon, they could have changed the situation right now by dealing directly with Level3, and why wait anymore ?
NetFlix has contracts with lots of folks - Level3 included. These are with respect to pushing content from NetFlix over the backbone into various networks.
NetFlix also promotes having a CDN end-point within an ISP's network to alleviate the need for as much peering; which is what I believe the NetFlix-Verizon deal was about, which Verizon may have (or may not have, we don't really know) charged NetFlix for installing in the datacenters/hubs/central-offices.
However, the fact that NetFlix has done that, which should IMPROVE speed on Verizon's network, and there are still major issues shows that there is something else wrong with Verizon's network. Of course, they might rely on the back bone having sufficient capacity to pull down the information over the CDN too; or it might be that NetFlix installs a direct pipe for the CDNs, we don't know the details. Most likely NetFlix has a contracted pipe with a Level3 interconnect to these, and that is why we're hearing all about it between Level3 and Verizon as Verizon doesn't want to increase their interconnects with Level3 over which those CDN systems are suppose to operate.
But that's just a bit of (educated) guess work as I don't know the details of the Netflix-Verizon arrangement or the network layouts or the NetFlix CDN provisions.
Truth is like the sun. You can shut it out for a time, but it ain't goin' away. - Elvis Presley (source: imdb.com)