Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

MojoKid (1002251) writes The ongoing battle between Netflix and ISPs that can't seem to handle the streaming video service's traffic, boiled over to an infuriating level for Colin Nederkoon, a startup CEO who resides in New York City. Rather than accept excuses and finger pointing from either side, Nederkoon did a little investigating into why he was receiving such slow Netflix streams on his Verizon FiOS connection. What he discovered is that there appears to be a clear culprit. Nederkoon pays for Internet service that promises 75Mbps downstream and 35Mbps upstream through his FiOS connection. However, his Netflix video streams were limping along at just 375kbps (0.375mbps), equivalent to 0.5 percent of the speed he's paying for. On a hunch, he decided to connect to a VPN service, which in theory should actually make things slower since it's adding extra hops. Speeds didn't get slower, they got much faster. After connecting to VyprVPN, his Netflix connection suddenly jumped to 3000kbps, the fastest the streaming service allows and around 10 times faster than when connecting directly with Verizon. Verizon may have a different explanation as to why Nederkoon's Netflix streams suddenly sped up, but in the meantime, it would appear that throttling shenanigans are taking place. It seems that by using a VPN, Verizon simply doesn't know which packets to throttle, hence the gross disparity in speed.

Comments

Thanks

[Anonymous Coward]

Now they'll just throttle VPN traffic too.

Re:Thanks

[Casualposter]

SO when you pay for that service it says something like "up to 75mbps" which in reality means that the speed test and google's home page could see that much speed and everyone else will look like dial up from the 1990's.

It would be much better if the services had to advertise their average speed across the most popular sites. That way if they throttle Netflix to .375mpbs, they have to inform customers that while they are paying $125/month for "blazing fast speed" they are actually getting blazingly fast dial up speeds.

Re:Thanks

[mysidia]

SO when you pay for that service it says something like "up to 75mbps" which in reality means that the speed test and google's home page could see that much speed and everyone else will look like dial up from the 1990's.

I have a suggestion.... Web browsers should take some measurements and display prominently in a visible status bar or other location.... average TCP throughput --- And Estimated average bandwidth;

Both a "this site" value, a "this browser session" value, and (Optionally) if the user decides to share their numbers, Community average bandwidth for this site, Community average bandwidth for this ISP, and Community average for this site on this ISP.

If Community average for this site on this ISP is more than a standard deviation below Community average for this site,

Then a little warning exclamation point should appear to the right of the browser bar. On mouseover, and for a few seconds after loading the page, a little warning bubble should appear for a few seconds. "Your internet service provider seems to have below average performance in loading this page."

Re:Thanks

[Anonymous Coward]

While such numbers might result in some net gain, it will probably end up pissing off ISPs to the point of either finding ways of faking the data, blocking the data, or just as policy telling customers to ignore the speed numbers. I still remember the stints I had doing help desk with people calling in for "MSN is down, but rest of the internet works, fix it fix it fix it," and "joesbasementserver.com is loading really slow, something must be wrong with my computer." A lot of people won't be able to distinguish when something is their ISP's fault and when it might be the end servers fault. In some cases, such numbers will make it clear, although in other cases, even when the numbers make it clear, people will use them to argue the opposite fault.

Re:Thanks

[sumdumass]

Wrong..

It is fraid for any ISP to purposely limit speed and bandwidth below the advertised speed speed while the network can handle it (congestion). No matter how you look at it, the up to speeds will never be availible when they purposely limit it.

This should be dealt with by consumer protection law (paying for services not delivered and possibly bait and switch or a host of others). People need to complain in those terms to thier state utilities commision or consumer protection department and file lawsuits over the said laws. Verizon and other ISPs will stop doing it and possibly be fined in the process.

Re:Thanks

[mysidia]

It will probably end up pissing off ISPs to the point of either finding ways of faking the data, blocking the data, or just as policy telling customers to ignore the speed numbers.

If the data is blocked, the browser should figure out why and explain to the user that there seems to be an issue with their network; in other words "Blocking" should make it even worse for the ISP. a smarter browser UI could be a tremendous help to support technicians, which the ISPs should absolutely love ---- perhaps even tell the user exactly which entity to contact, even display their ISP's support number on the screen, to help accelerate the problem resolution process, and providing access to comments by other users of the same ISP, leading to happier customers, and customers who can share info with each other pertinent to troubleshooting or why this is happening, etc.

A lot of people won't be able to distinguish when something is their ISP's fault and when it might be the end servers fault.

I am suggesting the browser should also take some responsibility to the interpretation of the results here. There should be a highly visible "troubleshooting" button that causes some tests to be run. Explanations should be right there in a natural language that any English speaker could understand.

The browser should not show an alert if there is not enough data to make a conclusion with a fair measure of statistical confidence.

We can definitely make a strong distinguishment between a "web site performance issue" and a client connectivity issue, with data from a sufficient number of users.

The browser would also need to take into account geographic location and client connectivity, however.

e.g. Is the site slow because the visitor is half way around the world from the nearest mirror, or is it slow because they're connecting over congested WiFi or 3G networks, instead of a wired connection?

I realize it's not "easy", but the web browser is the only software component that is in a position to take the kinds of measurements that are required and help alert the user to the problem, tell the user which entity they should contact, and assist with troubleshooting.

Re:Thanks

[davester666]

because you generally are getting that rated speed....to Verizon's Internet or Comcast's Internet or whatever.

what they never mention is that most web sites don't pay extra to be part of Verizon's Internet or Comcast's Internet, so they get throttled by poor interconnects.

Re:Thanks

[mysidia]

You could even run a network monitoring app. But the browser is one highly visible one that most people already have installed.

Perhaps you could, but now essentially you are having "users that think they have problems" downloading an extra application and they start monitoring after there's a problem most likely.

This means your app cannot get the right data on what's normal for the user or for the world, because you have a sample of app users that are biased towards users that already are experiencing network issues of some sort, and you don't have a good baseline for the user that installed it either.

Re:Thanks

[sociocapitalist]

SO when you pay for that service it says something like "up to 75mbps" which in reality means that the speed test and google's home page could see that much speed and everyone else will look like dial up from the 1990's.

I have a suggestion.... Web browsers should take some measurements and display prominently in a visible status bar or other location.... average TCP throughput --- And Estimated average bandwidth;

Both a "this site" value, a "this browser session" value, and (Optionally) if the user decides to share their numbers,
Community average bandwidth for this site, Community average bandwidth for this ISP, and Community average for this site on this ISP.

If Community average for this site on this ISP is more than a standard deviation below Community average for this site,

Then a little warning exclamation point should appear to the right of the browser bar.
On mouseover, and for a few seconds after loading the page, a little warning bubble should appear for a few seconds.
"Your internet service provider seems to have below average performance in loading this page."

So tcp 80 won't be throttled but whatever netflix (or whatever) uses will be -

Could be a different route involved for the VPN

[Anonymous Coward]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

Re:Could be a different route involved for the VPN

[knightghost]

Not so much of a leap since Verizon and Comcast have admitted to such.

Wouldn't a simple tracert show the route (and any differences)?

Re:Could be a different route involved for the VPN

[pla]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

Failing to have peerage agreements in place to honor your downstream sales commitments is a form of throttling - Or, I would daresay, a form of outright fraud.

If I offer to sell you "unlimited" beers from my fridge for $50 a month, but I only resupply it at a rate of one six-pack per week, I have intentionally cheated you. That basic relationship doesn't magically change because of some hand-waving technobabble about peerage agreements and network congestion.

(Yes, I know those don't strictly count as technobabble, and what they really mean - But they effectively reduce to Verizon having zero interest in upgrading its infrastructure to support its commitments to their customers as long as the FCC and FTC will allow them to outright lie)

Re:Could be a different route involved for the VPN

[Anonymous Psychopath]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

This is exactly what's happening. I do the same thing for a specific server I use. Standard routing via FiOS results in consistent 1mb download speeds. I set up a GRE tunnel to my VPS host and I get consistent 10mb download speeds. The culprit appears to be a shitty peering connection between so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) and 0.ae2.XL3.CHI13.ALTER.NET (140.222.225.187).

Re:Could be a different route involved for the VPN

[Drakonblayde]

You do not understand how BGP works.

The problem isn't the data that verizon is sending to netflix, it's the data netflix is sending to verizon. Verizon messing with routing policy on netflix's announced prefix's wouldn't have an effect on verizon's streaming speeds. The traffic flows from Netflix to Verizon.

Therefore, in order to influence streaming speeds, Verizon would have to change their routing policy on how they announce their own routes in order to influence which links netflix traffic can come in on. The problem is, there's no way within BGP for Verizon to say I want Netflix traffic to only come in over these specific links. It would influence *all* traffic from that peer. Routing policy is destination based, not source based, and not source-destination based. By simply announcing the routes are more preferable over Level 3 saturated links, that forces traffic Level3 delivers for those prefixes to come in over those links.

Sure, Level3 could do some traffic engineering of their own and ignore or mutate some parts of Verizon's route announcements, and force that traffic in over unsaturated links Verizon may have with Level 3 (if there are any), but Level3 is a middleman. Doing so would take them out of their middleman status and put them firmly on Netflix's side. Verizon's likely response would be to immediately de-peer Level3.

The only folks who can effectively change how the traffic reaches Verizon's network is Netflix. They determine their outbound routing policy, but only up until their own border. Once it transits to another AS, it will be forwarded according to the upstream AS's routing policy. If Netflix wants to avoid saturated Verizon-L3 interconnects, the only thing they can do is not send traffic to Level3 for Verizon prefixes. They could easily modify their inbound route policy to send traffic for Verizon's prefixes via another peer. This is something that Level3 does not want, because it effects their revenue, hence their seeming to take sides with Netflix on the matter. It's one thing for Level3 to have an opinion on what Verizon is doing, that doesn't really effect operations. The second you change operations to try and force that opinion, well, you're likely to invoke the Law of Unintended Consequences.

Now, for whatever reason, Netflix has decided to go ahead and keep sending Verizon traffic to Level3. The reality is that if Verizon has decided to be douchebags about this, then they can do the same thing for whatever peer the traffic is ingressing through. Maybe all of Netflix's other peers ultimately transit to Verizon via Level3 anyway, which would make any change of forwarding policy moot.

About the only way for Netflix to solve this is to go ahead and cut out the middleman and just pay Verizon directly for interconnects into their network. This is what Verizon wants: another revenue source for traffic their going to deliver anyway. This is what Level 3 does not want: When you cut out the middleman, the middleman makes no money.

Netflix has already done it with Comcast and AT&T, so it's not surprising that Verizon wants in on this action as well, and will continue to be douchebags about it.

In the meantime, savvy customers can come up with their own solutions in order to avoid having netflix traffic destined for them coming in over saturated links. VPN and tunneling are two perfectly valid solutions.

Re:Could be a different route involved for the VPN

[140Mandak262Jamuna]

If I offer to sell you "unlimited" beers from my fridge for $50 a month, but I only resupply it at a rate of one six-pack per week, I have intentionally cheated you. That basic relationship doesn't magically change because of some hand-waving technobabble about peerage agreements and network congestion.

This analogy is a little flawed. Let me correct it. Let us say the local municipality has granted pla (258480) a local monopoly in selling beer to its residents. And you sell beer at different service level all unlimited number of trips to the fridge, but at 1 trip/hr, 1trip/6 hours, 1 trip/min, 1 trip/sec etc. And you stock it with brewed-by-your-local-sewage-company beer all the time, and stock Buds, Coors and Coronas one bottle a month. Then your analogy is complete.

What is really insidious is, pla is NOT buying any beer. All the beer companies come stock the fridge for free. Pla's only cost is keeping the beer cool. And it does not cost any more to cool a bottle of Corona than to cool a bottle of brew-from-sewer. Just because pla noticed people are drinking Corona more, pla wants Corona to pay him more money. Remember it is a monopoly. Corona has no other way of selling its beer without going through pla's fridge. Now you get the idea.

Re:Could be a different route involved for the VPN

[mjm1231]

This doesn't change the fact that the customer paid for 75Mbps and got... a lot less.

Re:Could be a different route involved for the VPN

[alen]

there are a dozen video providers i can name who stream video with no problems. Hulu, HBO Go, Amazon, Vudu, Cinemanow, Apple itunes, ABC, History Channel, Disney, Lifetime, PBS, Fox, ESPN, NBC and others. most of this is video on demand, but they have some live video they stream as well.
only netflix is having these issues. difference is that everyone uses a CDN to host their content inside the ISP's network or close to. there are at least a half dozen CDN's that do most of the hosting
the way it has worked for 15 years since Akamai made the first CDN because people figured out a long time ago you can't send large files over the internet. the way it works is the CDN pays the ISP for hosting and bandwidth and the customer pays the CDN
netflix used to use one of these CDN's, forgot exactly who but they let the contract expire last year
instead netflix came up with a cool CDN box but they demand ISP's host it for free and give them free bandwidth as well. i guess netflix got so big they think they can extract payment from everyone

and if there is a lawsuit it will be brought up that this is only a netflix issue and there would be lots of discovery on netflix's business for the last 5 years

Re:Could be a different route involved for the VPN

[InvalidError]

Failing to have peerage agreements in place to honor your downstream sales commitments is a form of throttling - Or, I would daresay, a form of outright fraud.

Only problem with that is Verizon has TONS of under-used transit capacity with other networks - when Verizon posted their thing about peering points with Netflix's partners, they also mentioned that their transit to other networks at times where Netflix was hitting 100% was only ~40% on average.

So, Verizon would have plenty of transit capacity if it was spread more evenly across all the peering Verizon has.

Re:Could be a different route involved for the VPN

[itsenrique]

"i guess netflix got so big they think they can extract payment from everyone" Now there's a good one. In reverse shill-logic world perhaps. For those interested in what's really happening I'll point you here ( http://www.extremetech.com/com... ). Headline: Verizon caught throttling Netflix traffic even after its pays for more bandwidth. And that is basically what they are doing, artificially restricting Netflix not going through VPN to (arguably) criminally low speeds by means of not upgrading hardware on purpose to thwart who they view as "competition. Although I'm not sure Verizon will sell you anything remotely useful for $8 a month. I quit Verizon for this reason although I never told the CS rep because they try to make it hard to quit anyway. Verizon seems to be trying to fool everyone with (what seems to most people) lots of mumbo jumbo and outright deception, I for one hope they don't continue to get away with this attempt to make there "competitors" look bad.

Re:Could be a different route involved for the VPN

[bigpat]

That is a load of horseshit!

Or let me be clear. That is a load of horeshit technobabble meant to obfuscate and mislead. Level 3 was pretty clear the other day when they offered to spend a few thousand dollars to upgrade their links to Verizon. Level 3 is a backbone Internet provider. There is no reason that any link between it and another network should remain saturated if both sides are acting in good faith to serve their respective customers, especially when L3 was willing to pay the costs to upgrade Verizon's own equipement to handle more traffic which it shouldn't have had to do because it is Verizon's customers who are requesting and already paying for the content in the first place.

Verizon is choosing to not upgrade its connections to shake down Netflix, and thus pass those costs on to Verizon customers. Period.

Re:Could be a different route involved for the VPN

[geoskd]

that link is dedicated netflix and it limits them to the amount of data they send. last year super hd was for a few selected ISP's but then netflix started sending it to everyone over Level 3 and screwed up everyone's service the point is netflix is trying to increase costs on their business partners who will then have to increase prices of their customers. customers will hate the ISP but like netflix. same strategy as TV companies have used with cable TV and forcing them to sell bundled channels, intel has done this, a lot of companies have done this. customer hates the company they do business with for high prices, but it's really because they are being forced to provide services some may not want current system is not perfect but it ensures that people who use the service pay the costs and not everyone pays

Verizon was given an "out" and they refused to take it. Netflix offered to provide co-located CDNs, and all Verizon had to provide was electricity and space (both of which are negligible compared to the cost for Verizon to pay for bandwidth.) Verizon elected not to take the option that would save them money, deciding instead to play a stupid game of chicken with Netflix. While this idiot game may work with smaller companies, Netflix is now the 800Lb gorilla, and Verizon has nothing but downside on this deal. Their best bet would be to quietly put their tail between their legs and give Netflix the Co-lo's they had been asking for. Verizon needs to understand that they do not really have any chance at becoming a significant content provider, and they should know their place. (getting my bits from place to place.) They offer the consumer specific bandwidth, which as far as the consumer is concerned includes Netflix traffic. Verizon needs to understand that in the consumers mind, the customers pay Netflix to provide the bits, and they pay Verizon to get them from Netflix to their device. Verizon can claim all they want to about Netflix not paying for this, or not paying for that, but as far as I am concerned, I am paying Verizon to deliver the bandwidth door to door. I have already payed Verizon for it, and now they are failing to deliver what I paid for.

Re:Could be a different route involved for the VPN

[Xenx]

As for number three.... Netflix is willing to provide their CDN hardware to ISPs for free. The ISP just has to house the hardware. I don't know enough about how all the back end works, so fair and equitable is up in the air. But, Verizon doesn't want to go that route without getting paid either.

Re:Could be a different route involved for the VPN

[hawguy]

that link is dedicated netflix and it limits them to the amount of data they send. last year super hd was for a few selected ISP's but then netflix started sending it to everyone over Level 3 and screwed up everyone's service

the point is netflix is trying to increase costs on their business partners who will then have to increase prices of their customers.

Netflix isn't trying to increase costs to ISP's, they aren't forcing data to their subscribers -- it's the ISP's customers that are already paying for broadband who are demanding high quality video to feed their 1080p (and soon, 4K) big screen TV's. What reason is there to pay for a 75mbit connection if you're not planning on using large amounts of bandwidth?

If Verizon has to charge their customers more money to provide them with the network capacity they thought they were already paying for, then that's what they should do. They shouldn't try to extract money from content providers to artificially subsidize internet connections to keep costs low -- this causes a larger barrier to entry to smaller ISP competitors that don't have the leverage to extract costs from content providers.

Re:Could be a different route involved for the VPN

[Austerity Empowers]

Never say "bandwidth provider" to a telco monopoly. It grates their teeth and raises their dental insurance. They've been staring down the barrel of this reality for 20 years and have done everything they can do, bought every politician and generally screamed bloody murder to get us to where we are. They seriously think the world wants to go back to using them as their AOL/Prodigy/Compuserve.

Re:Could be a different route involved for the VPN

[NoKaOi]

Why should Verizon spend many billions of dollars to subsidize Netflix?

They're not. Verizon's customers are paying them to provide a service. Just because a bulk of the traffic is coming from a particular source doesn't mean it's okay for them to charge their customers for a service that they're not providing. It all comes down to Verizon trying to double-dip.

Re:Could be a different route involved for the VPN

[evilviper]

Netflix offered to provide co-located CDNs, and all Verizon had to provide was electricity and space (both of which are negligible compared to the cost for Verizon to pay for bandwidth.) Verizon elected not to take the option that would save them money

If Netflix got a free CDN setup without paying ISPs anything, Verizon would quickly see all the other CDNs refusing to pay them, too.

On that same note, I know where you can get a FREE 40-hour/week job... You won't have to pay a penny for this FREE job.

Re:Could be a different route involved for the VPN

[NeutronCowboy]

Verizon is choosing not to upgrade it's peering points with Level-3 because they are no longer evenly sharing traffic up/down as all free peering arrangements have ALWAYS required, yet Level-3 doesn't want to pay for the imbalance, and Netflix doesn't want to shift some of their Verizon traffic to a different transit provider than Level-3.

Considering the huge imbalance in download and upload speeds, how exactly is anybody supposed to peer with Verizon? Verizon knowingly set up a situation in which it is impossible for any peer to be on traffic parity with Verizon. Furthermore, traffic parity is almost impossible from a business perspective. Verizon and the last-mile providers have consumers and creators at one end, everyone else has pretty much only creators. The only way for corps like Level 3 to achieve traffic parity is to offer last-mile services, which is impossible, because Verizon frequently has a local monopoly.

So - the technobabble refers to the fact that the technological discussion is largely irrelevant when it comes to Net Neutrality. Anyone trying to bring technological issues into the discussion is just trying to muddy the waters of what is a market power discussion.

Re:Could be a different route involved for the VPN

[Chibi Merrow]

Netflix isn't using any of Verizon's bandwidth.

Verizon's customers are using Verizon's bandwidth.

Alternative explanation

[IamTheRealMike]

Routing traffic via the VPN changes the path the traffic flows over, possibly avoiding routes that are saturated and (who knows) pending upgrade.

It's tempting to imagine the internet as a giant blob of fungible bandwidth, but in reality it's just a big mess of cables some of which are higher capacity than others. Assuming malice is fun, but there isn't enough data here to say one way or another.

Re:Alternative explanation

[Ecuador]

Except Verizon here lets just some "low capacity" cables connect them to Netflix's provider on purpose (as illustrated in a recent /. article), so there can be no other reason apart from extorting money. And for the speed to actually going down with time it probably means that instead of "upgrading" capacity they are probably doing the opposite to force Netflix. And they are lowering the speed slowly otherwise their customers would figure it out and start rioting (but many don't have any recourse as in, alternative ISP).
Remember, Netflix actually offers installing their servers within the ISP's network for free, which would mean no interconnection.
And, finally, this is 1 week old "news for nerds". I have read about it in so many tech sites, I was certain it would be a dupe (but a quick search seems to indicate it is not).

Re:Alternative explanation

[jaseuk]

This has been rather done to death (http://www.extremetech.com/computing/186576-verizon-caught-throttling-netflix-traffic-even-after-its-pays-for-more-bandwidth) , but Verizon doesn't appear to be throttling or shaping Netflix. They are running their peered links to Layer 3 at 100% capacity. Traffic that doesn't go via Layer 3 does not suffer. So if you find an alternative route that doesn't use Netflix's Layer 3 peering connections (such as a VPN) then things run well.

For this to be resolved, people really need to find non-Netflix services that are equally impacted and bring this up. It may well be that 90% of Verizon's Layer 3 pipe is for Netflix, but there are bound to be other services suffering. If this can be demonstrated this puts other parties into the equation and should encourage Verizon to take up Layer-3s offer of additional free peering capacity.

I suspect that Verizon would rather that Netflix isn't running at full-speed as it quietens down their overall network usage and can somewhat claim they are not capping or throttling. Perhaps Layer-3 should shut down these peering points for maintenance and let the Verizon find a way through another peer / transit, it might melt the whole of Verizon's network that way and encourage them to solve it.

Jason.

Re:Alternative explanation

[amorsen]

Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

No. That is not how it works. The truth is that the smaller provider pays the larger provider, no matter which direction the traffic flows. Some companies, like Netflix, are nice enough to not use their size as an excuse to charge people -- they offer free peering at internet exchanges. Other companies are maximally greedy.

Re:Alternative explanation

[Anonymous Psychopath]

It's not just Level3 (not Layer 3), it's also Alternet and possibly others. Peering has gotten tough. It's supposed to be hey, let's connect our stuff together because I want to send you a bunch of stuff and you want to send me a bunch of stuff and we both win. The Internet has evolved and that has resulted in asymmetric traffic flows where one party carries more (sometimes far more) of the burden than the other, but the cost models have remained the same.

In Verizon's mind, they receive no benefit from increasing peering capacity in cases where they receive far more traffic than they can send. They forgot one thing, though; their residential customers. They are the ones who need the additional capacity, and without it their service will continue to degrade.

Re:Alternative explanation

[jedidiah]

That is true perhaps.

However, this is all entirely Verizon's fault. They are the entity in this arrangement that has actively encouraged assymetric use of the net by offering assymeteric service. It's really rich to see ISPs complain that they are getting too much traffic all in one direction then that's how they f*cking design their service.

Verizon is selling massive downloads. So is every other consumer ISP.

Re: Alternative explanation

[MrL0G1C]

You obviously missed the article where Netflix supplies a tower-pc sized box with all of netflix on it to ISPs for free:

Netflix Boxes

Re:Alternative explanation

[kqs]

...because Netflix's provider (which is Level3) isnt paying for the bandwidth disparity between Level3 and Verizon on purpose.

The bandwidth disparity argument is bunk. I've love if Netflix or Level 3 would set up some data sinks in their network so I could use my FIOS to send them random data 24/7 and help even the disparity.

Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

So... you're saying that Verizon should be paying me! I mean, they send me ALL THIS DATA (much of it sourced from Netflix), but I hardly send them anything. This makes me both a selfish person and someone who deserves a large monthly Verizon cheque.

Note that if Verizon doesn't want to pay a few grand for a few more 10GE ports and some short cables, they could pay even less and accept the caching servers that Netflix offers to all large ISPs; those cost just a few rack units and watts of power.

But Verizon would rather limit Netflix so that they can push their own video products.

Re: Alternative explanation

[kqs]

No, Netflix (and Youtube and some others large ones) don't buy CDN hosting; they offer it. They offer free CDN servers which large ISPs can put in their datacenters. Doesn't matter how much Netflix offered to pay, I doubt if any existing CDN could handle Netflix's traffic along with their other customers.

Many ISPs take advantage of this, but Verizon would rather degrade Netflix's products so they can push their own products.

Re: Alternative explanation

[Drakonblayde]

There are at least three underlying problems for the congestion issue - one is the DMCA and related copyright laws that prevent any sort of sane caching, the general fear of multicast that everybody on the Internet still seems to have (half a million unicast streams of the same show is insane - where are the global warming people on this?), and the grants of monopolies and/or prohibitions on competition that prevent local competition.

I agree with you that half a million unicast streams can be nuts, when it comes to on demand content, multicast is a non-starter (we'll ignore the fact that multicast is sorely lacking in security features and would require some serious re-engineering on many networks to work... there's a very big reason why inter-domain multicast routing is not seriously employed, and it has nothing to do with fear).

Think about how this traffic flows -

Sub1 wants show A and starts playing it on Netflix.

10 minutes later, Sub2 wants show A as well.

What happens if this stream is multicast? Well, Sub2 gets the show 10 minutes later, assuming he's joining the same multicast group as Sub1. Sub2 is not happy, he wanted to watch the entire show.

How to get around this? Well, ok, so Netflix could just mux the feed for Sub2 inside the feed for Sub1, and presumably, the client would be able to tell which parts of the feed were for which sub. However, the problem is that the data for Sub2 would still be delivered to sub1, sub1 would just throw it away and pay attention to it's own data. However the data for both feeds have to transit the same backbone, which drives capacity usage up. This is also unsustainable, as eventually, as more subscribers joined, the feed would grow so large that it would saturate the downstream of all the subscribers receiving it and eventually lead to packet loss.... which would lead to loss of video, stuttering, etc. All the same shit thats going on now.

Ok, so muxing different streams into the same feed for the same show isn't going to work.

So Sub2 could just start getting the feed over a different multicast group, that would solve the ever growing feed problem!

Except that if you do that, there is functionally no difference between sending the traffic to a multicast destination and a unicast destination.

So sure, while sending half a billion unicast streams seems insane, especially since alot of those will be watching the same show, the fact is that a very small percentage are going to be watching the same show at the exact point in the show. For the most part, they will all be at different parts. Multicast is a solution when the data for the given event is live, or when it's linear (ie, this is the point we're at, and there's no going back). On Demand services does not fit either of those profiles.

Multicast has it's place in the on demand world, but only on the backend. It's wonderful for things like distributing a new asset to all of the streaming sources, or for filling caching servers that will the streaming boxes will need to pull from, but multicast is simply not a workable or superior implementation when it comes to delivery of content to subscribers.

Re:Alternative explanation

[ArcadeMan]

People assume that the Internet is a strict connection from server to client. But actually from a non-linear, non-connected viewpoint it's more like a big ball of wibbly-wobbly, streamey-wimey... bits.

Re:Alternative explanation

[Drakonblayde]

I suspect that whats going on is that Netflix put the majority of their traffic on Level3 and Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer. Verizon said "No" and told Netflix to go with another peer. So Verizon has plenty of bandwidth, Netflix has plenty of bandwidth... it's where those peers are located that's the problem.

Ok, but you're wrong.

Level3 has admitted they have settlement free peering with Verizon. Level3 does not pay Verizon anything. Verizon does not pay Level3 anything.

Netflix pays Level3. This is why Level3 gives a shit about this situation.

What's going on is that Verizon is trying to cut out the middleman. Verizon wants Netflix to pay them to get traffic into their network instead of paying Level3 to deliver traffic into the Verizon network. Why? Because they don't make any money from Level3.

Naturally, Level3 is all in a huff about Verizon trying to fuck with their revenue stream.

Re: Alternative explanation

[faedle]

Bandwidth is perhaps cheaper than you suspect.

I worked for a regional ISP that serves about 50.000 subscribers. We had multiple 10 Gigabit Ethernet connections to various peering points, one of which happens to be where Netflix peered with us. Total cost for that peerage: the cost of the extra fiber capacity, plus engineering the peer.

As opposed to housing Netflix servers at our data center. First off, to service that many potential streams might require a few boxes and a not insignificant storage array. We actually did have a similar arrangement with another very large content provider: their stuff took about a half-rack. It then needs to be added to network monitoring, and you need to train your NOC staff what to do when that little red light comes on. And the equipment will fail: the "other content providers" equipment had a MTBF of a couple of months. The hard drives will take a pounding.

And we were small enough that when we asked Netflix to co-locate in our data center for free they actually said "Not interested."

Re:Alternative explanation

[evilviper]

Level3 does not pay Verizon anything. Verizon does not pay Level3 anything.

Yes, but there's 2 things going on that you're neglecting to mention.

1) When Level3 started delivering Netflix streams, suddenly their peering traffic was extremely unbalanced... That's when one side has to start paying the other. Verizon wants Level3 to pay, but they won't, so Verizon refuses to upgrade the interconnect. In the bad old days, these peering disputes ended with a disconnect, and big portions of the internet broke off and many sites went offline. These days it's just lots of congestion.

2) Level3 seriously pissed-off ISPs, when they entered the distributed content delivery business. ISPs get paid good money by Akamai and others to host their caching servers. Level3 thought they could get in on this racket by hosting servers just outside their ISP peering points, cutting the ISPs out of the profits. This pissed off ISPs, and more than that, massively screwed-up their peering ratios.

So, on both sides, Level3 is the one who screwed-up their peering agreement with Verizon and others, without considering the consequences.

Netflix has the ability to fix it, though... If their software would tell all the clients to upload random junk to some random Netflix servers (preferably UDP, so the server doesn't even have to really exist), even when idle and not watching videos, they could move Level3's ratios back to even up/down distribution, and really punish the local ISPs who claim they want even up/down peering, at the same time.

Role reversal

[Anonymous Coward]

I have to wonder, what would happen if customers were to start throttling the payment of ISP's?
"You will get your payment when you actually fulfil your end of our contract, but not before."

Re:Role reversal

[Travis Mansbridge]

You really have to wonder? Late payment fees is all they will get.

Re:Role reversal

[Omeganon]

(*) actual speeds not guaranteed.

It's in every agreement so to them, they _are_ providing the service they claim.

Re:Role reversal

[wvmarle]

Those customers will just get disconnected.

Re:Role reversal

[green1]

Simple, they cut off your service for non-payment, and you move your internet connection to the competi....er... well, does Netflix still run a DVD service?

Re:Role reversal

[ToasterMonkey]

I have to wonder, what would happen if customers were to start throttling the payment of ISP's?
"You will get your payment when you actually fulfil your end of our contract, but not before."

You should go review your contract.

Too bad it won't do any good.

[DewDude]

Verizon doesn't care. They own RedBox Instant; they last thing they want is customers using Netflix. We're not gonna get net neutrality out of the FCC (the public comments are a sham; the FCC only care about the businesses involved in the decision); so this is not going to get fixed. If Netflix uses Level3; they were cripple all level3 connectivity.

What the hell kind of dialup did you have?

[Anonymous Coward]

375k!?

No, I get your point... just...

A little disappointed in slashdot

[butchersong]

This is not news. Verizon, Netflix and customers all already know what the problem is. Routing around the bottleneck (saturated interconnect) by using a VPN will obviously avoid the bottleneck... I don't understand why this was posted as a story.

Re:The Answer is Violence

[alphatel]

Verizon is too big, and our government does not care.

The only answer is to actively work to destroy Verizon until they acquiesce or no longer exist.

Strange, it feels like we already did that once.

Seperate content creation from Bandwidth provision

[bl968]

The FCC should make Comcast and Verizon and all these other companies chose... They can be content creation companies or bandwidth providers, but not both. This is all about making it easier and more convenient to pay them for their video services than to attempt to get it from a third party it's monopolistic and anti-competitive behavior...

Difference is the route, not the protocol

[billstewart]

The reason the VPN connection went fast isn't that EEEVILLL Verizon was throttling the customer's Netflix connections by doing deep packet routing and didn't do that to the VPN. The pipes Netflix bought to deliver movies to their paying customers who use Verizon weren't big enough to carry all the demand, at least at the peering* point that customer's traffic went through, while the pipes they bought or peering they got for free were big enough to reach the VPN endpoint, and the VPN endpoint had bought enough bandwidth from their ISPs to get from there to their peering point with Verizon, so there was enough bandwidth on the whole route to carry the movie that way.

That's not to say that there aren't ISPs harassing particular content (there was at least one well-publicized case a few years ago of some telco ISP blocking VOIP, and of course most of the cable modem and some DSL providers block home web servers), but this ain't one of them.

(*Peering unfortunately means two different things here - it's giving each other service for free, and it's having BGP-managed interconnections, usually at the big internet exchange locations, to pass traffic, not necessarily for free.)

Faster VPN Access

[Agripa]

Back when I was in the process of switching providers, I bought a subscription to a VPN service so I could have a secure connection and routable IP through public internet access points. Later one of the things I noticed with my new AT&T U-Verse service was that *all* access was faster in either latency or throughput using the VPN to tunnel through U-Verse to half way across the US including things like DNS. Some things were a little bit faster and some things were an order of magnitude faster.