Enraged Verizon FiOS Customer Seemingly Demonstrates Netflix Throttling

MojoKid (1002251) writes The ongoing battle between Netflix and ISPs that can't seem to handle the streaming video service's traffic, boiled over to an infuriating level for Colin Nederkoon, a startup CEO who resides in New York City. Rather than accept excuses and finger pointing from either side, Nederkoon did a little investigating into why he was receiving such slow Netflix streams on his Verizon FiOS connection. What he discovered is that there appears to be a clear culprit. Nederkoon pays for Internet service that promises 75Mbps downstream and 35Mbps upstream through his FiOS connection. However, his Netflix video streams were limping along at just 375kbps (0.375mbps), equivalent to 0.5 percent of the speed he's paying for. On a hunch, he decided to connect to a VPN service, which in theory should actually make things slower since it's adding extra hops. Speeds didn't get slower, they got much faster. After connecting to VyprVPN, his Netflix connection suddenly jumped to 3000kbps, the fastest the streaming service allows and around 10 times faster than when connecting directly with Verizon. Verizon may have a different explanation as to why Nederkoon's Netflix streams suddenly sped up, but in the meantime, it would appear that throttling shenanigans are taking place. It seems that by using a VPN, Verizon simply doesn't know which packets to throttle, hence the gross disparity in speed.

Comments

Thanks

[Anonymous Coward]

Now they'll just throttle VPN traffic too.

Re:Thanks

[Zero__Kelvin]

Someone should invent SSL!

Re:Thanks

[Casualposter]

SO when you pay for that service it says something like "up to 75mbps" which in reality means that the speed test and google's home page could see that much speed and everyone else will look like dial up from the 1990's.

It would be much better if the services had to advertise their average speed across the most popular sites. That way if they throttle Netflix to .375mpbs, they have to inform customers that while they are paying $125/month for "blazing fast speed" they are actually getting blazingly fast dial up speeds.

Re:Thanks

[Anonymous Coward]

Now they'll just throttle VPN traffic too.

That takes this whole thing out of context. Automatically throttling VPN Traffic would risk breaking peering arrangements Verizon has made.

Re:Thanks

[mysidia]

SO when you pay for that service it says something like "up to 75mbps" which in reality means that the speed test and google's home page could see that much speed and everyone else will look like dial up from the 1990's.

I have a suggestion.... Web browsers should take some measurements and display prominently in a visible status bar or other location.... average TCP throughput --- And Estimated average bandwidth;

Both a "this site" value, a "this browser session" value, and (Optionally) if the user decides to share their numbers, Community average bandwidth for this site, Community average bandwidth for this ISP, and Community average for this site on this ISP.

If Community average for this site on this ISP is more than a standard deviation below Community average for this site,

Then a little warning exclamation point should appear to the right of the browser bar. On mouseover, and for a few seconds after loading the page, a little warning bubble should appear for a few seconds. "Your internet service provider seems to have below average performance in loading this page."

Re: Thanks

[Anonymous Coward]

What we need is a TOR like VPN, with a single hop, which can dynamically reroute based on best throughput. The customers ISP would not know what packets to filter and the network would self optimize giving all users the best performance possible using a tcp protocol that can dynamically shift under ligidimate conjestion conditions.
 

Re:Thanks

[Anonymous Coward]

While such numbers might result in some net gain, it will probably end up pissing off ISPs to the point of either finding ways of faking the data, blocking the data, or just as policy telling customers to ignore the speed numbers. I still remember the stints I had doing help desk with people calling in for "MSN is down, but rest of the internet works, fix it fix it fix it," and "joesbasementserver.com is loading really slow, something must be wrong with my computer." A lot of people won't be able to distinguish when something is their ISP's fault and when it might be the end servers fault. In some cases, such numbers will make it clear, although in other cases, even when the numbers make it clear, people will use them to argue the opposite fault.

Re:Thanks

[sumdumass]

Wrong..

It is fraid for any ISP to purposely limit speed and bandwidth below the advertised speed speed while the network can handle it (congestion). No matter how you look at it, the up to speeds will never be availible when they purposely limit it.

This should be dealt with by consumer protection law (paying for services not delivered and possibly bait and switch or a host of others). People need to complain in those terms to thier state utilities commision or consumer protection department and file lawsuits over the said laws. Verizon and other ISPs will stop doing it and possibly be fined in the process.

Re:Thanks

[Anonymous Coward]

Web browsers should offer more visible metrics but sadly this is not the trend. The 3 big browsers developers seems to think that what we need is browsers that hide all valuable information.

Re:Thanks

[xeoron]

Then switch to SSH instead of VPN and tell your browser to access the tunnel via a socks proxy.

Re:Thanks

[mysidia]

It will probably end up pissing off ISPs to the point of either finding ways of faking the data, blocking the data, or just as policy telling customers to ignore the speed numbers.

If the data is blocked, the browser should figure out why and explain to the user that there seems to be an issue with their network; in other words "Blocking" should make it even worse for the ISP. a smarter browser UI could be a tremendous help to support technicians, which the ISPs should absolutely love ---- perhaps even tell the user exactly which entity to contact, even display their ISP's support number on the screen, to help accelerate the problem resolution process, and providing access to comments by other users of the same ISP, leading to happier customers, and customers who can share info with each other pertinent to troubleshooting or why this is happening, etc.

A lot of people won't be able to distinguish when something is their ISP's fault and when it might be the end servers fault.

I am suggesting the browser should also take some responsibility to the interpretation of the results here. There should be a highly visible "troubleshooting" button that causes some tests to be run. Explanations should be right there in a natural language that any English speaker could understand.

The browser should not show an alert if there is not enough data to make a conclusion with a fair measure of statistical confidence.

We can definitely make a strong distinguishment between a "web site performance issue" and a client connectivity issue, with data from a sufficient number of users.

The browser would also need to take into account geographic location and client connectivity, however.

e.g. Is the site slow because the visitor is half way around the world from the nearest mirror, or is it slow because they're connecting over congested WiFi or 3G networks, instead of a wired connection?

I realize it's not "easy", but the web browser is the only software component that is in a position to take the kinds of measurements that are required and help alert the user to the problem, tell the user which entity they should contact, and assist with troubleshooting.

Re:Thanks

[pedestrian crossing]

I wonder how hard it would be to do a browser plug-in for that...

Re:Thanks

[DarkOx]

Right because there is no way a little traffic analysis can't tell the difference between some typical GET and POST request sent on an SSL channel and video stream. /sarcasm off

It might be slightly harder to tell the difference between a video stream and a large file download but by no means impossible. You can do this much with (relatively) inexpensive COTS routing and fire-walling equipment now.

Nope just SSL or just tunneled ipsec for that matter won't cut it; you going to have to put some traffic analysis thwarting measures in there too, which means even more waste than the tunnel overhead. So ultimately VZ is making things worse for themselves in terms of network traffic, unless they are only going to allow busty stuff like webpage downloads.

Still even the typical slashdot'ers home setup is very much on the losing end of the arms race as things stand today. The only reason VPN success is VZ isn't really trying that hard.

Re:Thanks

[davester666]

because you generally are getting that rated speed....to Verizon's Internet or Comcast's Internet or whatever.

what they never mention is that most web sites don't pay extra to be part of Verizon's Internet or Comcast's Internet, so they get throttled by poor interconnects.

Re:Thanks

[HiThere]

There actually are LOTS of components in a position to do that. You could even run a network monitoring app. But the browser is one highly visible one that most people already have installed.

Re:Thanks

[HiThere]

One that keeps working when they change versions?

Re:Thanks

[mysidia]

You could even run a network monitoring app. But the browser is one highly visible one that most people already have installed.

Perhaps you could, but now essentially you are having "users that think they have problems" downloading an extra application and they start monitoring after there's a problem most likely.

This means your app cannot get the right data on what's normal for the user or for the world, because you have a sample of app users that are biased towards users that already are experiencing network issues of some sort, and you don't have a good baseline for the user that installed it either.

Re:Thanks

[Kernel Kurtz]

Or just bookmark Glasnost http://broadband.mpi-sws.org/t... and test regularly.

Re:Thanks

[phorm]

A certain internet company in eastern Canada appears to have already done that, except back in the days of torrents. If you torrented, suddenly yours speeds dropped dramatically (regardless of whether the torrent was anywhere near to using your capacity).
That was annoying enough, but then when most torrent streams started getting encrypted etc, they started doing it for SSH traffic on non-standard ports.
Everything would be working just fine until I opened an SSH connection/tunnel to work, and then suddenly *all* my connections would plummet in speed.

Re:Thanks

[Thantik]

And then you're limited to TCP traffic.

Re:Thanks

[buckfeta2014]

They all did... except only Bell and Cogeco got nailed for it... I still maintain that Cogeco still does it, but I don't have the money to sue.

Re:Thanks

[CodeBuster]

It depends upon how the VPN is set up but if it's using L2TP/IPSec then every non header part of every packet will be encrypted. There will be no pattern to analyze because the encrypted packets will appear to be random data. The only things the ISP could do then would be to block the network hosting the VPN server, which is difficult because most VPN providers have lots of servers with some of them probably hosted in popular networks, or throttle all connections that cannot be decoded by the traffic shaper which would cripple SSL too. Throttling all encrypted connections is an obvious non-starter since it would ruin online e-commerce and just imagine trying to shut off all traffic to Amazon Web Services just because some VPN providers host servers with a cloud provider. That wouldn't work well either. No, the ISPs much prefer to ignore VPNs for now, since relatively few of their subscribers use them, rather than engaging in a high profile arms race with sophisticated users which would only serve to popularize the concept of VPNs and make them easier for the unwashed masses to use.

Re:Thanks

[DarkOx]

Even with a tunneled VPN there is a lot of traffic shaping they could do that probably would not interfere with most online commerce and business users activities.

Web browsing is very busty. You simply allow full bandwidth when connections start. Then you count packets per second and packet sizes, which you can do encrypted or not. Unless deliberate steps to introduce noise are taken media streams are going to be characterized by relatively fixed size packets are fixed rates. Connections for things webpage loads or form posts and Outlook syncs probably will be in most cases to short lived for your analysis phase to complete and won't be affected at all.

Bigger file transfers (assuming you don't want to throttle them too) will in most cases show more variation in packets per second as network conditions change, because in most cases they running as fast as things will allow, wherever the bottle neck might be, unlike streams which are hopefully running at a negotiated rate that is fully sustainable.

Re:Thanks

[david672orford]

While such numbers might result in some net gain, it will probably end up pissing off ISPs to the point of either finding ways of faking the data, blocking the data, or just as policy telling customers to ignore the speed numbers.

I don't know what data you think they could fake or block. Mysidia is proposing that the web browser measure the speed at which the content is delivered. If they block that data, the page won't open. And they can't fake delivering the page more quickly, can they?

The real problem with this proposal is that such measurements would show that there is a bottleneck but would not show where it was. People would blame their ISPs even if another ISP were at fault or the server's Internet connection wasn't fast enough. The ISPs would indeed tell their customers to ignore these numbers, tand they would be right.

Re:Thanks

[sociocapitalist]

SO when you pay for that service it says something like "up to 75mbps" which in reality means that the speed test and google's home page could see that much speed and everyone else will look like dial up from the 1990's.

I have a suggestion.... Web browsers should take some measurements and display prominently in a visible status bar or other location.... average TCP throughput --- And Estimated average bandwidth;

Both a "this site" value, a "this browser session" value, and (Optionally) if the user decides to share their numbers,
Community average bandwidth for this site, Community average bandwidth for this ISP, and Community average for this site on this ISP.

If Community average for this site on this ISP is more than a standard deviation below Community average for this site,

Then a little warning exclamation point should appear to the right of the browser bar.
On mouseover, and for a few seconds after loading the page, a little warning bubble should appear for a few seconds.
"Your internet service provider seems to have below average performance in loading this page."

So tcp 80 won't be throttled but whatever netflix (or whatever) uses will be -

Re:Thanks

[PlusFiveTroll]

>And they can't fake delivering the page more quickly, can they?

Well, possibly.

On non-SSL sites they can proxy content, which in general is a good thing. Trouble comes if they start gaming the system and not showing the freshest content to give a perceived gain in speed.

Re:Thanks

[PlusFiveTroll]

Netflix uses http and https, mostly off of Amazon cloud services.

Re:Thanks

[sociocapitalist]

Netflix uses http and https, mostly off of Amazon cloud services.

Okay - does that change what I said in any way?

Re:Thanks

[Dusty101]

Web browsing is very busty.

Um... I think that very much depends on which particular pr0n websites the user chooses to frequent...

Re: Thanks

[MrResistor]

That's not faking delivery though. The data is actually being delivered, just from a different source. I don't give a shit where the data comes from our how it gets to me*, I care about how fast my page loads.

* except for the handful of times per month I do something sensitive, like online banking, but that's all encrypted.

Re:Thanks

[BitZtream]

http is tcp port 80, so what you said is:

'so tcp port 80 won't be throttled by tcp port 80 (or whatever) uses will be-'

Re:Thanks

[herbierobinson]

TCP performance on the Internet is almost totally limited by latency (AKA RTT or round trip time for the ACKs), not the bandwidth.

Streaming data isn't limited by the latency -- totally different beast.

Re:Thanks

[Rakarra]

On non-SSL sites they can proxy content, which in general is a good thing. Trouble comes if they start gaming the system and not showing the freshest content to give a perceived gain in speed.

That's also the strategy that Netflix offered to Comcast, to let Comcast proxy Netflix data. Comcast refused, because they want the Netflix problem. It's to their benefit that the "problem" exists, and they don't want a clever technological solution to the issue as Netflix competes with Comcast's own services. They just want Netflix to pay them cash instead.

Re:Thanks

[sociocapitalist]

http is tcp port 80, so what you said is:

'so tcp port 80 won't be throttled by tcp port 80 (or whatever) uses will be-'

Um...yes and? They're the same thing - so?

Re:Thanks

[RockDoctor]

their average speed across the most popular sites.

And exactly how are you going to define that?

Re:Thanks

[Fjandr]

It was just pointing out that your statement was the equivalent of "So tcp 80 won't be throttled but tcp 80 uses will be -"

Re:Thanks

[mysidia]

TCP performance on the Internet is almost totally limited by latency (AKA RTT or round trip time for the ACKs), not the bandwidth.

Modern TCP stacks, including Windows 7, 8 and Linux these days have a feature called TCP Timestamping, where an RTT estimate is taken for the connection, and a feature called TCP Autotuning where the window size is automatically scaled up to fill a Long fat pipe.

So no... the days where TCP throughput of a session was totally limited by latency are long gone.

Re:Thanks

[niftymitch]

Right because there is no way a little traffic analysis can't tell the difference between some typical GET and POST request sent on an SSL channel and video stream. /sarcasm off

It might be slightly harder to tell the difference between a video stream and a large file download but by no means impossible. .......

Netflix may not be the only victim and Verison may not be the only service playing games.

I noticed that it took MANY retries to download the new Beta from Apple.
My ISP is not Verision. It is that fickled one that was at the beginning
of the alphabet and now wants to be at the end perhaps because X is searched
for in all the STEM math questions (or not).

It may prove very obvious to Apple which ISPs are good guys if they look at their download logs.
The Apple download in this case apparently cannot continue after an interruption. I suspect
partly because the download requires a special token to validate the download.

But it does make the point that an OS download is big enough to trigger ISP tom foolery.
AND the Apple logs would let someone schooled in the art build a map of interesting
ISP trouble makers world wide. I think Netflix should file a legal action to get them.

Re:Thanks

[sociocapitalist]

It was just pointing out that your statement was the equivalent of "So tcp 80 won't be throttled but tcp 80 uses will be -"

Okay - I see...so in that case NBAR (or the equivalent) would be needed but it's still doable.

Thanks for pointing out that it's just port 80 though. I'm in Europe and I haven't bothered setting up a VPN to get this service so I have no direct experience with netflix at all.

Re:Thanks

[herbierobinson]

Large window sizes only help when there is absolutely no packet loss (which is not the Internet) and even then it takes a long time to scale up. Even when there is a large window size, the slow start mechanism is really really slow when the RTT is in the 100 msec range (the Internet). If you have regular low level packet loss, the actual throughput plot looks like a sawtooth wave and the upward ramp is very slow. The congestion window rarely even gets close the max allowed by the window size in the TCP header. It can take several minutes for the congestion window to ramp up with 100msec latency.

SACK might help a little bit if implemented properly, but even SACK is still required to do the slow start congestion avoidance.

There are RFCs (I haven't had time to read, yet) where routers get involved with congestion control. But the RFCs haven't become standard, yet, and the chances that every router in an Internet path supports them is probably negligible.

Re:Thanks

[mysidia]

Large window sizes only help when there is absolutely no packet loss (which is not the Internet) and even then it takes a long time to scale up.

When packet loss is significant, it will indicate congestion or a network issue and be a more important factor than latency.

Large windows still help. Packet losses are usually less than 0.01%. Anything above approximately .1% is a strong indication of a possible issue.

Often websites are bypassing slow start by starting with an expanded initial congestion window.

Re:Thanks

[herbierobinson]

Large window sizes only help when there is absolutely no packet loss (which is not the Internet)
and even then it takes a long time to scale up.

When packet loss is significant, it will indicate congestion or a network issue and be a more important factor than latency.

Packet loss and latency are both significant. The recovery time from a single packet loss is proportional to the latency.

Large windows still help. Packet losses are usually less than 0.01%. Anything above approximately .1% is a strong indication of a possible issue.

Large windows should still help; although, there might be some rare situations where they make performance worse (if the network they are on is saturated and the extra traffic pushes it past a tipping point so packet loss increases).

When one is dealing with large windows, even 0.01% packet loss hurts. This is because it takes longer to get back up to full speed after a packet loss. It will still be faster than a smaller window size, but the packet loss will eat away more overall throughput than it would with a smaller window size.

Often websites are bypassing slow start by starting with an expanded initial congestion window.

Yes, if they are following the RFCs, they can use up to 4 packets (assuming the MTU is small) for the initial congestion window. That cuts two RTTs off the ramp up time (very roughly 10%). The last time I checked, web sites were usually transferring less than 100K bytes -- The use of 4 packets for the initial congestion window should actually help them a lot more than having a large window size (that they will never actually ramp up to). Where the large window sizes really help are large uni-directional transfers (ftp, sftp, etc).

Re:Thanks

[mysidia]

Yes, if they are following the RFCs, they can use up to 4 packets

There are many cases where companies are varying slightly from the RFCs, when there is an apparent benefit of doing so.

www.Google.com has been known to send an IW of 8 packets before the first acknowledgement for their homepage.

Of course the latency and packet loss together are extremely important, and any significant losses create a performance issue ---- they are also definitive signs of a poor connection. That's different from the throughput being driven by latency, however.

Could be a different route involved for the VPN

[Anonymous Coward]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

Re:Could be a different route involved for the VPN

[knightghost]

Not so much of a leap since Verizon and Comcast have admitted to such.

Wouldn't a simple tracert show the route (and any differences)?

Re:Could be a different route involved for the VPN

[pla]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

Failing to have peerage agreements in place to honor your downstream sales commitments is a form of throttling - Or, I would daresay, a form of outright fraud.

If I offer to sell you "unlimited" beers from my fridge for $50 a month, but I only resupply it at a rate of one six-pack per week, I have intentionally cheated you. That basic relationship doesn't magically change because of some hand-waving technobabble about peerage agreements and network congestion.

(Yes, I know those don't strictly count as technobabble, and what they really mean - But they effectively reduce to Verizon having zero interest in upgrading its infrastructure to support its commitments to their customers as long as the FCC and FTC will allow them to outright lie)

Re:Could be a different route involved for the VPN

[mrzaph0d]

If I offer to sell you "unlimited" beers from my fridge for $50 a month,

uh. that's not a car analogy. where's my freakin' car analogy?!

Re:Could be a different route involved for the VPN

[Anonymous Psychopath]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

This is exactly what's happening. I do the same thing for a specific server I use. Standard routing via FiOS results in consistent 1mb download speeds. I set up a GRE tunnel to my VPS host and I get consistent 10mb download speeds. The culprit appears to be a shitty peering connection between so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.151.246) and 0.ae2.XL3.CHI13.ALTER.NET (140.222.225.187).

Re:Could be a different route involved for the VPN

[guises]

I... offer to rent you a car with a premium agreement for unlimited mileage and free gas at participating gas stations*.

* Gas allotment limited to one gallon per week.

Re:Could be a different route involved for the VPN

[Wdomburg]

I would think the more apt analogy is that you sold me unlimited access to your fridge (bandwidth) but Netflix (content provider) is only restocking at a rate of one six-pack per week. IOW, Netflix is the one failing to have peerage agreements in place to honor their downstream sales commitments.

Re:Could be a different route involved for the VPN

[Drakonblayde]

You do not understand how BGP works.

The problem isn't the data that verizon is sending to netflix, it's the data netflix is sending to verizon. Verizon messing with routing policy on netflix's announced prefix's wouldn't have an effect on verizon's streaming speeds. The traffic flows from Netflix to Verizon.

Therefore, in order to influence streaming speeds, Verizon would have to change their routing policy on how they announce their own routes in order to influence which links netflix traffic can come in on. The problem is, there's no way within BGP for Verizon to say I want Netflix traffic to only come in over these specific links. It would influence *all* traffic from that peer. Routing policy is destination based, not source based, and not source-destination based. By simply announcing the routes are more preferable over Level 3 saturated links, that forces traffic Level3 delivers for those prefixes to come in over those links.

Sure, Level3 could do some traffic engineering of their own and ignore or mutate some parts of Verizon's route announcements, and force that traffic in over unsaturated links Verizon may have with Level 3 (if there are any), but Level3 is a middleman. Doing so would take them out of their middleman status and put them firmly on Netflix's side. Verizon's likely response would be to immediately de-peer Level3.

The only folks who can effectively change how the traffic reaches Verizon's network is Netflix. They determine their outbound routing policy, but only up until their own border. Once it transits to another AS, it will be forwarded according to the upstream AS's routing policy. If Netflix wants to avoid saturated Verizon-L3 interconnects, the only thing they can do is not send traffic to Level3 for Verizon prefixes. They could easily modify their inbound route policy to send traffic for Verizon's prefixes via another peer. This is something that Level3 does not want, because it effects their revenue, hence their seeming to take sides with Netflix on the matter. It's one thing for Level3 to have an opinion on what Verizon is doing, that doesn't really effect operations. The second you change operations to try and force that opinion, well, you're likely to invoke the Law of Unintended Consequences.

Now, for whatever reason, Netflix has decided to go ahead and keep sending Verizon traffic to Level3. The reality is that if Verizon has decided to be douchebags about this, then they can do the same thing for whatever peer the traffic is ingressing through. Maybe all of Netflix's other peers ultimately transit to Verizon via Level3 anyway, which would make any change of forwarding policy moot.

About the only way for Netflix to solve this is to go ahead and cut out the middleman and just pay Verizon directly for interconnects into their network. This is what Verizon wants: another revenue source for traffic their going to deliver anyway. This is what Level 3 does not want: When you cut out the middleman, the middleman makes no money.

Netflix has already done it with Comcast and AT&T, so it's not surprising that Verizon wants in on this action as well, and will continue to be douchebags about it.

In the meantime, savvy customers can come up with their own solutions in order to avoid having netflix traffic destined for them coming in over saturated links. VPN and tunneling are two perfectly valid solutions.

Re:Could be a different route involved for the VPN

[Andrio]

If only magical boxes existed that could intelligently and automatically find the fastest route for traffic.

But what would we call these magical routing boxes?

Re:Could be a different route involved for the VPN

[Aqualung812]

Perhaps you could just link to the article or a particularly insightful comment you made instead of making a post that adds nothing to the conversation.

Not all of us read every comment on /.

I'm very curious as to why Netflix would degrade their own service and why Comcast and Verizon wouldn't point to this smoking gun every time they're accused of throttling.

Re:Could be a different route involved for the VPN

[140Mandak262Jamuna]

If I offer to sell you "unlimited" beers from my fridge for $50 a month, but I only resupply it at a rate of one six-pack per week, I have intentionally cheated you. That basic relationship doesn't magically change because of some hand-waving technobabble about peerage agreements and network congestion.

This analogy is a little flawed. Let me correct it. Let us say the local municipality has granted pla (258480) a local monopoly in selling beer to its residents. And you sell beer at different service level all unlimited number of trips to the fridge, but at 1 trip/hr, 1trip/6 hours, 1 trip/min, 1 trip/sec etc. And you stock it with brewed-by-your-local-sewage-company beer all the time, and stock Buds, Coors and Coronas one bottle a month. Then your analogy is complete.

What is really insidious is, pla is NOT buying any beer. All the beer companies come stock the fridge for free. Pla's only cost is keeping the beer cool. And it does not cost any more to cool a bottle of Corona than to cool a bottle of brew-from-sewer. Just because pla noticed people are drinking Corona more, pla wants Corona to pay him more money. Remember it is a monopoly. Corona has no other way of selling its beer without going through pla's fridge. Now you get the idea.

Re:Could be a different route involved for the VPN

[ArcadeMan]

I would even say, it's quite a Quantum Leap, unfortunately it's unavailable to stream on Netflix Canada. Which is funny because it's probably available on Netflix USA which I could access via VPN too.

Re:Could be a different route involved for the VPN

[mjm1231]

This doesn't change the fact that the customer paid for 75Mbps and got... a lot less.

Re:Could be a different route involved for the VPN

[alen]

there are a dozen video providers i can name who stream video with no problems. Hulu, HBO Go, Amazon, Vudu, Cinemanow, Apple itunes, ABC, History Channel, Disney, Lifetime, PBS, Fox, ESPN, NBC and others. most of this is video on demand, but they have some live video they stream as well.
only netflix is having these issues. difference is that everyone uses a CDN to host their content inside the ISP's network or close to. there are at least a half dozen CDN's that do most of the hosting
the way it has worked for 15 years since Akamai made the first CDN because people figured out a long time ago you can't send large files over the internet. the way it works is the CDN pays the ISP for hosting and bandwidth and the customer pays the CDN
netflix used to use one of these CDN's, forgot exactly who but they let the contract expire last year
instead netflix came up with a cool CDN box but they demand ISP's host it for free and give them free bandwidth as well. i guess netflix got so big they think they can extract payment from everyone

and if there is a lawsuit it will be brought up that this is only a netflix issue and there would be lots of discovery on netflix's business for the last 5 years

Re:Could be a different route involved for the VPN

[Anonymous Coward]

A CDN is just one way of delivering the data. If Netflix has the outbound bandwidth and is willing to peer settlement free, then what difference does it make to the ISP whether the data comes in through a network cable from outside the data center or from inside the data center? Oh right, they could charge Netflix for hosting the machines.

Re:Could be a different route involved for the VPN

[InvalidError]

Failing to have peerage agreements in place to honor your downstream sales commitments is a form of throttling - Or, I would daresay, a form of outright fraud.

Only problem with that is Verizon has TONS of under-used transit capacity with other networks - when Verizon posted their thing about peering points with Netflix's partners, they also mentioned that their transit to other networks at times where Netflix was hitting 100% was only ~40% on average.

So, Verizon would have plenty of transit capacity if it was spread more evenly across all the peering Verizon has.

Re:Could be a different route involved for the VPN

[itsenrique]

"i guess netflix got so big they think they can extract payment from everyone" Now there's a good one. In reverse shill-logic world perhaps. For those interested in what's really happening I'll point you here ( http://www.extremetech.com/com... ). Headline: Verizon caught throttling Netflix traffic even after its pays for more bandwidth. And that is basically what they are doing, artificially restricting Netflix not going through VPN to (arguably) criminally low speeds by means of not upgrading hardware on purpose to thwart who they view as "competition. Although I'm not sure Verizon will sell you anything remotely useful for $8 a month. I quit Verizon for this reason although I never told the CS rep because they try to make it hard to quit anyway. Verizon seems to be trying to fool everyone with (what seems to most people) lots of mumbo jumbo and outright deception, I for one hope they don't continue to get away with this attempt to make there "competitors" look bad.

Re:Could be a different route involved for the VPN

[InvalidError]

The problem with the 'fastest' route is that it may not be the CHEAPEST route.

If L3 really wanted to relieve pressure on their bottlenecked links to Verizon instead of trying to turn this into a PR exercise to make Verizon cave in, they could re-route traffic through Verizon's other peers with under-loaded links but that could cost L3 more money and possibly cause peering disputes with those other peers.

Re:Could be a different route involved for the VPN

[Greyfox]

Wow alternet, there's a name I haven't heard in a while. Maybe they just never upgraded their T1 line.

Re:Could be a different route involved for the VPN

[Anonymous Coward]

The problem with the 'fastest' route is that it may not be the CHEAPEST route.

Fine. Let's tax slower routes until the fastest route is the cheapest route.

Re:Could be a different route involved for the VPN

[Blrfl]

Verizon has pointed that out. Multiple times:

http://publicpolicy.verizon.co...
http://publicpolicy.verizon.co...
http://publicpolicy.verizon.co...

Re:Could be a different route involved for the VPN

[grahamsz]

But what Verizon aren't really saying is whether the bottleneck on that path is on their side or netflix' side.

I recognize that Neflix hold the power here, they have the popular service, they deliver most of the internet's traffic (in the US at least) and when they say "jump" my ISP should say "how high?". They are the primary reason a lot of us have broadband.

Re:Could be a different route involved for the VPN

[mimino]

You are absolutely right. Sadly, no one listens to arguments like this and choses to ignore these facts in favor of a more news-worthy "Faster via slow VPN [that utilizes Verizon-XO peer link] than directly [via 40Gbit/s Verizon-Level3 peer link]" that deliberately omitted the square-bracketed parts.

Re:Could be a different route involved for the VPN

[alen]

it doesn't matter
settlement free links have always had friendly agreements about traffic ratios and the reason behind them was to exchange data on the internet without the pain in the ass of a budget. it worked ok until cogent and L3 decided to throw those agreements away to take the netflix business.

in the past it was always OK for cogent and L3 to de-peer from a partner who didn't live up to their end until they wanted some free bandwidth and charge netflix for it on the other end.

Re:Could be a different route involved for the VPN

[bigpat]

More like "Payola" than anything else. You subscibe to their service and then they are shaking down the content providers for bribes in order to give them higher speeds to you, while they are telling you that you are getting a certain speed and lying about the reasons why you are not getting content at the speeds you are paying for. Seems like the anology of radio stations taking bribes to feature certain content without informing the consumers of that relationship is most appropriate.

Re:Could be a different route involved for the VPN

[bigpat]

That is a load of horseshit!

Or let me be clear. That is a load of horeshit technobabble meant to obfuscate and mislead. Level 3 was pretty clear the other day when they offered to spend a few thousand dollars to upgrade their links to Verizon. Level 3 is a backbone Internet provider. There is no reason that any link between it and another network should remain saturated if both sides are acting in good faith to serve their respective customers, especially when L3 was willing to pay the costs to upgrade Verizon's own equipement to handle more traffic which it shouldn't have had to do because it is Verizon's customers who are requesting and already paying for the content in the first place.

Verizon is choosing to not upgrade its connections to shake down Netflix, and thus pass those costs on to Verizon customers. Period.

Re:Could be a different route involved for the VPN

[geoskd]

that link is dedicated netflix and it limits them to the amount of data they send. last year super hd was for a few selected ISP's but then netflix started sending it to everyone over Level 3 and screwed up everyone's service the point is netflix is trying to increase costs on their business partners who will then have to increase prices of their customers. customers will hate the ISP but like netflix. same strategy as TV companies have used with cable TV and forcing them to sell bundled channels, intel has done this, a lot of companies have done this. customer hates the company they do business with for high prices, but it's really because they are being forced to provide services some may not want current system is not perfect but it ensures that people who use the service pay the costs and not everyone pays

Verizon was given an "out" and they refused to take it. Netflix offered to provide co-located CDNs, and all Verizon had to provide was electricity and space (both of which are negligible compared to the cost for Verizon to pay for bandwidth.) Verizon elected not to take the option that would save them money, deciding instead to play a stupid game of chicken with Netflix. While this idiot game may work with smaller companies, Netflix is now the 800Lb gorilla, and Verizon has nothing but downside on this deal. Their best bet would be to quietly put their tail between their legs and give Netflix the Co-lo's they had been asking for. Verizon needs to understand that they do not really have any chance at becoming a significant content provider, and they should know their place. (getting my bits from place to place.) They offer the consumer specific bandwidth, which as far as the consumer is concerned includes Netflix traffic. Verizon needs to understand that in the consumers mind, the customers pay Netflix to provide the bits, and they pay Verizon to get them from Netflix to their device. Verizon can claim all they want to about Netflix not paying for this, or not paying for that, but as far as I am concerned, I am paying Verizon to deliver the bandwidth door to door. I have already payed Verizon for it, and now they are failing to deliver what I paid for.

Re:Could be a different route involved for the VPN

[Xenx]

Verizon is intentionally restricting traffic flow, degrading the quality of service for THEIR paying customers. It really doesn't matter about the rest. They're willing to make their paying customers suffer. However, Netflix and Level 3 are willing to work with Verizon to resolve the issue. I'm not saying Verizon is wrong for wanting additional compensation in the peerage agreements, but they're going above and beyond at the expense of their own customers.

Re:Could be a different route involved for the VPN

[Xenx]

As for number three.... Netflix is willing to provide their CDN hardware to ISPs for free. The ISP just has to house the hardware. I don't know enough about how all the back end works, so fair and equitable is up in the air. But, Verizon doesn't want to go that route without getting paid either.

Re:Could be a different route involved for the VPN

[NeutronCowboy]

If Comcast can't deliver on the service they promised, that's not my problem.I would just switch to someone else. Unfortunately, I have no way of actually switching to an ISP which might cost more, but can deliver the service they promise. Which means that Comcast has exactly zero incentive to come to a compromise with Netflix. They can just play hardball all the way.

The technical aspects of what Comcast is and exactly isn't doing is purely that - technical pissing around. Net neutrality is concerned about what kind of business decisions drive the technical implementations. And that's where Comcast, ATT, and all the other last-mile providers really hope that no one calls them on their bullshit and their misdirection into technical nonsense.

Re:Could be a different route involved for the VPN

[Christopher_G_Lewis]

[...] bottle of Corona than to cool a bottle of brew-from-sewer.

Um, isn't that the same thing?

Re:Could be a different route involved for the VPN

[NeutronCowboy]

You're right, Netflix will lose any lawsuit. That's exactly the problem, and why everyone is so up in arms about this. There is no legal recourse to force the last mile providers to actually provide what they're selling, there's no commercial recourse, and there's barely a technical workaround (VPN providers cannot sustain everyone using them for streaming Netflix).

You're completely missing the forest for the tree.

Re:Could be a different route involved for the VPN

[ArmoredDragon]

Yeah but you do two traceroutes. First, from outside of the VPN, you'd have to trace the route to the VPN entry point IP address. Then from inside the VPN you'd trace to your destination. When you trace inside of a VPN, the first hop is always the VPN endpoint, so you'd never see which verizon peers you're going through if you do it that way.

Re:Could be a different route involved for the VPN

[sumdumass]

Wouldn't that be netflix's ISP?

Or do ypu think it is proper to charge someone in addition to what your ISP charges for access?.That is what it seems like you are saying. You build a website, find a host, and it is your fault it is slow for your users because you didn't negotiate with all the other networks that your users might be on despite the role of a host or the users ISPs..

Sounds insane to me.

Re:Could be a different route involved for the VPN

[hawguy]

that link is dedicated netflix and it limits them to the amount of data they send. last year super hd was for a few selected ISP's but then netflix started sending it to everyone over Level 3 and screwed up everyone's service

the point is netflix is trying to increase costs on their business partners who will then have to increase prices of their customers.

Netflix isn't trying to increase costs to ISP's, they aren't forcing data to their subscribers -- it's the ISP's customers that are already paying for broadband who are demanding high quality video to feed their 1080p (and soon, 4K) big screen TV's. What reason is there to pay for a 75mbit connection if you're not planning on using large amounts of bandwidth?

If Verizon has to charge their customers more money to provide them with the network capacity they thought they were already paying for, then that's what they should do. They shouldn't try to extract money from content providers to artificially subsidize internet connections to keep costs low -- this causes a larger barrier to entry to smaller ISP competitors that don't have the leverage to extract costs from content providers.

Re:Could be a different route involved for the VPN

[gnupun]

This doesn't change the fact that the customer paid for 75Mbps and got... a lot less.

Maybe there's a conflict of interest... Netflix's movie streaming service competes with Verizon cable tv, so the latter is trying to reduce quality of service of Netflix so customers switch. Maybe ISPs should not be allowed to run cable tv/satellite tv services... they should branch off cable/satellite to a different company.

Re:Could be a different route involved for the VPN

[thaylin]

And they have never been between ISPs and the backbone providors. Verizon is trying to act like something it is not. It is an ISP, by definition it will always have a imbalance of traffic, that is its business.

Re:Could be a different route involved for the VPN

[Austerity Empowers]

I don't agree, the internet is the very definition of "all inclusive". There is a ton of stuff on the internet that I do not want, and in some cases outright fear. But I am paying for "internet access" and with that every beltsanders-and-geese-porn type content that someone may unearth from the deepest parts of a russian brothel. It's a given that there is nothing "efficient" about the internet, it is one, giant, bundle. But it's a bundle that allows us to exclude the monopolistic evil that sits between us and the rest of the world so we take it.

Verizon can choose to install a Netflix CDN, or choose to upgrade the peerage link. It's on them to figure out how manage bandwidth costs while delivering optimum service. It costs what it costs. We'd believe Verizon more if they and their ilk didn't so thoroughly block any efforts to have competition, such that prices would balance themselves out.

Re:Could be a different route involved for the VPN

[Austerity Empowers]

Never say "bandwidth provider" to a telco monopoly. It grates their teeth and raises their dental insurance. They've been staring down the barrel of this reality for 20 years and have done everything they can do, bought every politician and generally screamed bloody murder to get us to where we are. They seriously think the world wants to go back to using them as their AOL/Prodigy/Compuserve.

Re:Could be a different route involved for the VPN

[sixsixtysix]

Except Buds, Coors and Coronas ARE the sewer brews.

Re:Could be a different route involved for the VPN

[NoKaOi]

Why should Verizon spend many billions of dollars to subsidize Netflix?

They're not. Verizon's customers are paying them to provide a service. Just because a bulk of the traffic is coming from a particular source doesn't mean it's okay for them to charge their customers for a service that they're not providing. It all comes down to Verizon trying to double-dip.

Re:Could be a different route involved for the VPN

[apraetor]

Having large sections of the backbone slowly replaced by websites peering directly to ISPs also is a threat to the internet. It allows levels of douchebaggery which can't be achieved when the bulk of traffic travels via L3 and their competitors. When ISPs start peering directly, beyond the level of simple CDNs, they're effectively becoming their own paywalled "internet".. AOL much?

Re:Could be a different route involved for the VPN

[kevmeister]

That SEEMS to point directly at Verizon as ALTERnet is the Verizon backbone. It was originally the IP backbone of UUnet which was purchased by Worldcom and then picked up out of the bankruptcy by Verizon. So the bottleneck appears to be between the Verizon backbone network and the Verizon ISP network in LA.

At the time Verizon bought it, UUnet and its ALTERnet backbone dominated the Internet. It's shrunk a bit since then.

If you noticed the "SEEMS" and "appears" above, that is because of the complexity of peering and routing in the Internet. Simple traces and such to track down bottlenecks often point to the wrong place. This is made even worse by networks hiding their cores behind MPLS clouds. And, that "hiding" is not to confuse people, but allows for more robustness and faster recovery from things like fiber cuts which don't heal themselves quickly at the routing layer.

I can say with great certainty that many of the comments on this are almost certainly wrong. I especially loved the one about using unused bandwidth on other providers for Netflix to get traffic to Verizon customers. At what Verizon charges for peering, you are not going to find spare capacity just lying around!

Re:Could be a different route involved for the VPN

[tricorn]

It would actually be fairly easy to show that it isn't traffic-analysis throttling going on - set up a server somewhere that you can get a 5Mbps stream going, and that also can get 3Mbps to Netflix, then use an un-encrypted port forward. Given that Verizon and Level3 have both shown that it's a bottleneck at their interconnect point, I'd expect that method to get you a full speed Netflix stream with no problem.

Now, that wouldn't necessarily be a real solution - the route you're getting would probably also be overwhelmed by the traffic if a large number of people were all routing traffic through it. What it does show is that Verizon needs to fix the bottleneck. That's what they're being paid for by their customers. The providers Netflix is using can handle the load, and they clearly have no incentive to not build out their networks in whatever way is needed to handle it properly.

If 90% of Verizon's traffic ends up coming from Netflix, so what? That means they only need 10% of their network for everything else. Their customers are already paying to receive that data, why should Netflix pay again?

The people talking about "unbalanced data flows" are missing the point. It wouldn't make things better if Netflix changed the protocol to require that customers send them as much data as they receive. Bits aren't a resource, nor are they toxic waste, the country won't start to tilt if Netflix sends too many bits in one direction without accepting the same number in return.

If that's the way it worked, then Netflix could simply set up a Cloud backup service.

Re:Could be a different route involved for the VPN

[evilviper]

This doesn't change the fact that the customer paid for 75Mbps and got... a lot less.

1) That's not a news story. It's been happening for a long time.
2) ISPs always warn you that they aren't responsible for the speeds of 3rd party servers, and can't guarantee they'll be fast, too.
3) He paid both Verizon and Netflix. You can make the case for either one of them being at-fault.

Re:Could be a different route involved for the VPN

[evilviper]

Failing to have peerage agreements in place to honor your downstream sales commitments is a form of throttling - Or, I would daresay, a form of outright fraud.

You're one small step away from saying that Verizon should be giving free internet service to every service that has content their customers want to send/receive.

That's never been how internet service has worked, and forcing Verizon to acquies to misbehaving peers will result in a huge increase in your internet service prices, as everything suddenly becomes client-pays, instead of both sides mutually paying their half.

Re:Could be a different route involved for the VPN

[evilviper]

Netflix offered to provide co-located CDNs, and all Verizon had to provide was electricity and space (both of which are negligible compared to the cost for Verizon to pay for bandwidth.) Verizon elected not to take the option that would save them money

If Netflix got a free CDN setup without paying ISPs anything, Verizon would quickly see all the other CDNs refusing to pay them, too.

On that same note, I know where you can get a FREE 40-hour/week job... You won't have to pay a penny for this FREE job.

Re:Could be a different route involved for the VPN

[sjames]

Except Verizon accidentally admitted that it's not Netflix doing the damage.

Re:Could be a different route involved for the VPN

[evilviper]

That is a load of horeshit technobabble meant to obfuscate and mislead.

No, his explanation is spot-on. If "technobabble" means you didn't understand it, that's besides the point.

There is no reason that any link between it and another network should remain saturated if both sides are acting in good faith to serve their respective customers

Level-3 is acting in bad-faith in a couple different ways. Offering to give Verizon a small amount of money does not obviate the large ways in which they are acting badly, which will make them even more money.

it is Verizon's customers who are requesting and already paying for the content in the first place.

Customers pay both Verizon and Netflix. Both sides are supposed to pay their own costs of transit and bandwidth. Saying that Verizon should acquies to badly-behaving peers is a small step away from saying that Verizon should provide free internet services for every service their customers request.

Verizon is choosing to not upgrade its connections to shake down Netflix

Verizon is choosing not to upgrade it's peering points with Level-3 because they are no longer evenly sharing traffic up/down as all free peering arrangements have ALWAYS required, yet Level-3 doesn't want to pay for the imbalance, and Netflix doesn't want to shift some of their Verizon traffic to a different transit provider than Level-3.

Re:Could be a different route involved for the VPN

[tricorn]

It's not like it's a surprise that there's a lot of Netflix traffic. I could forgive an ISP for not having the connections in place to handle that amount of traffic if all of a sudden it sprang up, but they should be able to handle it by now.

Customers are paying for that level of service. If most of their traffic is coming from Netflix, that's because THAT'S what's driving their customers to pay more for higher speed service. That means that they're getting more money, but most of the capacity increase for their network can be concentrated on serving the Netflix traffic. That's probably less expensive than building out the capacity to handle all those high-bandwidth customers spreading it around more.

Re:Could be a different route involved for the VPN

[geoskd]

If Netflix got a free CDN setup without paying ISPs anything, Verizon would quickly see all the other CDNs refusing to pay them, too.

I don't have a problem with that. If Verizon thinks that $60 / month for Internet access in my area is not enough, they are welcome to raise their rates, but TWC doesn't seem to have any problems with providing 50Mb down for that price, and I'm going to guess that they allowed Netflix to co-lo without the double-dipping fees. Again, I re-iterate, I have paid Verizon to move the bits, they have no right to bitch about having to actually move them, and nor do they have the right to charge anyone else to move those same bits.

Re:Could be a different route involved for the VPN

[sl149q]

This is exactly what Level 3 claims.

http://blog.level3.com/global-...

The above uses the Los Angeles interconnect as an example. Four 10-GB ethernet connections between Level 3 and Verizon and they are saturated.

Level 3 wants to add additional 10-GB connections (and even offered to buy the router cards, cable and do the install :-) ). Verizon refuses.

"So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion."

It *could* be that Verizon realizes that currently, as configured, the Verizon network is running nicely (as the nice diagram shows). And that doubling the amount of traffic by doubling the interconnect to Level 3 (and thereby satisfying their customers demand for Netflix VOD) would cause a meltdown.

But in any respect, Verizon is selling a service and not providing it. Netflix is paying for transit TO Verizon and Verizon's customers are paying for transit (to Verizon) for transit from Level 3. Verizon needs to charge their customes what they need to provide the service.

Re:Could be a different route involved for the VPN

[sl149q]

I'm zero steps away from saying Verizon needs to provide adequate peering for the services THEIR customers want to use.

In this case Verizon's customers want Netflix and think they are paying an adequate fee for a high bandwidth (e.g. 75MB) connection that will support a reasonable VOD stream.

Verizon needs to provide that service. Netflix needs to pay to get the bits TO the peer point. Verizon's customers need to pay to get the bits FROM the peer point.

At some point an enterprising class action legal firm is going to crunch the numbers on behalf of Verizon's customers. It won't be Netflix they will go after.

Re:Could be a different route involved for the VPN

[sl149q]

Since (well at least for the Los Angeles peering point) the finger is pointed directly at Verizon.

Level 3 has requested, Verizon has declined, to install additional 10 GB connections to increase bandwidth that is at 100% capacity.

A very low cost solution to allow traffic that Verizon's customers have requested to transit from Level 3 to Verizon's network.

Re:Could be a different route involved for the VPN

[symbolset]

The bandwidth is already 10x what it needs to be. Netflix gives free CDN boxes to ISPs that virtually eliminate this backbone traffic. That is the right way to solve this problem technically. It is cheap and eliminates redundant backbone traffic. Google Fiber takes the boxes even though they have ample backbone transit because that is the best solution for their customer, providing higher resolutions and lower latency to the video fan.

Re:Could be a different route involved for the VPN

[NeutronCowboy]

Verizon is choosing not to upgrade it's peering points with Level-3 because they are no longer evenly sharing traffic up/down as all free peering arrangements have ALWAYS required, yet Level-3 doesn't want to pay for the imbalance, and Netflix doesn't want to shift some of their Verizon traffic to a different transit provider than Level-3.

Considering the huge imbalance in download and upload speeds, how exactly is anybody supposed to peer with Verizon? Verizon knowingly set up a situation in which it is impossible for any peer to be on traffic parity with Verizon. Furthermore, traffic parity is almost impossible from a business perspective. Verizon and the last-mile providers have consumers and creators at one end, everyone else has pretty much only creators. The only way for corps like Level 3 to achieve traffic parity is to offer last-mile services, which is impossible, because Verizon frequently has a local monopoly.

So - the technobabble refers to the fact that the technological discussion is largely irrelevant when it comes to Net Neutrality. Anyone trying to bring technological issues into the discussion is just trying to muddy the waters of what is a market power discussion.

Re:Could be a different route involved for the VPN

[evilviper]

Verizon just changed all their FIOS plans to be symmetric up/down, so they're at least less guilty than other ISPs.

In addition, Verizon isn't just customers. They host sites, too. More Redbox streaming going over Level-3 would help to level things out.

Or Level-3 could offer cut-rates to online backup service providers, who recieve a lot of traffic from customers, and only a fraction as many requests. Or Netfliix could change their player to upload junk data to some random server all-the-time, which would help tremendously.

The point remains, no-fee peering has always required roughly equivalent up/down traffic, so the horrible imbalance Netflix causes, is going to cause peering disuptes, legitimately, without any evil conspiracy from Verizon and others. And that's not even getting started on Level-3's poorly concieved CDN, taking money from ISPs while futher imbalancing their peering arrangements.

Re:Could be a different route involved for the VPN

[evilviper]

Peering agreements have never depended on who requested what data. They're much simpler up/down traffic ratios.

Everybody seems to have enough hate of Verizon, and love of Netflix, that they just want to punish the former, in favor of the later, no matter the circumstances.

Re:Could be a different route involved for the VPN

[evilviper]

So it's your assertion that Verizon should pay to run lines to the Netflix data center, and give them all the free bandwidth they can use?

Re:Could be a different route involved for the VPN

[evilviper]

In the old days, when peering became imbalanced, ISPs would shut-off peering with each other, bifurcating the internet for weeks until one side agreed to pay another. These days, they just let the peering points get congested, and don't upgrade it.

I'm sure if Level-3 would agree to pay Verizon for the peering imbalance, Verizon would upgrade the peering points, but that would cost Level-3 more in the long-term. Level-3 pointing the finger at Verizon, and shaming them in the court of public opinion, is cheaper than fixing their peering imbalance.

Re:Could be a different route involved for the VPN

[Chibi Merrow]

Netflix isn't using any of Verizon's bandwidth.

Verizon's customers are using Verizon's bandwidth.

Re:Could be a different route involved for the VPN

[bondsbw]

It seems the easiest way to get to the bottom of this is for Netflix to allow a third party to send packets from Netflix IP addresses on its backbone connections through the last-mile service providers to endpoints owned by the third party. If the packets transmit at the same or expected rates, the problem is with Netflix. If not, it's with the ISP or backbone provider.

Assuming Netflix had nothing to hide, they should be fine with allowing this. Same with Verizon.

Re:Could be a different route involved for the VPN

[grahamsz]

When have consumer ISPs ever been equal peering partners? Most of them offer asynchronous service and actively discourage the running of servers and peer to peer applications. They've always received far more data than they've sent.

Presumably netflix have a boatload of available inbound bandwidth. Maybe they could set verizon users so they streamed all the data back to them. That'd make verizon happy with their untimely claim about traffic ratios

Re:Could be a different route involved for the VPN

[Munchr]

It seems to me that it is much more accurate to talk about this "data Verizon is requesting from Netflix" rather than "data Netflix is sending Verizon". It's not like Netflix is passing data through Verizon to reach Comcast subscribers. Rather, Verizon is demanding this data from Netflix, on behalf of it's own subscribers, and intentionally limiting how much of it actually reaches their subscribers and how quickly.

Re:Could be a different route involved for the VPN

[the_B0fh]

the point is netflix is trying to increase costs on their business partners who will then have to increase prices of their customers. customers will hate the ISP but like netflix.

No. The traffic goes like this:

Business/Service Their ISP Your ISP You

It doesn't matter who the business is. The issue is the interconnect between their ISP and your ISP.

Your ISP shouldn't be able to blackmail a Business into paying them, like Comcast did. In that case, Comcast blackmailed NetFlix into peering with them. This turns the above diagram into:

NetFlix Netflix's ISP/Your ISP You

Re:Could be a different route involved for the VPN

[the_B0fh]

Are you that stupid, or do you just play one on the Internet? This issue has been debated endlessly and there are lots of nice graphs and diagrams, even here on /.

No, it is not the assertion - what Level3 has pointed out is that in the peering hostel, there is sufficient capacity but Verizon refuses to use it. That is the place where Netflix's ISP and Your ISP (Verizon) meets, to pass traffic to each other.

http://tech.slashdot.org/story...

Re:Could be a different route involved for the VPN

[Alsee]

a small step away from saying that Verizon should provide free internet services for every service their customers request.

Screw "a small step away".
Verizon should provide free internet services for every service their customers request.

The customer is paying for internet service, and the ISP goddamn well needs to round-trip delivery of the customer's internet data, up to the quantity and speed THAT THE CUSTOMER PAYED FOR.

The truly insane thing here is that Level3 has gone to the absurd length of offering to pay 100% of the cost GIFTING Verizon with the additional network cards and cables to expand the link and fix the problem. Verizon refused. Verizon isn't happy being a network provider - they see the revenue Netflix and others gets being a content providers, and Verizon doesn't want the connection problem fixed for free.... Verizon wants to extort Netflix to give them a permanent revenue stream from the content pie. Verizon is abusing their monopoly power to bottleneck customer's data.... trying to force Netflix to raise prices and pay that extra money as a KICKBACK to Verizon. Verizon is abusing their monopoly position to try to gouge their own customers - and trying to force Verizon's price-gouging to show up on customer's Netflix bills rather than appearing on Verizon's own bills.

-

Re:Could be a different route involved for the VPN

[bigpat]

So, Verizon would have plenty of transit capacity if it was spread more evenly across all the peering Verizon has.

Transit capacity is irrelevant. L3 is a backbone Internet provider. They have plenty of bandwidth to get all the packets to Verizon's networks in the areas Verizon serves. Verizon is just unwilling to provide their own Verizon customers with the bandwidth they require to access the content they want.

Re:Could be a different route involved for the VPN

[thule]

Thanks for this! Both you and the previous poster explaining BGP. So many people have misconceptions on how the Internet works. Then there is the added complexity of business.

I really proves nothing that Netflix over a VPN is faster than without a VPN. We already know Verizon-Level3 peering is saturated. Both sides have admitted it. It comes down to how to solve the problem. It is not a technical problem. It is a business problem

So what if Level3 offers to pay for the upgraded link. If the existing agreement is settlement-free upgrading the link will likely push the traffic exchange outside the agreement. So if Level3 starts sending more traffic than it received from Verizon, then they should pay Verizon for transit of that traffic. Verizon has probably told them that. Level3 comes back and says, "But we'll pay for the upgraded equipment." Verizon says, "So what? If the traffic isn't equal, then you pay." And on and on it goes. So, as stated above, the best thing to do is for Netflix to create peering connections with Verizon that have no expectation of equal traffic. They will have to pay Verizon for these connections.

This is NOTHING new people. This is how the Internet has always worked.

Re:Could be a different route involved for the VPN

[bigpat]

traffic parity with Verizon

Traffic "parity" is only relevant when you are talking about backbone Internet providers providing alternative routes in order to help make the Internet routing more robust. Basically an I'll scratch your back and you scratch my back mutually beneficial scenario where having two equal backbone Internet partners is more robust than having just one route over a long haul network. So for instance having a peering agreement with a company that has a wire from LA to Boston so that if your LA to Boston wire goes down, then your traffic can still get through.

But when you are talking about last-mile customers who are the ones initiating the requests for content, then peering is not about traffic parity, it is about providing your customers with the bandwidth to the content they want. As long as L3 is willing to provide adequate bandwidth connections to Verizon's networks in the local metropolitan areas, then they are fulfilling their end of the bargain as a backbone provider peering with a local ISP.

Sure, if L3 was just saying they were going to dump all the Internet Traffic destined for the East Coast in LA and Verizon could deal with getting it across the country, then that wouldn't fly. But as far as I know L3 is ready willing and able to send all the traffic across the country and put it as geographically close to the Verizon customers who are requesting the content as Verizon will allow. But it is Verizon simply saying they won't allow L3 to increase the bandwidth to Verizon Customers until Verizon gets a bigger cut of the action.

Re:Could be a different route involved for the VPN

[evilviper]

Stupid? I'm the only one here who seems to know what peering agreements are, and how they've worked for the past several decades.

There's no question Verizon has plenty of bandwidth. The problem here is Level-3 breaking their peering agreement, and not wanting to renegotiate, so Verizon has ever right to disconnect Level-3 and Netflix from their customers. Instead, they let the peering point get congested, until a new agreement is worked out.

It's how peering has always worked. You're the one arguing we need to erase the history of the internet, and turn it into a receiver-pays model, where every site you visit gets a few cents from Verizon.

Re:Could be a different route involved for the VPN

[evilviper]

I'm trying, but this isn't the /. I remember. We've been overrun with AOLers, so of course my comment (with all those pesky little facts that get in the way of the anti-ISP ranting) gets modded-down.

I only hope SoylentNews turns out better...

Re:Could be a different route involved for the VPN

[geoskd]

Stupid? I'm the only one here who seems to know what peering agreements are, and how they've worked for the past several decades. There's no question Verizon has plenty of bandwidth. The problem here is Level-3 breaking their peering agreement, and not wanting to renegotiate, so Verizon has ever right to disconnect Level-3 and Netflix from their customers. Instead, they let the peering point get congested, until a new agreement is worked out. It's how peering has always worked. You're the one arguing we need to erase the history of the internet, and turn it into a receiver-pays model, where every site you visit gets a few cents from Verizon.

The peering model is flawed. There are effectively four different entities in any given transaction: ISP1 and Customer1 (aka consumer and ISP). ISP2 and customer 2 (aka service company). Eash customer pays their own ISP to get the bits to or from them to the border between ISPs. These customers have paid their ISP to do this regardless of the direction of the bits. I pay TWC and Verizon to move bits to and from my machines. Google and Netflix pay their providers to do the same. Companies like Level 3 should get paid to move bits from one ISP to another when the ISPs cannot reasonably connect directly to each other. They are paid by the ISPs to move bits from the ISP to the other ISP. As such, the ISPs should never be getting paid by anyone other than the consumer, and the tier 3 providers should not get paid by anyone other than the ISPs. In this regard, direct peering saves both ISPs money in direct proportion to the amount of their own customers traffic. In that regard, peering is always equitable and should be a cash neutral arrangement for everyone involved.

The only time the concept gets more complicated is if there needs to be a 5th party involved in moving bits from one ISP to another, and these parties should be paid by all ISPs, as they have no direct customers of their own. Where it starts getting stupid is when a customer has an arrangement directly with one of these 5th parties. Now everyone seems to think they are owed money by everyone else, when in reality, once the 5th party starts providing internet service directly to the customer, they effectively become either ISP1 or ISP2, and all payments to or from other ISPs should stop. Allowing any other arrangement is idiotic, and will lead only to the kinds of dimwitted inbred redneck fighting we have now. If a tier 3 provider really wants to go into the ISP business, then they should keep that business as a separate entity at arms length. otherwise ISP1 would be fully justified in claiming that it the 5th entity is really ISP2, and as such demanding that the arrangement is a peering one, and not a transit arrangement.

In the end these stupid fights hurt only the consumers and the Internet in general. A tier 3 provider can usually be bypassed pretty automatically, but for end consumers bypassing their own ISP is far more difficult, as many have defacto monopolies in large areas of the country. In the end, Netflix is trying to alleviate the problem by providing content delivery from the ISPs own local blockhouse, thus saving everyone the bandwidth, but when an idiot like Verizon thinks that they are somehow entitled to be paid for allowing someone to save them money, they need to get smacked around.

Re:Could be a different route involved for the VPN

[quetwo]

But you are under the grand assumption that Level-3 is going to charge Verizon more because of the imbalance. Something they have NOT said. In fact, they have a blog post on their site (quoted multiple times in this thread already) they will NOT be charging Verizon more, and have offered to pay for the equipment to do it.

In the old-world, peering arrangements between backbone providers and ISPs were all about symmetrical data. That day is LONG gone. Now it is all about bandwidth and that is it. It's well known that the last-mile providers are heavy downloaders and that large hosting companies are large uploaders. It's just the nature of the internet as we know it. Peering arrangements between large backbone providers (which is the knowledge you are referring to) are generally seen as symmetrical.

Re:Could be a different route involved for the VPN

[quetwo]

You are already late to the game. Tons of ISPs have direct contracts with companies for direct peering. Things like ESPN360 and the like have existed for years and are ISP specific.

What we are seeing Netflix do here is smart for both them and the ISPs that are offering to host them. They send one data stream to a colo box that the ISP is hosting, and the the massive bandwidth that it generates is moved from their expensive pipes to the "free" pipes they already own. They aren't offering any services that aren't available to the rest of the world (in fact Netflix is doing this all over the place -- ISPs, Universities, etc, and building their own CDN).

Re:Could be a different route involved for the VPN

[sociocapitalist]

It is also possible the the VPN packets are transiting a different upstream peer from Verizon and bypassing the peering bottleneck at issue. Assuming that Verizon is performing inspection of packets and throttling only Netflix packets is quite a leap.

My understanding is that Netflix peers directly with Verizon so unless the VPN was to a Netflix IP across the same interco between Verizon and Netflix then the traffic would have to take a longer path to get to Netflix to start with.

For the same reason, Verizon may have rate limiting / shaping / whatever to police traffic down to whatever value they want on the direct peering links which wouldn't be visible to anyone outside Verizon.

Even if traffic were to take the same path it would be easy to rate limit / police / shape / whatever based on protocol ports (or NBAR or the equivalent).

Re:Could be a different route involved for the VPN

[quetwo]

Dude. You are still talking about backbone-to-backbone peering arrangements. This hasn't been the case for ISPs in probably 10 years. Just stop it.

Re:Could be a different route involved for the VPN

[dissy]

No, his explanation is spot-on. If "technobabble" means you didn't understand it, that's besides the point.

Perhaps you can explain better, as your post still doesn't clear that bit up.

How does traffic generated within verisonz ASN, and exists within the same verizon ASN, even need BGP to function?

Start there at basics, and once you explain how internal traffic that never once touches a peer point still relies on this BGP "magic", then you can go into details about BGP...

Re:Could be a different route involved for the VPN

[geoskd]

So it's your assertion that Verizon should pay to run lines to the Netflix data center, and give them all the free bandwidth they can use?

No, that would make them Netflix's ISP as well, in which case I would have no issue with them charging Netflix whatever they normally charge commercial customers.

What I have an issue with is Verizon charging Netflix' ISP (or Netflix directly) for moving bits which effectively now belong to me. If they can get Netflix to agree to it great, but if Netflix refuses, and Verizon starts limiting the transmission of those bits through actions within their control, then they are going to answer to me, because *I* am their customer. I am considering dropping their service as a result of this squabble. Not because I use Netflix (I really don't), but because I have to ask the question: How long until they get around to playing this dumb game with a service that I DO use. If they pulled this crap with Google, Skype, Crashplan, or any of the other online services I use, It would take the duration of a phone call to TWC to dump Verizon.

My issue with this whole thing, is what happens to the subscribers who live where there is no alternative? What do they do? The FCC is supposed to look out for those folks, but they have been completely asleep at the switch...

Re:Could be a different route involved for the VPN

[cthulhu11]

This is the obvious answer, wish I had mod points for you. So many people out there just don't understand how the net works, at even a basic level.

Re:Could be a different route involved for the VPN

[cthulhu11]

Verizon is choosing not to upgrade it's peering points with Level-3 because they are no longer evenly sharing traffic up/down as all free peering arrangements have ALWAYS required, yet Level-3 doesn't want to pay for the imbalance, and Netflix doesn't want to shift some of their Verizon traffic to a different transit provider than Level-3.

That's been clear to me for some time, but so few seem to get it.

Considering the huge imbalance in download and upload speeds, how exactly is anybody supposed to peer with Verizon?

Not everyone offers such imbalance traffic.

Verizon knowingly set up a situation in which it is impossible for any peer to be on traffic parity with Verizon. Furthermore, traffic parity is almost impossible from a business perspective. Verizon and the last-mile providers have consumers and creators at one end, everyone else has pretty much only creators. The only way for corps like Level 3 to achieve traffic parity is to offer last-mile services,

No, they could pay Verizon for transit, as is the customary practice, rather than expecting Verizon to give them service for free, because you know all the other links in Verizon's networks cost $0/month and Juniper/Cisco give them M20's and 9922's for free.

Re:Could be a different route involved for the VPN

[InvalidError]

Verizon's subscribers would be able to get the content they want if Netflix routed traffic to Verizon through other peers than L3.

Verizon upgrading their connectivity with L3 to infinity and beyond would not be good business practice since Verizon would be screwed the second Netflix decides to change their transit mix to move away from L3 and then Verizon would have to start over.

It makes sense that Verizon would want to force Netflix to diversify its peering.

Re:Could be a different route involved for the VPN

[evilviper]

Verizon providing free access to Netflix is the logical conclusion of your attitude. You don't care about the technical details, and just want everything handed to you. That gives upstream services the upper hand to demand absolutely anything and everything from Verizon, including making them build-out their network as cheaper ISPs put more burden on Verizon, or even making Verizon pay for access, lest they lose customers.

Re:Could be a different route involved for the VPN

[Zaelath]

Would it be easier for you if you just changed the carrier from Interwebs to USPS?

Netflix sell you unlimited CD/s month, as long as you return one they'll send out another (there's some supply and sorting issues I believe but that's not the point)

USPS deliver what they're given as fast as they can, so generally you get something like a 3 day turn around on CDs.

Now USPS says "You know, you Netflix people are making more money than we thought you would, we'd like a piece of that. We can offer you a priority 3 day turn around your CDs for a small consideration, otherwise we might just experience some packet loss in the sorting room. Oh noes, your mail seems to be getting lost a lot, huh?"

Also in this analogy, USPS runs their own CD rental service...

Re:Could be a different route involved for the VPN

[KingMotley]

The customer paid for a 75Mbps link between him and the first hop. He got that. He didn't pay for a 75Mbps link between him and every possible host on the internet. That's silly.

Re:Could be a different route involved for the VPN

[EvilJoker]

Point being, for Verizon/Netflix to make any arrangement, the benefits most flow both ways

It does. In exchange for hosting the box, Verizon only needs to transfer a fraction of the data. This saves on bandwidth costs (on both sides), as well as increases customer satisfaction with both companies.

The general consensus, however, is Verizon is holding out, expecting an even larger payout directly from Netflix, for doing the same thing.

Re:Could be a different route involved for the VPN

[EvilJoker]

Part of the confusion here is caused by vertical integration.

The claims, if they were simply made by Verizon FiOS (a tier-3 ISP), would be total bullshit. Tier-3 ISPs do not have peering arrangements. They buy bandwidth from 1 or more upstream providers.

Verizon Business is an old-school telecom company, and owns the former UUnet. This part of the company is a tier-1 network, as is Level 3. They do have peering arrangements.

Since they have the same name, and are at least partially the same company, this leads to a lot of mistaken conclusions. Naturally, FiOS buys most/all of their bandwidth from Verizon Business.

Re:Could be a different route involved for the VPN

[Yakasha]

I would think the more apt analogy is that you sold me unlimited access to your fridge (bandwidth) but Netflix (content provider) is only restocking at a rate of one six-pack per week. IOW, Netflix is the one failing to have peerage agreements in place to honor their downstream sales commitments.

So if a city promises "With this new tax, you will have perfect roads throughout the city!", and you pay the tax, but the roads aren't there, you blame Ford for not ensuring the roads are available?

That makes sense to you?

Re:Could be a different route involved for the VPN

[evilviper]

How does traffic generated within verisonz ASN, and exists within the same verizon ASN, even need BGP to function?

This discussion is all about data going from: Netflix --> Level-3 --> Verizon

How are you getting Verizon --> Verizon out of this, and how is that relevant to Netflix?

Re:Could be a different route involved for the VPN

[Parts09]

This is what irritates me so much about discussions like this. the ISP of the content requester is the one paying the ISP to transit this information. The ISP is doing nothing for free. Peering agreements were to help the ISP so they wouldn't scream so much. But in the end they need to remember that they aren't screwing Netflix. They are screwing their paying internet clients.

Alternative explanation

[IamTheRealMike]

Routing traffic via the VPN changes the path the traffic flows over, possibly avoiding routes that are saturated and (who knows) pending upgrade.

It's tempting to imagine the internet as a giant blob of fungible bandwidth, but in reality it's just a big mess of cables some of which are higher capacity than others. Assuming malice is fun, but there isn't enough data here to say one way or another.

Re:Alternative explanation

[Ecuador]

Except Verizon here lets just some "low capacity" cables connect them to Netflix's provider on purpose (as illustrated in a recent /. article), so there can be no other reason apart from extorting money. And for the speed to actually going down with time it probably means that instead of "upgrading" capacity they are probably doing the opposite to force Netflix. And they are lowering the speed slowly otherwise their customers would figure it out and start rioting (but many don't have any recourse as in, alternative ISP).
Remember, Netflix actually offers installing their servers within the ISP's network for free, which would mean no interconnection.
And, finally, this is 1 week old "news for nerds". I have read about it in so many tech sites, I was certain it would be a dupe (but a quick search seems to indicate it is not).

Re:Alternative explanation

[jaseuk]

This has been rather done to death (http://www.extremetech.com/computing/186576-verizon-caught-throttling-netflix-traffic-even-after-its-pays-for-more-bandwidth) , but Verizon doesn't appear to be throttling or shaping Netflix. They are running their peered links to Layer 3 at 100% capacity. Traffic that doesn't go via Layer 3 does not suffer. So if you find an alternative route that doesn't use Netflix's Layer 3 peering connections (such as a VPN) then things run well.

For this to be resolved, people really need to find non-Netflix services that are equally impacted and bring this up. It may well be that 90% of Verizon's Layer 3 pipe is for Netflix, but there are bound to be other services suffering. If this can be demonstrated this puts other parties into the equation and should encourage Verizon to take up Layer-3s offer of additional free peering capacity.

I suspect that Verizon would rather that Netflix isn't running at full-speed as it quietens down their overall network usage and can somewhat claim they are not capping or throttling. Perhaps Layer-3 should shut down these peering points for maintenance and let the Verizon find a way through another peer / transit, it might melt the whole of Verizon's network that way and encourage them to solve it.

Jason.

Re:Alternative explanation

[dfghjk]

"It's tempting to imagine the internet as a giant blob of fungible bandwidth, but in reality it's just a big mess of cables some of which are higher capacity than others."

No, it's a giant blob of fungible bandwidth when you are talking about large ISPs and major media sites. It's not the dark ages.

Re:Alternative explanation

[Charliemopps]

Routing traffic via the VPN changes the path the traffic flows over, possibly avoiding routes that are saturated and (who knows) pending upgrade.

It's tempting to imagine the internet as a giant blob of fungible bandwidth, but in reality it's just a big mess of cables some of which are higher capacity than others. Assuming malice is fun, but there isn't enough data here to say one way or another.

I suspect that whats going on is that Netflix put the majority of their traffic on Level3 and Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer. Verizon said "No" and told Netflix to go with another peer. So Verizon has plenty of bandwidth, Netflix has plenty of bandwidth... it's where those peers are located that's the problem. Level3 has been giving Netflix huge discounts to try and force ISPs into unfriendly peer agreements.

So yes, if you VPN'd out to somewhere else... somewhere that's not an ISP and place where Level3 isn't trying to screw people, then yes, you'd avoid the route in question and get great service. Move all of Verizons traffic that way and see what happens. Ignoring saturation of the peers... It would work until that VPN services peering agreements ran out and then they'd be getting the same treatment.

The FCC, ISPs and Netflix need to stop screwing with net neutrality and fix the god damn peering agreement process. I've been involved with them peripherally and they're like the wild west when the 2 sides can't agree on something.

Re: Alternative explanation

[bill_mcgonigle]

If there are many paths to a node their system should be choosing the fastest path.Verizon obviously is not doing that and deliberately allowing congestion.

And Netflix would happily give them OpenConnect appliances too, to avoid _their_ bandwidth costs as well. But Netflix competes with Verizon's VoD services - this isn't hard to figure out.

There are at least three underlying problems for the congestion issue - one is the DMCA and related copyright laws that prevent any sort of sane caching, the general fear of multicast that everybody on the Internet still seems to have (half a million unicast streams of the same show is insane - where are the global warming people on this?), and the grants of monopolies and/or prohibitions on competition that prevent local competition.

Label me shocked if the Netflix app on mobile devices does not have a P2P mode working in the lab right now, as a workaround for us running a sub-par Internet.

Re:Alternative explanation

[Anonymous Coward]

Here's the thing: If I'm a customer of a given ISP, I DON'T GIVE A FLYING FUCK IF THEIR PEERED LINK TO LAYER 3 IS AT 100% CAPACITY!

If they're having capacity problems, even with thousands or even millions of customers paying hundreds of dollars a month for service, then there's only one thing they can do: BUILD OR ACQUIRE MORE PEERING CAPACITY WITH LAYER 3!

Re:Alternative explanation

[amorsen]

Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

No. That is not how it works. The truth is that the smaller provider pays the larger provider, no matter which direction the traffic flows. Some companies, like Netflix, are nice enough to not use their size as an excuse to charge people -- they offer free peering at internet exchanges. Other companies are maximally greedy.

Re:Alternative explanation

[GNious]

Wouldn't a trace-route serve to show whether traffic is flowing over a distinctly different route?

Re: Alternative explanation

[amorsen]

Multicast is not a viable technology for truly large scale deployments (more than a few hundred thousand hosts perhaps). Routers and switches do not have the required resources to maintain multicast routing/switching tables for millions of multicast sessions.

The correct way to solve the problem is to push it to the end nodes. They have much more CPU power and memory than routers and switches. The technology to do so has existed for a long time: P2P.

Re:Alternative explanation

[Anonymous Psychopath]

It's not just Level3 (not Layer 3), it's also Alternet and possibly others. Peering has gotten tough. It's supposed to be hey, let's connect our stuff together because I want to send you a bunch of stuff and you want to send me a bunch of stuff and we both win. The Internet has evolved and that has resulted in asymmetric traffic flows where one party carries more (sometimes far more) of the burden than the other, but the cost models have remained the same.

In Verizon's mind, they receive no benefit from increasing peering capacity in cases where they receive far more traffic than they can send. They forgot one thing, though; their residential customers. They are the ones who need the additional capacity, and without it their service will continue to degrade.

Re: Alternative explanation

[alen]

It's up to Netflix to buy cdn hosting to improve their speeds. ISP is just a dumb pipe, remember?

Everyone buys cdn hosting for their content except Netflix

Re:Alternative explanation

[guises]

I suspect that whats going on is that Netflix put the majority of their traffic on Level3 and Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer.

It's Verizon who is trying to charge for access to their customers (who have already payed for the service that they're not getting), not the other way around.

Re:Alternative explanation

[jedidiah]

That is true perhaps.

However, this is all entirely Verizon's fault. They are the entity in this arrangement that has actively encouraged assymetric use of the net by offering assymeteric service. It's really rich to see ISPs complain that they are getting too much traffic all in one direction then that's how they f*cking design their service.

Verizon is selling massive downloads. So is every other consumer ISP.

Re:Alternative explanation

[silas_moeckel]

But that is not how the internet has traditionally worked. Teir 1 providers (those that pay nobody for bandwidth) which L3 and Verizon are do not pay each other ever. L3 is more than happy to increase port capacity/count and Verizon is refusing to do so. Put more simply Verizon is refusing to increase capacity in hopes to get the Comcast deal with Netflix. Netflix is also happy to give Verizon CDN gear to deploy on their network again a common practice.

Verizon is a teir 1 but pretty much an eyeball network, they send very little traffic compared to what they consume. They are effectively gaming the settlement free peering that has made the internet work to date in attempt to extort netflix and the like to buy their more expansive bandwidth to use their CDN etc etc. If you extend that logically we fall back to the walled garden days of compuserve aol etc which seems to what Verizon wants people paying to consume the things they are getting payed to send them.

For the obligatorily car analogy it's the car dealership making the factory pay them to take the cars and making the consumer pay for the car irregardless if it's the one they want coupled with the government only allowing a couple car dealerships sell to people in a given area, effectively take what they offer or get nothing.

Re: Alternative explanation

[Drakonblayde]

You do not do networking for a living very well then.

The problem isn't how Verizon is egressing traffic to Netflix, it's in how Verizon is allowing traffic from Netflix to ingress to their network. Once the Netflix traffic makes it into their network, I have no doubt it is being delivered over the most efficient route.... which means jack shit when the ingress point is such a tight bottleneck. Inbound traffic engineering is a very different animal from outbound traffic engineering.

You're not incorrect in that the fault lies with Verizon, but I seriously doubt you actually understand why or how.

Re: Alternative explanation

[MrL0G1C]

You obviously missed the article where Netflix supplies a tower-pc sized box with all of netflix on it to ISPs for free:

Netflix Boxes

Re:Alternative explanation

[Anonymous Psychopath]

They are the entity in this arrangement that has actively encouraged assymetric use of the net by offering assymeteric service.

This is half true. Verizon is selling asymmetric services (although they are changing most FiOS plans to symmetric).

What is not true is that asymmetric connections encourages asymmetric usage. It's the other way around, and has been since the days of dialup.

Re:Alternative explanation

[kqs]

...because Netflix's provider (which is Level3) isnt paying for the bandwidth disparity between Level3 and Verizon on purpose.

The bandwidth disparity argument is bunk. I've love if Netflix or Level 3 would set up some data sinks in their network so I could use my FIOS to send them random data 24/7 and help even the disparity.

Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

So... you're saying that Verizon should be paying me! I mean, they send me ALL THIS DATA (much of it sourced from Netflix), but I hardly send them anything. This makes me both a selfish person and someone who deserves a large monthly Verizon cheque.

Note that if Verizon doesn't want to pay a few grand for a few more 10GE ports and some short cables, they could pay even less and accept the caching servers that Netflix offers to all large ISPs; those cost just a few rack units and watts of power.

But Verizon would rather limit Netflix so that they can push their own video products.

Re:Alternative explanation

[anegg]

I had responsibility for a corporate data network a number of years ago, when cross-country link speeds were substantially lower than they are now. About 80 sites distributed across the US. We charged a flat rate based on number of "subscribers" (network users) at each location to put a location on the network as part of our cost-recovery strategy. The CIO asked me to develop a traffic-based charge instead/in addition to the subscriber charge. We analyzed the situation as follows:

If the source of traffic is charged for providing data to the network, it will limit the services that the source chooses to provide, even if those services are very beneficial to the rest of the network.

If the sink for the traffic is charged for the data it receives from the network, it can cause the sink to be charged for data it didn't request or cause that site to stop using services that create a better overall result for the corporation as a whole.

Locations subscribe to the network because they want access to services and because they want to be able to provide services. Charging by traffic would force providers and consumers into a level of analysis and complexity that would ultimately limit the usefulness of the network, and stifle creativity and growth. On top of that, adding cost-accounting to the network based on traffic would add about 30% to the cost of operating the network.

All kinds of "unfairness" exist in the network world. Our more distant locations thought it very unfair that they had to pay big bucks for a lower speed connection than our customers located at a corporate hub site, even though our actual cost to connect those customers was several times what we charged them, for example. Because we were a corporation, we could decide that the cost of the network was a pooled cost that benefited everyone, and that the best cost-model for the corporation's benefit was the flat-rate subscriber cost regardless of distance.

The commercial world is a little bit different than the corporate world, because the sources/sinks are more polarized (I'm suspect NetFlix puts more traffic onto the network than they pull off of the network). But the arguments seem similar in nature. Consumers on the network are there specifically because they expect to get traffic from providers. Verizon would have a much harder time selling Verizon's services (especially higher download rates) if the rest of the Internet didn't have other firms providing data that Verizon's consumer customers wanted to download. The value of the network is in both the sourcing and sinking of the traffic flows combined. Focusing the accounting on just one direction of flow ignores that value.

Re:Alternative explanation

[johnw]

Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

Really? I'm only an end user, but my experience is that the charging is the other way round. Traffic to me is metered (and I pay for) whilst traffic which I originate is un-metered.

Re: Alternative explanation

[kqs]

No, Netflix (and Youtube and some others large ones) don't buy CDN hosting; they offer it. They offer free CDN servers which large ISPs can put in their datacenters. Doesn't matter how much Netflix offered to pay, I doubt if any existing CDN could handle Netflix's traffic along with their other customers.

Many ISPs take advantage of this, but Verizon would rather degrade Netflix's products so they can push their own products.

Re: Alternative explanation

[Drakonblayde]

There are at least three underlying problems for the congestion issue - one is the DMCA and related copyright laws that prevent any sort of sane caching, the general fear of multicast that everybody on the Internet still seems to have (half a million unicast streams of the same show is insane - where are the global warming people on this?), and the grants of monopolies and/or prohibitions on competition that prevent local competition.

I agree with you that half a million unicast streams can be nuts, when it comes to on demand content, multicast is a non-starter (we'll ignore the fact that multicast is sorely lacking in security features and would require some serious re-engineering on many networks to work... there's a very big reason why inter-domain multicast routing is not seriously employed, and it has nothing to do with fear).

Think about how this traffic flows -

Sub1 wants show A and starts playing it on Netflix.

10 minutes later, Sub2 wants show A as well.

What happens if this stream is multicast? Well, Sub2 gets the show 10 minutes later, assuming he's joining the same multicast group as Sub1. Sub2 is not happy, he wanted to watch the entire show.

How to get around this? Well, ok, so Netflix could just mux the feed for Sub2 inside the feed for Sub1, and presumably, the client would be able to tell which parts of the feed were for which sub. However, the problem is that the data for Sub2 would still be delivered to sub1, sub1 would just throw it away and pay attention to it's own data. However the data for both feeds have to transit the same backbone, which drives capacity usage up. This is also unsustainable, as eventually, as more subscribers joined, the feed would grow so large that it would saturate the downstream of all the subscribers receiving it and eventually lead to packet loss.... which would lead to loss of video, stuttering, etc. All the same shit thats going on now.

Ok, so muxing different streams into the same feed for the same show isn't going to work.

So Sub2 could just start getting the feed over a different multicast group, that would solve the ever growing feed problem!

Except that if you do that, there is functionally no difference between sending the traffic to a multicast destination and a unicast destination.

So sure, while sending half a billion unicast streams seems insane, especially since alot of those will be watching the same show, the fact is that a very small percentage are going to be watching the same show at the exact point in the show. For the most part, they will all be at different parts. Multicast is a solution when the data for the given event is live, or when it's linear (ie, this is the point we're at, and there's no going back). On Demand services does not fit either of those profiles.

Multicast has it's place in the on demand world, but only on the backend. It's wonderful for things like distributing a new asset to all of the streaming sources, or for filling caching servers that will the streaming boxes will need to pull from, but multicast is simply not a workable or superior implementation when it comes to delivery of content to subscribers.

Re:Alternative explanation

[module0000]

You say "Layer 3"(a step in the OSI model), do you mean "Level 3"(an ISP)? They are the ISP and backbone provider that has owns the CDN appliances caching and delivering the [majority of] Netflix streams in the USA.

Re:Alternative explanation

[Anonymous Psychopath]

Thats how the internet is paid for. The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

No. That is not how it works. The truth is that the smaller provider pays the larger provider, no matter which direction the traffic flows. Some companies, like Netflix, are nice enough to not use their size as an excuse to charge people -- they offer free peering at internet exchanges. Other companies are maximally greedy.

That's not how it works, either. A peering arrangement is an interconnection between two providers in which neither pays the other for traffic. Verizon would like to change this model and receive payment.

Re:Alternative explanation

[ArcadeMan]

People assume that the Internet is a strict connection from server to client. But actually from a non-linear, non-connected viewpoint it's more like a big ball of wibbly-wobbly, streamey-wimey... bits.

Re:Alternative explanation

[MrL0G1C]

No, the traceroute wouldn't show the hops between your PC and the VPN server, so that part of path could not be compared. So the point the VPN packets leave the ISP network also couldn't be compared.

You could probably compare an unencrypted proxy with a direct route to Netflix.

Re:Alternative explanation

[Drakonblayde]

I suspect that whats going on is that Netflix put the majority of their traffic on Level3 and Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer. Verizon said "No" and told Netflix to go with another peer. So Verizon has plenty of bandwidth, Netflix has plenty of bandwidth... it's where those peers are located that's the problem.

Ok, but you're wrong.

Level3 has admitted they have settlement free peering with Verizon. Level3 does not pay Verizon anything. Verizon does not pay Level3 anything.

Netflix pays Level3. This is why Level3 gives a shit about this situation.

What's going on is that Verizon is trying to cut out the middleman. Verizon wants Netflix to pay them to get traffic into their network instead of paying Level3 to deliver traffic into the Verizon network. Why? Because they don't make any money from Level3.

Naturally, Level3 is all in a huff about Verizon trying to fuck with their revenue stream.

Re: Alternative explanation

[oobayly]

Have they started naming and shaming the ISPs who refuse to host a Netflix Open Connect box in their data centres?

Re:Alternative explanation

[oobayly]

Surely what you'd do is traceroute to the VPN server, which will show you where the packets leave the ISP network (as long as the VPN is outside of it), and then traceroute to Netflix via the VPN. The compare it do the route taken directly to Netflix.

Re:Alternative explanation

[Rockoon]

I will be an ISP that starts requesting random files from everybody else and CHARGE them if they are dumb enough to send them!!!

Thats better than the alternative, where you can just send unrequested data out to arbitrary IP addresses and then expect payment. In the scenario you suggest, the sender can at least opt out and stop sending if someone is abusing them.

This is the same reason that the sender pays the postage on snail mail and that it really cannot rationally be any other way except in rare circumstances that cannot be the general case.

Netflix pays Level3 a fee to send data to you. Verizon wants to be paid for handing data that originates from Level3's customers that is destined for its customers. In real terms Level3 should be charging Netflix a premium for destinations such as Verizons network so that Verizon can be reimbursed the expenses associated with handling the volume of traffic originating from Level3.

Well in the case of snail mail, thats exactly what happens. If you want to send a letter from Canada to the United States (or vise-versa) the regular local postage fee isnt enough. The sender has to pay a premium so that the other countries postal service can be reimbursed even in the case where the person its destined for requested the letter.

The exception is when its postage due, but we already know that everyone is against Verizon charging customers that use Netflix more than the customers that do not use Netflix.. "Net Neutrality" and all that.

Re:Alternative explanation

[MrL0G1C]

Good point, I didn't think of that doh.

Re:Alternative explanation

[Anonymous Coward]

That's not how it works, either. There's settlement-free peering (the "real" peering) and there's paid peering and there's transit. Peering is when two networks exchange data which is destined for the respective other network. Transit is when two networks exchange traffic that is destined for some other network. Transit is the real cost factor, because nobody carries transit traffic for free: You want me to carry your data to someone else? What's in it for me? Whether peering is settlement free or not is a matter of negotiation. Most networks publish peering policies in which they describe where and with whom they will peer and what the conditions are. For example: Google, Comcast, Verizon, and few others. If you want to dig even deeper, there's a database of peerings (use guest login). It is indeed often a matter of size, and the resulting negotiating power, who pays whom. There are however "peering sluts": CDNs will typically peer settlement-free with anyone above a relatively small minimum size, even though CDNs are true behemoths on the internet. That's because their business depends on reaching everybody, and settlement-free peering is still a lot cheaper than the transit for their huge traffic flows. Netflix is in this category, for the same reason.

Re:Alternative explanation

[wvmarle]

They forgot one thing, though; their residential customers. They are the ones who need the additional capacity, and without it their service will continue to degrade.

You're giving Verizon too much credit: the way you write this, you imply they care about their customers and the service they offer.

Re:Alternative explanation

[i.r.id10t]

If residential use was closer to symmetric, just what would the outgoing content be, and how many people would want to be consuming it?

Not only that, but just how many of the residential type customers *want* to be hosting services and offering content? How many kernel source mirrors, debian/ubuntu/mint/whatever binaries mirrors, etc. do we need?

Residential, and probably quite a few business accounts, are always going to be consuming more than producing.

Re:Alternative explanation

[i.r.id10t]

Is Verizon service even available wherever the main netflix datacenter(s) is/are? If it is, wouldn't they just need to offer a better monthly connection rate, or better speed for the same rate, and let it be a "normal" competitive service for business?

Re: Alternative explanation

[Frankie70]

They need to pay the ISP for hosting and maintaining it.

Re:Alternative explanation

[CAPSLOCK2000]

The bandwidth disparity argument is bunk. I've love if Netflix or Level 3 would set up some data sinks in their network so I could use my FIOS to send them random data 24/7 and help even the disparity.

Netflix could create a peer-to-peer network for streaming video, just like Spotify used to do. Everyone streaming video from the Netflix network would also be sending data back. That would more or less balance the situation. Customers with data-caps or on saturated networks might nog like it, but that problem could be avoided by giving customers a choice. Either get a discount on your Netflix subscription for uploading to the network or pay a premium for a download-only subscription.

Re: Alternative explanation

[MrL0G1C]

Why? It's saves the ISP massive amounts of bandwidth, I'm sure that saving hugely outweighs the cost of electricity for a few hard drives and a network cable. (yes, and the occasional HDD swap).

Re:Alternative explanation

[alen]

they are in the process of doing direct peering with netflix like comcast is doing, but it's taking a while

when the comcast deal was announced they had already done the engineering, etc
with verizon they have to build out the peering location and they are saying sometime this year

Re: Alternative explanation

[CAPSLOCK2000]

Multicast is not a viable technology for truly large scale deployments (more than a few hundred thousand hosts perhaps). Routers and switches do not have the required resources to maintain multicast routing/switching tables for millions of multicast sessions.

The reason why routers are so underpowered is that nobody uses multicast. If there was a strong demand for multicast I'm sure that the manufacturers would increase the capacity of their hardware.

The correct way to solve the problem is to push it to the end nodes. They have much more CPU power and memory than routers and switches. The technology to do so has existed for a long time: P2P.

Using P2P does not lower the total load on the network, it just spreads it out more evenly.

Re: Alternative explanation

[faedle]

Bandwidth is perhaps cheaper than you suspect.

I worked for a regional ISP that serves about 50.000 subscribers. We had multiple 10 Gigabit Ethernet connections to various peering points, one of which happens to be where Netflix peered with us. Total cost for that peerage: the cost of the extra fiber capacity, plus engineering the peer.

As opposed to housing Netflix servers at our data center. First off, to service that many potential streams might require a few boxes and a not insignificant storage array. We actually did have a similar arrangement with another very large content provider: their stuff took about a half-rack. It then needs to be added to network monitoring, and you need to train your NOC staff what to do when that little red light comes on. And the equipment will fail: the "other content providers" equipment had a MTBF of a couple of months. The hard drives will take a pounding.

And we were small enough that when we asked Netflix to co-locate in our data center for free they actually said "Not interested."

Re:Alternative explanation

[Zanthras]

Is Verizon service even available wherever the main netflix datacenter(s) is/are? If it is, wouldn't they just need to offer a better monthly connection rate, or better speed for the same rate, and let it be a "normal" competitive service for business?

Netflix and verizon do indeed have pops in the same facilities. There was a recent post by a verizon suggested that all the congestion between netflix and them was due to a 100% saturated interconnect in LA to L3. That of course means that *all* verizon traffic going through that node is badly impacted. Verizon and netflix are currently working out a deal to peer directly. This will likely entail several hundred 10g ethernet interconnects spread out across numerous locations. But until then verizon is happily screwing over their customers just for a leg up in negotiations with netflix.

Re: Alternative explanation

[alen]

every other CDN pays ISP's to host their boxes. netflix wants an advantage

Re:Alternative explanation

[alen]

i'm a light netflix user and the last thing i want is to pay more money just because someone can't live without their netflix
netflix needs to raise their prices and pay a CDN to host their data like they used to last year

Re:Alternative explanation

[alen]

if the file is 3000 miles away across three networks it's not your ISP's fault it takes a while
it's up to your provider to host the file close to you

Re: Alternative explanation

[Lehk228]

how many wifi routers do i need to buy in order to qualify as an ISP?

Re:Alternative explanation

[kqs]

Do you really think that studios would let their movies stream peer-to-peer, which would involve being stored on home-user's computers encrypted with a known key (aka "effectively not encrypted")? Plus, ISPs are rolling out CGN which makes peer-to-peer very difficult, and most residential connections have very slow upload speeds. Finally, this would just be a way to work around the fact that Verizon is not giving its customers what the customers have paid for: high-speed internet access. When there is a bully, you don't just sneak around behind the bully's back and hope he won't notice you.

Re:Alternative explanation

[sixsixtysix]

So comcast should pay me for the data I recieve from them. My personal network being a recieivng provider and all. Seems legit.

Re:Alternative explanation

[evilviper]

Level3 does not pay Verizon anything. Verizon does not pay Level3 anything.

Yes, but there's 2 things going on that you're neglecting to mention.

1) When Level3 started delivering Netflix streams, suddenly their peering traffic was extremely unbalanced... That's when one side has to start paying the other. Verizon wants Level3 to pay, but they won't, so Verizon refuses to upgrade the interconnect. In the bad old days, these peering disputes ended with a disconnect, and big portions of the internet broke off and many sites went offline. These days it's just lots of congestion.

2) Level3 seriously pissed-off ISPs, when they entered the distributed content delivery business. ISPs get paid good money by Akamai and others to host their caching servers. Level3 thought they could get in on this racket by hosting servers just outside their ISP peering points, cutting the ISPs out of the profits. This pissed off ISPs, and more than that, massively screwed-up their peering ratios.

So, on both sides, Level3 is the one who screwed-up their peering agreement with Verizon and others, without considering the consequences.

Netflix has the ability to fix it, though... If their software would tell all the clients to upload random junk to some random Netflix servers (preferably UDP, so the server doesn't even have to really exist), even when idle and not watching videos, they could move Level3's ratios back to even up/down distribution, and really punish the local ISPs who claim they want even up/down peering, at the same time.

Re:Alternative explanation

[evilviper]

If residential use was closer to symmetric, just what would the outgoing content be,

Backups. Encrypt and upload your entire harddrive to Dropbox. Then upload anything that has changed, every single day...

Eventually, you'll need to download everything, but you'll generally be uploading many, many times more than you download.

Re:Alternative explanation

[evilviper]

However, this is all entirely Verizon's fault. They are the entity in this arrangement that has actively encouraged assymetric use of the net by offering assymeteric service.

Verizon just became the first consumer ISP to switch to symmetric connections, with all their FIOS services.

Re: Alternative explanation

[evilviper]

Netflix supplies a tower-pc sized box with all of netflix on it to ISPs for free

That's like saying your boss is willing to give you a full-time job for FREE. In other words, failing to pay you... But it's FREEEEE! Don't you feel special, getting something for FREE?

It's Netflix who's trying to get something very valuable for free, out of the deal. Other content delivery network providers, like Akamai, pay good money for the privilege of having their servers hosted by ISPs on their fat pipes near to customers.

 

Re: Alternative explanation

[apraetor]

Verizon was called-out for refusing to do it, simultaneously refusing to increase their L3 peering (for free!) to compensate, and then claiming Netflix was the cause of degraded Netflix streaming: http://blog.level3.com/global-...

Re: Alternative explanation

[evilviper]

Except that if you do that, there is functionally no difference between sending the traffic to a multicast destination and a unicast destination.

Nonsense. With a subscriber base as large as Netflix's, there will always be several people requesting the same video stream, within a minute or so of each other.

Well, Sub2 gets the show 10 minutes later, assuming he's joining the same multicast group as Sub1. Sub2 is not happy, he wanted to watch the entire show.

Actually, when Sub2 joins, his system can start buffering the show 10-minutes in, and while doing so, send a request to resend the first 10 minutes of the video. Netflix servers save bandwidth, and both users watch the video they wanted, when they wanted.

Re:Alternative explanation

[NormalVisual]

No, the traceroute wouldn't show the hops between your PC and the VPN server, so that part of path could not be compared.

This is true as long as the VPN link is up. If it's down, then it's trivial to do a traceroute between yourself and the VPN server to fill in the missing hops.

Re:Alternative explanation

[sjames]

You're confusing transit where balance is a reasonable requirement and delivery where it is not at all reasonable.

Re:Alternative explanation

[sl149q]

I would believe the bandwidth disparity argument if end user connections where also balanced, same upstream and downstream bandwidth.

If Verizon is selling asymmetric connections to end users then how can those connections generate a balanced load (same up and down)?

My local cable connection is 75MB/s down and 5MB/s up. And if I try and use a substantial portion of the UP the ability of TCP to pull large amounts of data down suffers.

Re: Alternative explanation

[amorsen]

The reason why routers are so underpowered is that nobody uses multicast. If there was a strong demand for multicast I'm sure that the manufacturers would increase the capacity of their hardware.

If you build it and it costs less than 6 figures USD, you will drown in customers. It would not be used primarily for multicast at first, it would be used to get BGP working better, but every major ISP would want your router.

Using P2P does not lower the total load on the network, it just spreads it out more evenly.

Correctly done P2P sends traffic through the best route, typically from someone on the same ISP as the recipient and preferably from the same neighbourhood. That lowers total load a lot. Most current P2P networks do not particularly worry about optimal routing; they are much more constrained by traffic shaping or (often artificially) limited last-mile upstream capacity. It would be fairly easy to give priority to low-latency peers.

Besides, P2P can solve the problem of subscribers not watching at exactly the same time. Multicast breaks as soon as someone presses pause.

Re:Alternative explanation

[symbolset]

I wonder why Level 3 doesn't just route most of the Netflix data though another backbone peer. Might as well saturate all of Verizon's peers.

Re:Alternative explanation

[sjames]

Verizon is paid for carrying the data, by it's customers. Nobody will pay them anything is they only connect their own customers together (see how that worked out for Prodigy).

Unless you know of some reason why bytes from Netflix cost more to carry than bytes from (for example) Hulu, then the rest of the world is pretty much in agreement, they shouldn't charge more.

Re:Alternative explanation

[sjames]

Especially while ISPs have no-server policies and provide connections that aren't suitable to servers.

Re: Alternative explanation

[symbolset]

The Netflix boxes are adapted from Backblaze 4U 45 drive storage pods. It is called the Open Connect Hardware Appliance. They are not insignificant and I hear each one is designed to support up to 50,000 end users.

If you have 50,000 people paying you $80 a month and 50% of what they use it for is Netflix, you can afford the power and rack space for one 4U box.

Re:Alternative explanation

[Rich0]

I don't care if it is malice or laziness.

I paid for connectivity at a certain bandwidth. I get that it is oversubscribed, and I have no expectation of running a server that sucks down data at 50Mbps 24x7. However, if I'm actively watching an online video stream with my own eyeballs I expect to get 50Mbps, because that is what I paid for. They have other services at lower costs, and if I were only paying for one of those I'd only expect to get what I paid for.

When I pay FedEx to deliver a 1 pound package at 10AM the next morning for $70, I don't expect to get an apology about rush-hour traffic. I expect my package to arrive on time, because rush-hour traffic is something that exists every day and FedEx should allow for it and price their service accordingly. If there is an earthquake that destroys the roads then I can understand a few excuses.

When you're an ISP with millions of customers, congestion is just an ordinary part of doing business and you're expected to manage it. Anybody competent will see the need to upgrade long before they hit 100% saturation.

Re:Alternative explanation

[Alsee]

Level3 is trying to charge Verizon an exorbitant rate for enough bandwidth to handle that peer. Verizon said "No"

No. Level3 offered to upgrade the connection FOR FREE. Level3 offered to pay 100% of the cost of the extra hardware to upgrade the link and GIFT it to Verizon.

The second part of your comment was correct.... the part about Verizon saying "No". Verizon doesn't want the problem fixed for free - Verizon wants to use their monopoly position to bottleneck their customer's datastreams, to try to extort a slice of the content-revenue-stream pie.

Verizon has plenty of bandwidth, Netflix has plenty of bandwidth

Yep. Verizon themselves put out a graphic showing that there's abundant bandwidth, and that the entire problem is the one chokepoint where they're linked to Level3. Which Level3 offered to foot 100% of the bill of fixing.

-
-

Re:Alternative explanation

[Alsee]

The sending provider pays the receiving provider for the bandwidth, and this is the only rational way it can be.

Right..... because when Verizon customer's pay for internet connection service, and Verizon customers request pages and media from Wikipedia.... Wikipedia should pay Verizon. That totally makes sense. On crack.

packets originating on their network

Everything is originating on Verizon's network..... Verizon customer's are the ones wanting to open a connection to Netfix and request the data.

When I make a phonecall to someone, and I spend 99% of the call listening to what that person has to say, NO ONE is going to buy that my local phone company can SEND A BILL TO THE PERSON I CALLED.

-

Re:Alternative explanation

[bigpat]

Naturally, Level3 is all in a huff about Verizon trying to fuck with their revenue stream.

And naturally Verizon customers are all in a huff about Verizon trying to charge them twice for bandwidth they aren't receiving.

Re:Alternative explanation

[bigpat]

Netflix has the ability to fix it, though... If their software would tell all the clients to upload random junk to some random Netflix servers (preferably UDP, so the server doesn't even have to really exist), even when idle and not watching videos, they could move Level3's ratios back to even up/down distribution, and really punish the local ISPs who claim they want even up/down peering, at the same time.

Yes, okay. That is a funny thought experiment, but Verizon isn't actually confused about the fact that as a local ISP Verizon customers are the ones actually requesting netflix video so the concept of peering traffic "parity" does not apply to local ISPs connecting to backbone providers.

Re:Alternative explanation

[evilviper]

the concept of peering traffic "parity" does not apply to local ISPs connecting to backbone providers.

It always has before.

And Verizon is a backbone provider in their own right, not just a local ISP.

Re: Alternative explanation

[Drakonblayde]

Actually, when Sub2 joins, his system can start buffering the show 10-minutes in, and while doing so, send a request to resend the first 10 minutes of the video. Netflix servers save bandwidth, and both users watch the video they wanted, when they wanted.

That's not how Multicast works. The source has absolutely no idea who the members of the stream are. Only the edge routers know that. IGMP joins do not get forwarded up the multicast tree, only PIM joins. There would have to be some additional intelligence built into the app that would tell the stream to start adding additional data for the buffered client to the stream. And then you *still* have the problem that all that additional buffering data? Gets delivered to *EVERY SINGLE MEMBER OF THE STREAM*. So you end up saturating and transiting data that not everyone needs. This is a large part of the reason why no one uses PIM Dense mode, the way it works, alot of unnecessary traffic gets flooded.

So sure you could have the late joining client join another stream to get the first 10 minutes they missed, staying in the old stream while they buffer so that when they're caught up they can just drop the catchup stream and stay in the original multicast. Or you could unicast the first 10 minutes to them.

Or, you could just unicast the entire damn thing, which is alot less complex operationally, and doesn't require your upstream providers and their customers to essentially reconfigure their entire networks.

Now, that's just the engineering side. Let's look at the business side. Doing that would require effort from software developers, network engineers, and network operators. All to improve Netflix's experience. For the providers that Netflix's customer base connects to, there's no incentive, especially since they likely already have complex multicast implemetations of their own (which will usually be the case for any provider that hosts video). They will expend alot of man hours, possibly some opex and capex for absolutely no return. If they're publicly traded, this will make shareholders unhappy.

For the transit providers, they have even less incentive to make this go. If it did decrease Netflix's bandwidth usage, it would directly effect their revenue, unless Netflix was stupid enough to agree to pay the same amount of money for less bandwidth used.

Re:Alternative explanation

[bigpat]

the concept of peering traffic "parity" does not apply to local ISPs connecting to backbone providers.

It always has before.

And Verizon is a backbone provider in their own right, not just a local ISP.

Again I am not sure why I am feeding a troll, but you seem to be in a gang of trolls. So sure... if we were talking about L3 sending data to Verizon that Verizon then had to ship across its own backbone then you would have a point. But in this case L3 is acting as the long haul backbone provider and all Verizon has to do is deliver the packets to the local customers that requested the data. Verizon is already charging its customers for the bandwidth, the only issue here is that they are choosing to not to deliver on that promise in order to try and shake down Netflix and by extension Verizons own customers for more money. This is a fraudulent business practice pure and simple. Enron would be proud.

Re:Alternative explanation

[Agripa]

It's tempting to imagine the internet as a giant blob of fungible bandwidth, but in reality it's just a big mess of cables some of which are higher capacity than others.

Wibbly Wobbly Timey Wimey ... Stuff.

Just wait until it becomes more economical to sell IPv4 addresses via tunneling.

Re:Alternative explanation

[evilviper]

There's never been any such exceptions to peering arrangements before. End of story. You think the way the internet has always worked should change, but don't want to admit it.

Consider it the other way around. Level-3 certainly wouldn't be jumping at the chance to offer free-peering with some tiny ISP. Is that a "fraudulent business practice" from Level-3?

Re: Alternative explanation

[evilviper]

Your first paragraph is a pointlessly incredulous waste of time.

The rest is vastly exaggerating the difficulty... There are several very small open source multicast file distribution software packages out there that already do all of this, and work perfectly. It is not a major technical feat.

Re:Alternative explanation

[devman]

I'm not against charging Netflix-FIOS mutual customers more than non-mutual FIOS customers. As long as it isn't because they are Netflix customers and instead because they are using more bandwidth I see no reason that it would run afoul of net neutrality. People should pay for what they use, If Verizon isn't charging enough for bandwidth that is a Verizon problem not a Netflix problem.

Re:Alternative explanation

[EvilJoker]

Netflix actually pays for multiple paths, including Akamai, LimeLight, and Cogent. Due to the way routing works, most, if not all, of the data from Netflix to Verizon goes via Level 3. I can promise that L3 (and the rest) charge Netflix accordingly for the volume of data they send.

As for charging extra depending on the destination, that would be even worse than the current discussions about net neutrality. Do you really want to have to figure which networks are between you and a customer? That sounds unbelievably awful.

Re:Alternative explanation

[EvilJoker]

To this day, you can find a lot of encrypted movies (WMV) on P2P networks, which require you to purchase a license to play. Apparently, this has been sufficient.

Also, I'm absolutely certain that Netflix is not re-encrypting every stream for every user.

Re:Alternative explanation

[evilviper]

the nature of peering agreements should change

I don't have any problem with that argument, as long as those who espouse it are honest and up-front about it, instead of claiming peering disputes are something new, and a criminal conspiracy, or other such utterly baseless bull.

Providers refuse to buy transit and refuse to peer

[Anonymous Coward]

Without enough transit from the backbone providers and without direct peering, there's going to be congestion. Providers want to get paid by both sides. Unfortunately Netflix has foolishly given in once. Now everybody wants a piece of the pie. Anyone who has a lot of data that users want should refuse to pay for peering with anyone who has the users who want the data, and vice versa. If they can't reach a peering agreement, both need to buy sufficient transit from third parties, so there's the incentive.

Role reversal

[Anonymous Coward]

I have to wonder, what would happen if customers were to start throttling the payment of ISP's?
"You will get your payment when you actually fulfil your end of our contract, but not before."

Re:Role reversal

[Travis Mansbridge]

You really have to wonder? Late payment fees is all they will get.

Re:Role reversal

[Anonymous Coward]

Oh you didn't get the full amount sent?

Must be a problem on your end.

Re:Role reversal

[Omeganon]

(*) actual speeds not guaranteed.

It's in every agreement so to them, they _are_ providing the service they claim.

Re:Role reversal

[Anonymous Coward]

Oh you didn't get the full amount sent?

Must be a problem on your end.

Oh, your service completely stopped working? Must be your modem. Try turning it off and on again. *click*

Re:Role reversal

[wvmarle]

Those customers will just get disconnected.

Re:Role reversal

[green1]

Simple, they cut off your service for non-payment, and you move your internet connection to the competi....er... well, does Netflix still run a DVD service?

Re:Role reversal

[ToasterMonkey]

I have to wonder, what would happen if customers were to start throttling the payment of ISP's?
"You will get your payment when you actually fulfil your end of our contract, but not before."

You should go review your contract.

Dupe

[Anonymous Coward]

This is a dupe of Verizon's Accidental Mea Culpa.

It's based on the same blog post by Colin Nederkoon. The explanation in the earlier story seemed much more convincing.

Re:Enraged Customer doesn't bother to research

[Khyber]

"So obviously the traffic through the VPN is taking a different path."

Oh? Funny, my VPN tracert shows it taking pretty much the same path across Verizon's networks before hitting Level3 and Netflix.

The Answer is Violence

[LordKaT]

Verizon is too big, and our government does not care.

The only answer is to actively work to destroy Verizon until they acquiesce or no longer exist.

Re:The Answer is Violence

[alphatel]

Verizon is too big, and our government does not care.

The only answer is to actively work to destroy Verizon until they acquiesce or no longer exist.

Strange, it feels like we already did that once.

well now that cat's out of the bag

[MrKaos]

vpn throttling, here we come

Re:well now that cat's out of the bag

[Virtucon]

I was going to say that. They'll just play whack-a-mole and start throttling that traffic next.

Re:Enraged Customer doesn't bother to research

[MrL0G1C]

If the start point of your VPN is not the VPN server IP address then your VPN is not working. Or your VPN exits within the same network as it starts which could render it pointless.

Too bad it won't do any good.

[DewDude]

Verizon doesn't care. They own RedBox Instant; they last thing they want is customers using Netflix. We're not gonna get net neutrality out of the FCC (the public comments are a sham; the FCC only care about the businesses involved in the decision); so this is not going to get fixed. If Netflix uses Level3; they were cripple all level3 connectivity.

Re: Do they deserve the benefit of the doubt?

[Anonymous Coward]

You win the most absurd analogy ever award.

Funny thing is

[smartin]

I recently upgraded my FIOS service and they used Netflix streaming as one of the reasons that I should do it. After going from 25/5 to 50/25 I still get downgraded quality when watching flix.

Re: Do they deserve the benefit of the doubt?

[jc42]

You win the most absurd analogy ever award.

Hey, nobody told me this was a contest ...

What the hell kind of dialup did you have?

[Anonymous Coward]

375k!?

No, I get your point... just...

Re:What the hell kind of dialup did you have?

[RivenAleem]

I only got his point just now! Damn throttling!

Throttling vs routing

[Omega Hacker]

First off, I assert that whether Verizon is actively throttling packets, or simply not providing sufficient peering to get to Netflix, they are committing fraud by advertising high speeds and not delivering them.

However, to *really* convince people, more rigorous experiment has to be performed: find a VPN (or set up your own with a colo) that's connected as closely to Verizon as possible, as close to their peering with Netflix as possible. That way the route between Verizon and your VPN/colo is as similar as possible to the Verizon<>Netflix route. You can then measure Netflix bandwidth to your VPN/colo, and the resulting full-path bandwidth.

I *strongly* suspect you'll see the exact same behavior, but by doing that you've proven beyond a shadow of a doubt that Verizon is absolutely to blame. It still doesn't separate the packet-throttling scenario from the insufficient-peering scenario, because even though your Verizon ingress point is ideally the same router, Netflix is *supposed* to peer to that router through dedicated lines (e.g. trunked 10G to the next room over where Netflix's router is).

Of course, since Netflix has offered to both purchase and install the 10G cards and wires on their own dime, that scenario is absolutely no different than packet-throttling. Except that in order to do packet throttling, Verizon had to spend *more* money on hardware than they would have to just add more capacity. Now *there's* a bit of research to do: $ to throttle vs $ to add capacity.....

A little disappointed in slashdot

[butchersong]

This is not news. Verizon, Netflix and customers all already know what the problem is. Routing around the bottleneck (saturated interconnect) by using a VPN will obviously avoid the bottleneck... I don't understand why this was posted as a story.

Re:Enraged Customer doesn't bother to research

[Khyber]

Funny enough, you're right, my VPN host is inside Verizon's network. But that still goes to show, it's quite possible Verizon is actively throttling that which they can see.

comcast

[GarretSidzaka2467]

My colleague and I can confirm Comcast does this as well. Put on the VPN and your streaming rates improve

Re:Enraged Customer doesn't bother to research

[alen]

because netflix contracts with level 3 and cogent to send their data to before it reaches the ISP and that is who it goes through. netflix can't just send their data to every tier 1 backbone on the internet without paying them

Re:Enraged Customer doesn't bother to research

[MrL0G1C]

A previous story says rather than actively throttling the connections, the bottleneck is the lack of an extra cable between 2 cabinets - Verizon's and the peering providers. The peering provider even offered to buy and install the cable for free!

Re:Enraged Customer doesn't bother to research

[MrL0G1C]

Ps, http://blog.level3.com/global-...

The fix was actually a couple of network cards + cable, which Level 3 offered to sort out.

We really need some laws agains false advertising

[Anonymous Coward]

This shit is just getting silly.

Vertical integration

[yet another SanTiago]

Seems like many peering conflicts are related to excessive vertical integration of some providers. Verizon is currently both tier 1 provider and consumer ISP. Suppose it would be split to Verizon1, tier 1 provider, and Verizon 2, tier 2 consumer ISP. Verizon1 would be one of several upstream providers to Verizon2. What wold happen in such case if there is such kind of peering conflict between Verizon1 and Lever3? Verizon2 would immediately change its routing in a way that traffic between Level3 and Verizon2 would go through another upsream provider of Verizon2, which would threaten potential revenues of Verizon1 from Verizon2, therefore Verizon1 would have much stronger incentive to promptly solve the dispute.

Not surprised

[Sleeping Kirby]

my wife: "Not that we're surprised, but it's nice to have concrete facts to back it up."

Alternative explanation

[bigpat]

and (who knows) pending upgrade.

I bet Verizon knows. ;) Pending an upgrade....Ya right, more like pending a nice fat check from Netlfix or L3

Re:Enraged Customer doesn't bother to research

[alen]

level 3 and cogent are sending the data over settlement free links. those will have a 2:1 or 3:1 max traffic ratio and are basically friendly links to exchange equal amounts of data for faster routing. the baby bells still charge for terminating calls on their networks, this is a system to bypass the admin burden of equal payments and keeping track of them

trouble is sending netflix data is A LOT of data and they want to take money from netflix as payment to deliver the data but not pay anything out to actually deliver it. so they can't just send it to other Tier 1's because there is a risk the other end will disable the connection.

verizon has offered to sell them links at CDN rates, but that would mean they lose money because they charged netflix too little. they tried to game the system taking money from netflix and sending the data over links they don't have to pay bandwidth for. and of course, only netflix has these problems because they have been buying bandwidth at below market rates for the last 5 years. everyone else pays CDN's and their video traffic comes in HD quality with very little problems

MOAR Laws != Answer

[smittyoneeach]

What we need is more competition and communication of just how abusive companies are.
If people have alternatives, then there is no reward for shenanigans.

fight, fight, fight...

[rewindustry]

if it's that easy to bypass throttling, then have at it, i say - the more the better.

there has to come a point when derision and cronies will be forced to admit they have been raping yet another dead horse.

when dealing with bullies it pays to be light on your feet, i find.

Nice to see Slashdot on the bleeding edge...

[TheLongshot]

This story is almost two weeks old. People here need to get on the ball.

Re:Nice to see Slashdot on the bleeding edge...

[Chas]

Yeah, I submitted this right after the story broke and it never went anyplace.

Re:Nice to see Slashdot on the bleeding edge...

[HiThere]

Has the problem been fixed? If not, then it's still reasonable information to share.

And nobody has claimed that the problem has been fixed.

Re: Speed tests

[mindcandy]

"speed tests" (the consumer oriented public sort, eg: speedtest.net) are a joke because Oookla (et.al) encourage the ISPs to run a local copy of it .. and naturally the configuration favors the *most local* one by default .. so unless you're paying attention or use a geek-oriented test (of which there is no shortage, but not quite as easy to use) you are just testing the "local loop" portion of your connection.

I should point out here that the CIR (bandwidth you are promised) is generally only stated in terms of the local loop, since normally that's all that your access-ISP has control over.

FWIW you can download the Oookla (speedtest.net) client to run on your own network, which is rather handy if you want to test out that fancy 802.11ac wifi setup (or whatever).

Re:We really need some laws agains false advertisi

[HiThere]

The problem is that the laws that already exist aren't enforced. It's a secondary problem that they are so written that it's relatively easy to weasel around them, but even the existing laws aren't enforced, so adding new laws (that aren't enforced) wouldn't do any good.

Seperate content creation from Bandwidth provision

[bl968]

The FCC should make Comcast and Verizon and all these other companies chose... They can be content creation companies or bandwidth providers, but not both. This is all about making it easier and more convenient to pay them for their video services than to attempt to get it from a third party it's monopolistic and anti-competitive behavior...

Re: Speed tests

[apraetor]

Try testmy.net, they don't colocate with ISPs.

Re:Enraged Customer doesn't bother to research

[Khyber]

So, why am I getting crap speeds for Netflix on both PS3 and PC except when I use my VPN, inside Verizon's own network?

Re: Could be a different route involved for the VP

[evilviper]

More peering links won't rectify the imbalance. Your claims of knowing anything about the subject fall quite flat.

Re:Enraged Customer doesn't bother to research

[MrL0G1C]

Any combination of:
A) The article is wrong.
B) Verizon are also throttling Netflix - which would most likely happen one or two hops from your gateway.
C) The VPN causes your packets to exit Verizon's network at a different location which has better connectivity.
D) The VPN provider has a SLA with Verizon so they send the packets along a less congested route.

Mod Parent Up Please - It's exactly right.

[billstewart]

Netflix either buys bandwidth or convinces ISPs to give it to them for free as peering. Its peers / transit providers are either big enough to peer with Verizon for free, or buy bandwidth from Verizon, and they don't have fat enough connections to Verizon to carry all of Netflix's traffic to this customer, at least at the peering point that Netflix's traffic uses to reach that customer (Netflix may have enough bandwidth to reach VZ customers in the other part of the country, e.g. LA peering is full while Seattle or DC isn't.)

The customer's VPN provider may very well provide free/cheap connectivity to Netflix (or at least fast enough), maybe even one of the providers Netflix buys from, because they're cheap. And they may also get it from somebody who has good bandwidth to Verizon. But peering's just peering - it's not transit.

VZ isn't capping the customer's broadband.

[billstewart]

The problem isn't from Verizon's backbone to the customer - otherwise the VPN connection would have also been slow. It's the pipes from Netflix and their peering/transit providers to Verizon, which aren't big enough to handle all the Netflix customers on Verizon, at least in that region.

Also, what do you mean "has to set aside 8tb for sync"? Do you mean the ISP has to provide 8TB of some kind of storage hardware, or 8 terabits per second of traffic from somewhere to somewhere else? That's fairly huge, considering that most data connections aren't much bigger than 10 Gbps per wavelength (some carriers use 40, but it's not usually price-competitive.)

Difference is the route, not the protocol

[billstewart]

The reason the VPN connection went fast isn't that EEEVILLL Verizon was throttling the customer's Netflix connections by doing deep packet routing and didn't do that to the VPN. The pipes Netflix bought to deliver movies to their paying customers who use Verizon weren't big enough to carry all the demand, at least at the peering* point that customer's traffic went through, while the pipes they bought or peering they got for free were big enough to reach the VPN endpoint, and the VPN endpoint had bought enough bandwidth from their ISPs to get from there to their peering point with Verizon, so there was enough bandwidth on the whole route to carry the movie that way.

That's not to say that there aren't ISPs harassing particular content (there was at least one well-publicized case a few years ago of some telco ISP blocking VOIP, and of course most of the cable modem and some DSL providers block home web servers), but this ain't one of them.

(*Peering unfortunately means two different things here - it's giving each other service for free, and it's having BGP-managed interconnections, usually at the big internet exchange locations, to pass traffic, not necessarily for free.)

Re:Difference is the route, not the protocol

[EvilJoker]

While this may bypass claims of malice, it does place it firmly in the realms of incompetence. If Verizon's link to a peer is overloaded, it either needs more/different peers, or a bigger link.
Getting a VPN is the end-user creating a different peer that Verizon doesn't already have.

Of course, this is all assuming that Verizon ISN'T trying to throttle Netflix, which doesn't seem like a safe assumption. BTW, no deep packet inspection needed, just a list of destination IPs. DPI is used to identify content, not end-points; it is easily beaten by decent encryption.

Re:Enraged Customer doesn't bother to research

[sl149q]

So connect to a different endpoint. My VPN provider has many dozens of different locations that I can connect to in all major locations in the US (and world wide.)

Re: Could be a different route involved for the V

[evilviper]

Who cares about the imbalance?

Verizon and Level-3.

It's the congestion we want fixed.

Good, then complain to Netflix. They can fix it.

So you are saying that Verizon is too stupid?

[Kludge]

So you are saying that Verizon is too stupid to balance connections and packets over multiple routes? You can even buy multi-WAN routers for your home that do this for you. Verizon can't figure it out?

Switch

[Murdoch5]

Go with a smaller ISP, I live in Toronto and we have 0 throttling, I get 5 MB/s at all times.

Re:Sounds like what comcast has been doing...

[PlusFiveTroll]

Sounds like bufferbloat from hell. Or a QoS rule from hell : /

Faster VPN Access

[Agripa]

Back when I was in the process of switching providers, I bought a subscription to a VPN service so I could have a secure connection and routable IP through public internet access points. Later one of the things I noticed with my new AT&T U-Verse service was that *all* access was faster in either latency or throughput using the VPN to tunnel through U-Verse to half way across the US including things like DNS. Some things were a little bit faster and some things were an order of magnitude faster.

Re: Could be a different route involved for the VP

[nctritech]

Mod parent up...heavily.

Re: Could be a different route involved for the VP

[bigpat]

Also how is it that L3 can deliver the data to Verizon, but Verizon is just somehow incapable of passing it along to their customers? The answer is that it is not a techn

Re: Could be a different route involved for the V

[bigpat]

Not a technical decision, but greed driven decision.

changing your routes changes the interconnects

[mr_mischief]

changing your routes changes the interconnects
changing your routes changes the interconnects
changing your routes changes the interconnects
changing your routes changes the interconnects
changing your routes changes the interconnects

Seriously, folks, changing your routes changes the interconnects.

His VPN provider probably has a much better route back to Verizon. Yes, Verizon is being somewhat dickish to not acknowledge that Netflix is a big driver for their higher speed plans and giving Netflix's bandwidth carriers a bit of a price break for that reason. No, this is no proof at all of throttling.

Is it evidence suggesting throttling? Well, yeah. Proof? Not even close. It's entirely consistent with what Verizon already said about an imbalanced interconnect that needs more hardware.

Re:We really need some laws agains false advertisi

[MooseMiester]

Correct, and god forbid the CONSUMER actually expends a tiny bit of EFFORT and educates themselves. Very few places left where you don't have more than one choice for getting your Internet.

Of course the lazy folks, who want someone else to do their thinking for them, want somebody else to do all the work... Despite the fact that this always turns out the same, the politicians and the business folks get rich, the consumer gets screwed. Doesn't matter which party is in charge.

So get out there and vote - with your wallet. It's the most powerful vote you have.

Re:Why not the otherway around?

[EvilJoker]

While that is possible, there doesn't seem to be any reason why Netflix would do that. Netflix has nothing to gain by unsatisfied customers on Verizon.

Verizon, OTOH, has quite a bit to gain by making their customers unsatisfied with Netflix.

If I have overlooked something, please let me know.

Re:Some sites slow on FIOS due to Verizon DNS issu

[EvilJoker]

It looks like you've stumbled onto one of the features of CDNs: Local copies. CDNs distribute content to a more local link, so that the same data isn't being re-transmitted through backbone links. Part of how this often works, is DNS. Each ISP operates their own DNS server, and has modified entries for CDN content. These point to the local CDN server, rather than the original source. The result is (supposed to be) a much faster and reliable connection.

When you use an independent or alternate DNS server, you do not connect to the local CDN. You get someone else's CDN, and the data still has to cross the internet again. It may, however, take a very different route, one that is not as congested.

Re:More likely

[EvilJoker]

Assuming Verizon wanted the traffic to come in a different route (e.g. LimeLight, etc), I believe they can update the BGP route they advertise to change that.

Otherwise, if Verizon just asks nicely, I can promise that Netflix would adjust the routing tables on their end to send it through another link. Just like they can expand the link with Level 3.

Re: Could be a different route involved for the VP

[InvalidError]

L3 cannot deliver the data to Verizon since there is not enough connectivity between L3 and Verizon to hand the data over at the interfaces where L3 is attempting to do so.

Verizon does not want to put all their bandwidth eggs in L3's basket just to accommodate Netflix so they want Netflix to either peer directly or force L3 and its other CDNs to re-route traffic through other Verizon peers.

Depending too heavily on a single upstream provider is not sound business practice and Verizon wants to avoid getting tied up in that sort of relationship with L3 mostly due to Netflix.